An underwater robot posture control method and device based on a PPO algorithm

Abstract

The embodiment of the application provides a kind of underwater robot posture control method and device based on PPO algorithm, belong to computer technology field.The method comprises: constructing first dynamics model, determines first training sample according to first dynamics model;The posture of underwater robot when shooting target is in the center of shooting picture is determined as preset posture, determines second training sample according to first training sample and preset posture;Based on Actor-Critic mechanism, construct deep neural network, input second training sample into deep neural network for training, obtain the target deep neural network trained;The first dynamics model is updated using the deep neural network, obtain second dynamics model;Underwater robot is controlled based on second dynamics model and first interference value.The application can optimize dynamics model, ensure the shooting stability and control accuracy of underwater robot in complex underwater environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an underwater robot posture control method and device based on PPO algorithm. BACKGROUND

[0002] With the development of marine tourism, more and more people use underwater robots to track and shoot. During the shooting process, the underwater robot shoots the target by action tracking. In the process of following the target, the underwater robot needs to adjust the motion posture to ensure the shooting effect. The current underwater robot posture control method generally includes the following three kinds:

[0003] I. By designing a switching function, the system state is kept sliding on the sliding surface to achieve the desired attitude control effect;

[0004] II. By adjusting the proportional, integral and differential parameters, the system posture in the environment with small linear system and static error is adjusted by the PID controller;

[0005] III. According to the change of system parameters, the control strategy is adjusted in real time, and the system posture is adaptively controlled under different operating conditions and environmental changes.

[0006] However, due to the complex ocean current environment in the ocean, the shooting stability and tracking shooting accuracy of the underwater robot are affected. With the continuous expansion of underwater shooting demand and the continuous improvement of shooting effect requirements, the current tracking shooting technology of underwater robot cannot meet the individual needs. SUMMARY

[0007] To solve the above technical problems, the purpose of the present application is to provide an underwater robot posture control method and device based on PPO algorithm, which can realize accurate tracking and shooting when the underwater robot shoots the shooting target, improve the stability and precision of the underwater robot in controlling the posture in the underwater environment, and help the underwater robot cope with complex underwater environment.

[0008] To achieve the above purpose, one aspect of the embodiment of the present application provides an underwater robot posture control method based on PPO algorithm, which comprises:

[0009] A first dynamic model is constructed, and a first training sample is determined according to the first dynamic model. The first training sample is the posture data of the underwater robot when it interacts with the underwater environment;

[0010] The posture of the underwater robot when the shooting target is in the center of the shooting picture is determined as a preset posture, and a second training sample is determined according to the first training sample and the preset posture. The shooting target is the target tracked and shot by the underwater robot;

[0011] constructing a deep neural network based on an Actor-Critic mechanism, inputting the second training sample into the deep neural network for training, and obtaining a trained target deep neural network;

[0012] obtaining a first interference value, the first interference value being an interference value generated by the underwater robot and the underwater environment when the underwater robot interacts with the underwater environment;

[0013] updating the first dynamic model by using the trained deep neural network, and obtaining a second dynamic model;

[0014] controlling the posture of the underwater robot based on the second dynamic model and the first interference value.

[0015] Optionally, the first dynamic model is constructed, and an expression of the first dynamic model is:

[0016]

[0017] wherein M represents a matrix composed of a system inertia matrix and an added mass matrix, C(v) represents a Coriolis and centripetal force matrix composed of a rigid body term and an added mass matrix, D(v) represents a damping matrix caused by resistance, g(η) represents a restoring force matrix caused by gravity and buoyancy, B represents a control matrix depending on the configuration of the underwater robot thruster, u represents a force and moment vector provided by the underwater robot thruster, ζ represents a force and moment vector generated by the underwater environment, represents a position and posture vector of the underwater robot in the earth coordinate system, represents a derivative of the position and posture vector of the underwater robot in the earth coordinate system, (x, y, z) represents coordinates of the underwater robot in the earth coordinate system, represents a posture vector of the underwater robot in the earth coordinate system, ε = (u, v, w, p, q, r) T represents a linear velocity and angular velocity vector of the underwater robot in the body coordinate system, represents a derivative of the linear velocity and angular velocity vector of the underwater robot in the body coordinate system, wherein (u, v, w) are tangential, radial and normal components of the linear velocity of the underwater robot in the body coordinate system, respectively.

[0018] Optionally, the posture of the underwater robot when the shooting target is in the center of the shooting picture is determined as the preset posture, and the second training sample is determined according to the first training sample and the preset posture, specifically including the following steps:

[0019] The posture of the underwater robot when the shooting target is in the center of the shooting picture is determined as the preset posture, and the expression of the preset posture is (φ d ,θ d ,ψ d ) and d, wherein (φ d ,θ d ,ψ d ) is a preset coordinate, and d is a preset distance between the underwater robot and the shooting target, and the shooting target is a target tracked and shot by the underwater robot.

[0020] The expression of the first sample is (φ c ,θ c ,ψ c ) and d c , wherein (φ c ,θ c ,ψ c ) is an actual coordinate between the underwater robot and the shooting target, and d c is an actual distance between the underwater robot and the shooting target.

[0021] The expression of the second training sample is:

[0022]

[0023] wherein, is a coordinate error, is a distance error between the underwater robot and the shooting target.

[0024] Optionally, the deep neural network is constructed based on the Actor-Critic mechanism, the second training sample is input into the deep neural network for training, and a target deep neural network trained is obtained, and the method specifically comprises the following steps:

[0025] The value network and the policy network are configured based on the Actor-Critic mechanism.

[0026] The second training sample is input into the value network and the policy network for training, and a first training result is output.

[0027] The parameters of the value network and the policy network are updated according to the first training result and the descending gradient parameters of the deep neural network, and a target deep neural network trained is obtained.

[0028] Optionally, the first interference value is obtained, and the expression of the first interference value is:

[0029]

[0030] wherein, is a first disturbance value, y is an output of a posture control system of the underwater robot, G(s) is a transfer function of an observer, M represents a matrix composed of a system inertia matrix and an added mass matrix, and ε represents a linear velocity and angular velocity vector of the underwater robot in a moving coordinate system, represents a derivative of the linear velocity and angular velocity vector of the underwater robot in the moving coordinate system, C(ε) represents a Coriolis centripetal force and centripetal moment matrix composed of a rigid body term and an added mass matrix, D(ε) represents a damping matrix caused by resistance, g(η) represents a restoring force matrix caused by gravity and buoyancy, B represents a control matrix depending on a propeller configuration of the underwater robot, and u represents a force and moment vector provided by a propeller of the underwater robot, and the observer is used to measure a disturbance caused by an underwater environment to the posture control of the underwater robot.

[0031] Optionally, the posture control of the underwater robot based on the second dynamic model and the first disturbance value specifically comprises the following steps.

[0032] A first control value output by the second dynamic model is acquired.

[0033] The first disturbance value is compensated into the first control value, and a second control value is calculated, and a calculation formula of the second control value is as follows:

[0034]

[0035] wherein a comp is the second control value, a is the first control value, is the first disturbance value.

[0036] Another part of the embodiment of the present application further provides an underwater robot posture control device based on a PPO algorithm, comprising:

[0037] A first module is configured to construct a first dynamic model, and determine a first training sample according to the first dynamic model, wherein the first training sample is posture data of an underwater robot when the underwater robot interacts with an underwater environment.

[0038] A second module is configured to determine a preset posture as a posture of the underwater robot when a shooting target is in a center of a shooting picture, and determine a second training sample according to the first training sample and the preset posture, wherein the shooting target is a target tracked and shot by the underwater robot.

[0039] A third module is configured to construct a deep neural network based on an Actor-Critic mechanism, input the second training sample into the deep neural network for training, and obtain a target deep neural network after training.

[0040] a fourth module configured to obtain a first interference value, the first interference value being an interference value generated by the underwater environment on the underwater robot when the underwater robot interacts with the underwater environment;

[0041] a fifth module configured to update the first dynamic model by using the trained deep neural network to obtain a second dynamic model;

[0042] a sixth module configured to perform attitude control on the underwater robot based on the second dynamic model and the first interference value.

[0043] Another aspect of the embodiment of the present application further provides an electronic device including a processor and a memory;

[0044] The memory is configured to store a program;

[0045] The processor executes the program to implement the method as described above.

[0046] The embodiment of the present application further discloses a computer readable storage medium, the storage medium stores a program, and the program is executed by a processor to implement the method as described above.

[0047] The embodiment of the present application further discloses a computer program product, including a computer program, and the computer program is executed by a processor to implement the method as described above.

[0048] The embodiment of the present application can realize optimization of the dynamic model, improve the intelligence of the dynamic model, and ensure the shooting stability and control precision of the underwater robot in the complex underwater environment. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0050] Figure 2 is a flowchart of an underwater robot attitude control method based on a PPO algorithm provided by an embodiment of the present application;

[0051] Figure 3 is a relative coordinate diagram between an underwater robot and a shooting target provided by an embodiment of the present application;

[0052] Figure 4 is an unfolded flowchart of constructing a deep neural network based on an Actor-Critic mechanism, inputting the second training sample into the deep neural network for training, and obtaining a trained target deep neural network provided by an embodiment of the present application;

[0053] Figure 5 is an unfolded flowchart of performing attitude control on the underwater robot based on the second dynamic model and the first disturbance value provided by an embodiment of the present application;

[0054] Figure 6 is a schematic diagram of an algorithm core of a reinforcement learning PPO algorithm combined with disturbance compensation provided by the present application;

[0055] Figure 7 is a flowchart of another underwater robot attitude control based on a PPO algorithm provided by the present application;

[0056] Figure 8 is a structural schematic diagram of an underwater robot attitude control device based on a PPO algorithm provided by an embodiment of the present application;

[0057] Figure 9 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation modes described in the following exemplary embodiments do not represent all implementation modes consistent with the embodiments of the present application, but are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0059] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to a determination".

[0060] The terms "at least one", "multiple", "each", "any", and the like used in the present application include one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by a person skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0062] With the rapid development of marine tourism, the number of diving enthusiasts has increased significantly, accompanied by the rapid rise of underwater photography demand. However, when diving personnel take photos underwater, they usually rely on the help of their companions or use underwater robots to complete the shooting. Since the untethered battery-powered underwater robot has good endurance and portability, it is favored by diving enthusiasts. However, when the underwater robot tracks and shoots the actions of the diving personnel in a complex ocean current environment, it is often disturbed by the ocean current, which may lead to unstable distance shooting or failure to accurately capture the target, which puts higher requirements on the control system of the underwater robot.

[0063] Therefore, in an uncertain ocean current environment, there is an urgent need for a strong underwater robot posture control method that can timely perceive environmental changes and adaptively adjust the posture, to ensure the stability of the hovering shooting of the underwater robot.

[0064] In the prior art, the posture control method of the underwater robot mainly includes sliding mode control, PID control, adaptive control, etc. The sliding mode control has strong robustness, and by designing a switching function, the system state is kept sliding on the sliding mode surface to achieve the desired posture control effect. The PID control is widely used because of its simplicity and ease of implementation. By adjusting the three parameters of proportion, integral and derivative, the PID controller can accurately adjust the system posture in a linear system and a system with small static error in the environment; the adaptive control can adjust the control strategy in real time according to the change of system parameters, and adaptively control the system posture under different operating conditions and environmental changes.

[0065] Compared with the deficiencies of traditional PID control, sliding mode control and adaptive control methods, such as fixed parameters being difficult to adapt to complex environment, control precision being limited, real-time adjustment being insufficient and the like, the PPO control combined with interference compensation in the embodiment of the application can dynamically cope with complex nonlinear and uncertain environment through adaptive learning and strategy optimization capability, so as to realize higher control precision and robustness, and is particularly suitable for attitude control of underwater robots under complex water flow conditions.

[0066] The PPO algorithm-based underwater robot attitude control method provided in the embodiment of the application relates to the technical field of computers. The PPO algorithm-based underwater robot attitude control method provided in the embodiment of the application can be applied to a terminal, can be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as a stand-alone physical server, can be configured as a server cluster or a distributed system formed by multiple physical servers, can be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network; the software can be an application that implements the conversion method for the subgrade slope model, and the like, but is not limited to the above forms.

[0067] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0068] In order to facilitate the understanding of the technical solutions of the present application, the technical feature-specific terms that can appear in the embodiments of the present application are first explained and described:

[0069] 1) PPO algorithm: PPO (Proximal Policy Optimization) algorithm is a policy gradient method used to solve the problem of policy update in reinforcement learning. PPO algorithm limits the difference between the old and new policies to ensure the stability of learning while improving data utilization efficiency. PPO algorithm is an on-policy method, but in practical applications, it is often converted into an off-policy method to improve efficiency.

[0070] 2) Actor-Critic mechanism: The core idea of Actor-Critic mechanism is to separate policy learning (Actor) and value function estimation (Critic) into two mutually cooperative components. The policy network is responsible for generating the action selection strategy of the agent in each state, and the value network is responsible for evaluating the goodness of the strategy selected by the Actor, i.e., estimating the state value function or action value function. Actor and Critic form a closed-loop feedback mechanism, and this collaborative working mode effectively combines the stability of value-based reinforcement learning methods and the flexibility of policy-based reinforcement learning methods.

[0071] 3) Dynamic model: Dynamic model is the core of underwater robot research, and its modeling methods include traditional analytical model and simulation-based computational model. In the model construction process, the constraint equation, dynamic equation and the action law of various external and internal forces of the robot need to be determined.

[0072] As shown in Figure 1 , it is an implementation environment schematic diagram provided by an embodiment of the present application. Referring to Figure 1 , the implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected through wireless or wired network to complete data transmission and exchange.

[0073] The server 101 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0074] In addition, the server 101 can also be a node server in a blockchain network. The blockchain is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies.

[0075] The terminal 102 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 102 and the server 101 can be connected directly or indirectly through wired or wireless communication, and the present application is not limited thereto.

[0076] Exemplarily based on Figure 1 The present application provides a PPO algorithm-based underwater robot posture control method, which is applied to the terminal 102. It should be understood that the PPO algorithm-based underwater robot posture control method can also be applied to the server 101.

[0077] In one aspect, referring to Figure 2 , Figure 2 The PPO algorithm-based underwater robot posture control method provided by the present application is shown in the flowchart. The execution subject of the PPO algorithm-based underwater robot posture control method can be any one of the aforementioned computer devices (including servers or terminals). Referring to Figure 2 , the method comprises the following steps:

[0078] S100, a first dynamic model is constructed, and a first training sample is determined according to the first dynamic model, the first training sample being posture data when the underwater robot interacts with the underwater environment.

[0079] Optionally, the expression of the first dynamic model is:

[0080]

[0081] where M represents a matrix composed of system inertia matrix and added mass matrix, C(v) represents a Coriolis centripetal force and centripetal moment matrix composed of rigid body items and added mass matrix, D(v) represents a damping matrix caused by resistance, g(η) represents a restoring force matrix caused by gravity and buoyancy, B represents a control matrix depending on the configuration of the underwater robot thruster, u represents a force and moment vector provided by the underwater robot thruster, ζ represents a force and moment vector generated by the underwater environment, and η=(x,y,z,φ,θ,ψ) T represents a position and attitude vector of the underwater robot in the earth coordinate system, represents a derivative of the position and attitude vector of the underwater robot in the earth coordinate system, (x,y,z) represents coordinates of the underwater robot in the earth coordinate system, (θ,ψ) represents an attitude vector of the underwater robot in the earth coordinate system, and ε=(u,v,w,p,q,r) T represents a linear velocity and angular velocity vector of the underwater robot in the body coordinate system, denotes the derivative of the linear and angular velocity vectors of the underwater robot in the body-fixed coordinate system, where (u, v, w) are the tangential, radial and normal components of the linear velocity of the underwater robot in the body-fixed coordinate system, respectively.

[0082] It should be noted that the first dynamic model is the dynamic model of the underwater robot in matrix form obtained according to the relevant derivation contents in the first chapter of the book Marine Control System by Fossen. When the underwater robot tracks and shoots the target in a complex ocean current environment, it is often disturbed by the ocean current, resulting in unstable fixed-distance shooting or failure to accurately capture the target.

[0083] Therefore, on the basis of the original derivation content, the dynamic disturbance of the ocean current is considered in the attitude control of the underwater robot, and specifically, the data ζ representing the force and torque vector generated by the underwater environment is compensated in the attitude control of the underwater robot, thereby significantly improving the stability and control accuracy of the underwater robot in a complex ocean current environment.

[0084] S200, determine the attitude of the underwater robot when the shooting target is in the center of the shooting picture as a preset attitude, and determine a second training sample according to the first training sample and the preset attitude, the shooting target being a target tracked and shot by the underwater robot.

[0085] It should be noted that in some embodiments, as shown in Figure 3 The relative coordinate diagram between the underwater robot and the shooting target is shown in FIG. 8. Based on the position of the underwater robot, the XYZ coordinate axes are set, and the underwater robot tracks and shoots the diver.

[0086] Exemplarily, in some specific embodiments, eight thrusters are arranged on the underwater robot, four thrusters distributed at both ends of the two sides of the body of the underwater robot are responsible for driving the XOY plane of the body, and four thrusters distributed inside the body are longitudinal thrusters responsible for driving the movement of the body on the Z axis. During the adjustment of the attitude of the underwater robot, the four thrusters distributed on both sides of the body can be adjusted according to the distance information obtained by adjusting the attitude of the underwater robot based on the PPO algorithm, so that the body advances or retreats to meet the requirement of the distance information; in addition, the eight thrusters can be driven according to the angle information obtained by adjusting the attitude of the underwater robot based on the PPO algorithm to meet the requirement of the angle information.

[0087] Optionally, the attitude of the underwater robot when the shooting target is in the center of the shooting picture is determined as the preset attitude, and the second training sample is determined according to the first training sample and the preset attitude, the shooting target being a target tracked and shot by the underwater robot, specifically:

[0088] The posture of the underwater robot when the shooting target is in the center of the shooting picture is determined as the preset posture, and the expression of the preset posture is and d, wherein, is a preset coordinate, and d is a preset distance between the underwater robot and the shooting target, and the shooting target is a target tracked and shot by the underwater robot;

[0089] Exemplarily, in some embodiments, the first training sample and the second training sample are used to train an agent in a neural network based on a PPO algorithm, which is used to enable the underwater robot to adjust the shooting target to the center of the shooting picture at a certain distance from the shooting target.

[0090] Specifically, the expression of the first sample is and d c wherein, is an actual coordinate of the underwater robot, and d c is an actual distance between the underwater robot and the shooting target;

[0091] The expression of the second training sample is:

[0092]

[0093] wherein, is a coordinate error, is a distance error between the underwater robot and the shooting target.

[0094] Exemplarily, in some embodiments, the posture adjustment process of the underwater robot by the agent in the neural network based on the PPO algorithm can be as follows:

[0095] 1. Project the shooting target on the XOY plane, and use the distance d measured by the fixed-distance sonar to obtain the actual distance of the underwater robot in the XOY plane to the target object c and set the error of the distance d in the xoy plane and the arbitrary posture to the posture error of the arbitrary posture The posture error is defined as the state space s as the input of the reinforcement learning state;

[0096] 2. Define as the posture angle and distance adjustment as a continuous action space a, and in some embodiments, the range of can be set within [-5°, 5°];

[0097] 3. After taking state s as input to the reinforcement learning state, the agent outputs the attitude adjustment information of the underwater robot. In some specific implementations, the attitude adjustment information may include: angle information for adjusting the three Euler angles of the underwater robot and distance information for adjusting the position of the underwater robot.

[0098] S300. Construct a deep neural network based on the Actor-Critic mechanism, input the second training sample into the deep neural network for training, and obtain the trained target deep neural network.

[0099] It should be noted that in some embodiments, such as Figure 4 As shown, constructing a deep neural network based on the Actor-Critic mechanism, and inputting the second training sample into the deep neural network for training to obtain the trained target deep neural network, may include the following steps:

[0100] S401. Configure the value network and policy network based on the Actor-Critic mechanism;

[0101] It should be noted that the Actor-Critic mechanism is the neural network mechanism in the PPO algorithm. The Actor network is the policy network, and the Critic network is the value network. Both the Actor and Critic networks include an input layer, a hidden layer, and an output layer. The activation function of the hidden layer can be the ReLU function, and the output layer can use the tanh function.

[0102] For example, in some specific implementations, configuring the value network and policy network based on the Actor-Critic mechanism may include initializing the parameters of the Actor network and the Critic network. The parameters used for initialization may include the learning rate of the two networks, the network update frequency, the experience pool size, and other parameters.

[0103] It should be noted that the Actor network can be used to output attitude adjustment information based on the actual attitude and distance error of the underwater robot. This attitude adjustment information can include: angle information for adjusting the three Euler angles of the underwater robot and distance information for adjusting the position of the underwater robot.

[0104] It should be noted that the Critic network can be used to output the state value function V(s) of the underwater robot performing the action in the actual state, based on the input state variable s and action variable a. The advantage value at time t can be obtained through V(s).

[0105]

[0106] Where λ is a subset of A tAn extra hyper-parameter introduced in the exponentially weighted average calculation, γ is the discount factor, V(s t+ι ) is the state value function at time t+1, r t+ι is the reward function at time t+1, and T is the maximum time step.

[0107] Exemplarily, in some embodiments, the reward function can be defined as C1, C2 are weight coefficients, r e (t), and X are defined as follows:

[0108]

[0109] where k i , i = 1, 2, 3, 4, 5, 6 are set reward values, and μ1, μ2, μ3 are tolerance errors.

[0110] S402, input the second training sample into the value network and the policy network for training, and output a first training result;

[0111] Exemplarily, in some embodiments, before training the value network and the policy network, the number of training rounds is set as TN, the number of training cycles is set as T, and the position of the underwater robot is initialized, an initial state vector s is obtained, and the initial time step i = 0; based on the current state s, the action a = π(s|θ) output by the Actor network is adopted, θ is a parameter of the policy network, representing the parameterization of the target policy π θ .

[0112] S403, updating the parameters of the value network and the policy network according to the first training result and the descending gradient parameters of the deep neural network, to obtain a trained target deep neural network.

[0113] Exemplarily, in some embodiments, the underwater robot interacts with the environment based on the current pose to obtain environmental feedback.

[0114] The pose data and environmental information obtained by the sensor can calculate the relative distance and pose error between the current position and the target position, and generate the corresponding action instruction a through the reinforcement learning algorithm, and store the information obtained by the sensor into the experience pool.

[0115] At the same time, the strength and direction of the ocean current disturbance are estimated in real time by using the compensation control observer The estimated value of the disturbance is compensated to the action output by the deep neural network to obtain the actual control value The action a compA new state s' and action a' are obtained, s is set to s', and then the immediate reward r, the action probability policy, the value, and the episode end flag done are calculated, and the training of the deep neural network for this round is ended.

[0116] After completing a time step, the state, action, immediate reward, action probability policy, value, and episode end flag done obtained by interaction at the current time step are recorded, and are sequentially stored in the experience pool to form a trajectory, and the data in the experience pool will be used for subsequent policy updating.

[0117] Let i be incremented and determine whether the time step number i reaches the maximum time step T or the episode end flag has been reached.

[0118] If not, the above loop is continued until the maximum time step T is reached or the episode end flag has been reached, and a complete trajectory is formed. If yes, the complete trajectory currently trained by the experience pool is taken out, the reward value r(t) is calculated, the advantage value A t and the TD error γV(s t+1 )+r t+1 -V(s t ) are calculated.

[0119] The mean square error (MSE) loss function is used, and the parameters of the value network Critic are updated by propagating in the gradient descent direction of the deep neural network.

[0120] The loss function L clip (θγ)=E t [min(r t (θ)A t ,clip(r t (θ),1-∈,1+ε)A t )], the parameters of the value network Critic are updated by propagating in the gradient descent direction of the deep neural network, and r t (θ) is the action probability ratio , π old represents the old policy, and ∈ is a hyperparameter in PPO, which is used to control the amplitude of policy updating.

[0121] S400, obtaining a first interference value, the first interference value being an interference value generated by an underwater environment on the underwater robot when the underwater robot interacts with the underwater environment;

[0122] Since the influence of ocean current on the movement of the underwater robot cannot be ignored in the underwater environment, in some specific embodiments, an observer for quantifying the ocean current interference can be set to calculate the first interference value. In a specific implementation, the observer can be an ocean current interference observer.

[0123] Optionally, the expression of the first interference value is:

[0124]

[0125] wherein, is the first interference value, y is the output of the attitude control system of the underwater robot, G(s) is the transfer function of an observer used to measure the disturbance to the attitude control of the underwater robot caused by the underwater environment, is the rate of change of the three Euler angles in the first sample and the second training sample of the underwater robot, and M represents a matrix composed of the system inertia matrix and the added mass matrix.

[0126] S500, updating the first dynamic model using the trained deep neural network to obtain a second dynamic model.

[0127] When the number of training rounds of the deep neural network reaches the preset value, the trained deep neural network is saved, the first dynamic model is updated to obtain a new second dynamic model when the attitude of the underwater robot is controlled, and the new second dynamic model is used to control the attitude of the underwater robot.

[0128] S600, controlling the attitude of the underwater robot based on the second dynamic model and the first interference value.

[0129] It should be noted that, in some embodiments, as shown in Figure 5 controlling the attitude of the underwater robot based on the second dynamic model and the first interference value can include the following steps:

[0130] S501, obtaining a first control value output by the second dynamic model;

[0131] S502, compensating the first interference value into the first control value to calculate a second control value, and the calculation formula of the second control value is:

[0132]

[0133] wherein, a comp is the second control value, a is the first control value, is the first interference value.

[0134] To explain the principle of the technical scheme of the present application, the overall process of the present application will be described below in conjunction with some specific embodiments. It should be easily understood that the following is an explanation of the technical principle of the present application and cannot be regarded as a limitation of the present application.

[0135] In the embodiments of the present application, a reinforcement learning PPO algorithm combined with interference compensation is provided, which is described with reference toFigure 6 , Figure 6 This demonstrates the core algorithm of a reinforcement learning PPO algorithm that incorporates interference compensation:

[0136] This will represent the underwater robot's current attitude and distance error. The Actor loss function is input into the Actor network, and the Actor network outputs the angle and thrust adjustment vector 'a' for this state. t .

[0137] The output of the ocean current interference observer The output of the Actor network is added to obtain the actual control values ​​for the underwater robot's attitude control. By executing the actual control value through the dynamic model of the underwater robot, the new state S of the underwater robot is obtained. t And action a', let S t =S t Then calculate the immediate reward r, the action probability policy logarithm, and the round end flag done.

[0138] Obtain the new state S of the underwater robot t Afterwards, error calculation is performed based on the preset attitude set for the underwater robot. Specifically, the preset attitude set for the underwater robot may include the preset coordinates (φ) of the underwater robot. d ,θ d ,ψ d ) and the preset distance d between the underwater robot and the target being photographed. d .

[0139] After obtaining the error calculation result, the immediate reward r is calculated based on the error calculation result. Specifically, the error calculation result may include the attitude error. Rate of change of Euler angle error Distance difference between the underwater robot's current posture and its preset posture

[0140] After obtaining the instantaneous reward r and the state function V of the Critic network output, the advantage value A at time t is obtained through V. t The CLIP model is calculated using a loss function, based on the advantage value A. t Calculate the Actor Loss parameter to update the Actor network. Update the Critic Loss parameter to update the Critic network using the Mean Squared Error (MSE) loss function.

[0141] In some specific application scenarios, such as Figure 7 As shown, another underwater robot attitude control method based on the PPO algorithm proposed in this application is specifically implemented as follows:

[0142] S701, a dynamics model of the underwater robot is established.

[0143] S702, a state space and an action space of the underwater robot are set, a corresponding reward function and a target weight are set, a deep neural network Actor-Critic network is initialized, a training round number and a training period are set.

[0144] S703, interaction with the environment is performed.

[0145] S704, an output action, a next time state and an immediate reward are obtained.

[0146] S705, the obtained state, action and immediate reward are saved into an experience pool.

[0147] S706, an action output by the deep neural network is compensated by an interference value output by an observer, and used to adjust a current posture of the underwater robot.

[0148] It should be noted that, in some embodiments, when the trajectory is not ended, i.e., the maximum time step T is not reached or the end flag done is not reached, step S703 is performed; when the trajectory is ended, step S707 is performed.

[0149] S707, the data in the experience pool is used to update the value network and the policy network.

[0150] It should be noted that, in some embodiments, when the training round number is not reached, step S702 is performed; when the training round number is reached, step S708 is performed.

[0151] S708, the data in the experience pool is used to update the value network and the policy network.

[0152] In summary, the underwater robot posture control method based on the PPO model-free learning algorithm is proposed, the interference of the ocean current on the underwater robot is considered in the posture control of the underwater robot, the ocean current interference can be observed and estimated, and the ocean current interference can be compensated in the posture control of the underwater robot, so that the shooting stability and control accuracy of the underwater robot in the complex underwater environment are ensured. Compared with the prior art, the present application at least has the following beneficial effects:

[0153] 1. The target distance error is introduced, the corresponding reward function is designed, the real-time iteration and the network parameter adjustment are achieved through training, the control strategy can be adaptively adjusted in the dynamically changing underwater ocean current environment, the accurate follow-up shooting of the shooting target is realized, the shooting target is ensured to be always in the center of the shooting picture, and the underwater robot can be stably kept at a certain distance from the shooting target, and the posture control stability of the underwater robot in the complex underwater current environment is effectively improved.

[0154] 2. The power disturbance of ocean current is considered in the control adjustment, so that the stability and control accuracy of the underwater robot in complex current conditions are significantly improved.

[0155] 3. The modeling of multi-dimensional state and action space is increased, the robot attitude error and target distance error are simultaneously introduced into the state space, and the adaptive adjustment of the action space is performed through the deep neural network. Specifically, the adaptive adjustment of the action space includes attitude angle adjustment and horizontal thrust adjustment, so as to realize accurate control and stable hovering of the underwater robot attitude, make the control effect easier to observe, and the control method more accurate.

[0156] Referring to Figure 8 The embodiment of the application also provides an underwater robot attitude control device 800 based on a PPO algorithm, which can realize the underwater robot attitude control method based on the PPO algorithm. The device comprises:

[0157] A first module 801 is configured to construct a first dynamic model and determine a first training sample according to the first dynamic model. The first training sample is attitude data of the underwater robot when interacting with the underwater environment.

[0158] A second module 802 is configured to determine the attitude of the underwater robot when the shooting target is in the center of the shooting picture as the preset attitude, determine a second training sample according to the first training sample and the preset attitude, and the shooting target is a target tracked and shot by the underwater robot.

[0159] A third module 803 is configured to construct a deep neural network based on an Actor-Critic mechanism, input the second training sample into the deep neural network for training until the number of training rounds reaches a preset value.

[0160] A fourth module 804 is configured to obtain a first interference value. The first interference value is an interference value generated by the underwater environment on the underwater robot when the underwater robot interacts with the underwater environment.

[0161] A fifth module 805 is configured to update the first dynamic model by using the trained deep neural network to obtain a second dynamic model.

[0162] A sixth module 806 is configured to perform attitude control on the underwater robot based on the second dynamic model and the first interference value.

[0163] It can be understood that the contents in the above method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the functions same as those of the above method embodiments, and achieve the same beneficial effects as those of the above method embodiments.

[0164] The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0165] It can be understood that the contents in the above method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the functions same as those of the above method embodiments, and achieve the same beneficial effects as those of the above method embodiments.

[0166] Please refer to Figure 9 , Figure 9 The electronic device of another embodiment is illustrated, which includes:

[0167] The processor 901 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0168] The memory 902 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 902 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are saved in the memory 902 and called and executed by the processor 901 to implement the underwater robot posture method based on the PPO algorithm of the embodiments of the present application.

[0169] The input / output interface 903 is used to realize information input and output.

[0170] The communication interface 904 is used to realize the communication interaction between the device and other devices, which can realize communication through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0171] A bus 905 transmits information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) in the device.

[0172] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are communicatively connected to each other within the device through the bus 905.

[0173] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the underwater robot posture method based on the PPO algorithm.

[0174] It can be understood that the contents in the above method embodiments are all applicable to the present storage medium embodiment, the present storage medium embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0175] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0176] The underwater robot posture control method and device based on the PPO algorithm provided by the embodiment of the present application consider the disturbance of the ocean current to the underwater robot in the posture control of the underwater robot, can implement observation and estimation of the ocean current disturbance, and compensate the ocean current disturbance to the posture control of the underwater robot, thereby ensuring the shooting stability and control precision of the underwater robot in a complex underwater environment.

[0177] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0178] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0179] The apparatus embodiments described above are merely exemplary, and the units described as separate units can or can not be physically separate, i.e., can be located in one place, or can be distributed over multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0180] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0181] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims of the foregoing drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so construed can be interchanged, such that, for example, without departing from the scope of the application, the embodiments described herein can be carried out in a different order than the one illustrated or described herein. In addition, the terms "comprising", "having" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or apparatus that comprises a list of steps or units not necessarily limited to those explicitly listed, but can include other not expressly listed or inherent to such processes, methods, products or apparatus.

[0182] It should be understood that in this application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0183] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0184] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e. they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0185] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0186] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0187] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A PPO algorithm-based underwater robot posture control method, characterized in that, The method comprises: constructing a first dynamic model and determining a first training sample according to the first dynamic model, the first training sample being attitude data of the underwater robot when interacting with the underwater environment; determining an attitude of the underwater robot when a shooting target is in the center of a shooting picture as a preset attitude, and determining a second training sample according to the first training sample and the preset attitude, the shooting target being a target tracked and shot by the underwater robot; The posture of the underwater robot when the shooting target is in the center of a shooting picture is determined as the preset posture, and an expression of the preset posture is and d, wherein, is a preset coordinate, and d is a preset distance between the underwater robot and the shooting target, and the shooting target is a target tracked and shot by the underwater robot. An expression of the first training sample is and wherein, is an actual coordinate of the underwater robot, is an actual distance between the underwater robot and the shooting target; an expression of the second training sample is: wherein, is a coordinate error, is a distance error between the underwater robot and the photographing target; constructing a deep neural network based on an Actor-Critic mechanism, inputting the second training sample into the deep neural network for training, and obtaining a trained target deep neural network; obtaining a first interference value, the first interference value being an interference value generated by the underwater environment on the underwater robot when the underwater robot interacts with the underwater environment; updating the first dynamic model by using the trained deep neural network to obtain a second dynamic model; controlling the attitude of the underwater robot based on the second dynamic model and the first interference value.

2. The method of claim 1, wherein, an expression of the first dynamic model is: where M represents a matrix composed of a system inertia matrix and an added mass matrix, represents a Coriolis and centripetal matrix composed of a rigid body term and an added mass matrix, represents a damping matrix due to drag forces, represents a restoring force matrix due to gravity and buoyancy forces, B represents a control matrix depending on the configuration of the underwater robot thrusters, u represents a force and moment vector provided by the underwater robot thrusters, represents a force and moment vector generated by the underwater environment, represents a position and attitude vector of the underwater robot in the earth-fixed frame, represents a derivative of the position and attitude vector of the underwater robot in the earth-fixed frame, (x, y, z) represents the coordinates of the underwater robot in the earth-fixed frame, represents an orientation vector of the underwater robot in the earth-fixed frame, represents a linear and angular velocity vector of the underwater robot in the body-fixed frame, represents a derivative of the linear and angular velocity vector of the underwater robot in the body-fixed frame, are respectively the tangential, radial and normal components of the linear velocity of the underwater robot in the body-fixed frame.

3. The method of claim 1, wherein, the deep neural network is constructed based on the Actor-Critic mechanism, the second training sample is input into the deep neural network for training, and a trained target deep neural network is obtained, specifically comprising the following steps: configuring a value network and a policy network based on the Actor-Critic mechanism; inputting the second training sample into the value network and the policy network for training, and outputting a first training result; updating parameters of the value network and the policy network according to the first training result and a descending gradient parameter of the deep neural network, and obtaining a trained target deep neural network.

4. The method of claim 1, wherein, an expression of the first interference value is: wherein, is a first disturbance value, is an output of a pose control system of the underwater robot, is a transfer function of an observer, M represents a matrix composed of a system inertia matrix and an added mass matrix, represents a linear velocity and angular velocity vector of the underwater robot in a body-fixed coordinate system, represents a derivative of a linear velocity and angular velocity vector of the underwater robot in a body-fixed coordinate system, represents a Coriolis and centripetal matrix composed of a rigid body term and an added mass matrix, represents a damping matrix caused by drag forces, represents a restoring force matrix caused by gravity and buoyancy forces, B represents a control matrix dependent on a thruster configuration of the underwater robot, u represents a force and torque vector provided by thrusters of the underwater robot, the observer being configured to measure a disturbance of an underwater environment on the pose control of the underwater robot.

5. The method of claim 1, wherein, controlling the attitude of the underwater robot based on the second dynamic model and the first interference value, specifically comprising the following steps: obtaining a first control value output by the second dynamic model; compensating the first interference value into the first control value to calculate a second control value, and a calculation formula of the second control value is: wherein is the second control value, a is the first control value, is the first interference value.

6. A PPO algorithm-based underwater robot attitude control device, characterized by, comprising: a first module for constructing a first dynamic model and determining a first training sample according to the first dynamic model, the first training sample being attitude data of the underwater robot when interacting with the underwater environment; a second module for determining an attitude of the underwater robot when a shooting target is in the center of a shooting picture as a preset attitude, and determining a second training sample according to the first training sample and the preset attitude, the shooting target being a target tracked and shot by the underwater robot; The second module is further configured to determine the posture of the underwater robot when the shooting target is in the center of a shooting picture as the preset posture, and an expression of the preset posture is and d, wherein, is a preset coordinate, and d is a preset distance between the underwater robot and the shooting target, and the shooting target is a target tracked and shot by the underwater robot. The expression of the first sample is and wherein, is the actual coordinate of the underwater robot, is the actual distance between the underwater robot and the shooting target; an expression of the second training sample is: wherein, is a coordinate error, is a distance error between the underwater robot and the photographing target; a third module for constructing a deep neural network based on an Actor-Critic mechanism, inputting the second training sample into the deep neural network for training, and stopping training until a preset value of a training round number is reached; The fourth module is configured to acquire a first interference value, which is an interference value generated by an underwater environment on the underwater robot when the underwater robot interacts with the underwater environment; The fifth module is configured to update the first dynamic model by using the trained deep neural network to obtain a second dynamic model; The sixth module is configured to perform attitude control on the underwater robot based on the second dynamic model and the first interference value.

7. An electronic device, comprising: The device comprises a processor and a memory; The memory is configured to store a program; The processor executes the program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by the processor to implement the method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method according to any one of claims 1 to 5.

Images