Pipeline pneumatic soft robot control method and system based on deep reinforcement learning
Through the improved DDPG algorithm and predictive reward learning method, combined with neural network generation and evaluation of actions, the problem of insufficient continuous control performance of pipeline pneumatic software robots is solved, and fast and accurate autonomous motion control is achieved in three-dimensional space.
Patent Information
- Application Number
- CN202411721522.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2025-03-07
- Estimated Expiration
- 2044-11-28
AI Technical Summary
In the prior art, the continuous control performance of pipeline pneumatic soft robots has not reached a good state, and it is difficult to achieve real-time and fast control effects.
The improved DDPG algorithm is adopted, combined with the predictive reward learning method, and the actions are generated and scored through the actor network and the critical network, and the forward model neural network predicts the state and rewards, so as to guide the axial actuator to perform the actions and realize the control of the pipe pneumatic soft robot.
Effectively control the pipe pneumatic soft robot to perform autonomous continuous movement in three-dimensional space, which can quickly and accurately guide the tip to the target point and target direction, improving the continuous motion control performance of the pipe pneumatic soft robot.
Smart Images

Figure CN119567248A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of pipeline pneumatic soft robot control, and in particular to a pipeline pneumatic soft robot control method and system based on deep reinforcement learning. Background Art
[0002] When designing a robot controller, it can be divided into model-based control strategies and model-free control strategies.
[0003] Model-based control methods can improve the overall predictability and adaptability of the control system. The piecewise constant curvature (PCC) assumption can be used to build models, which relies on the deformation of each part of the robot along a circular arc and assumes that there is no gravity. It is usually applicable to super-redundant or rigid continuous robots. Unlike traditional rigid robots that have been widely studied for many years and have accurate analytical kinematics, soft robots are mostly composed of continuously deformable materials or mechanisms, with more degrees of freedom and high compliance. The PCC assumption is not applicable to the precise control of soft robots. Finite element-based models do not need to rely on such strict assumptions as PCC, and have been proven to be suitable for modeling nonlinear soft materials and their interactions with the environment. Wu et al. proposed a finite element-based nonlinear controller for a cable-driven soft robot. The robot's hysteresis and friction were taken into account in the model and good disturbance suppression was achieved. However, the huge computational cost makes the above finite element model impossible to use in a real-time closed control loop.
[0004] Therefore, soft robots will have modeling uncertainties due to internal and external factors, such as nonlinear deformation caused by drive, material elasticity and interaction with the environment. They face major challenges in accurate analytical modeling, which may reduce the effectiveness of model-based control methods. It is also difficult to achieve real-time and rapid control effects. The modeling strategy requires a difficult-to-optimize trade-off between the flexibility and controllability of the soft robot.
[0005] Model-free control strategies are independent of the shape of the robot and do not require definition of parameters in the configuration space or joint space. Due to the rich sample data and sensory noise, complex kinematic models can be created. Model-free control strategies may perform better for control systems that are highly nonlinear, non-uniform, affected by gravity, or operate in unstructured environments.
[0006] In model-free control strategies, the Jacobian matrix can be used to train neural networks directly from real data. Lu et al. proposed a data-driven method for the 3D shape servo control of a soft robot based on proprioceptive feedback. This method first obtains the initial estimate of the Jacobian matrix offline, and then continuously estimates the unknown model of the soft robot online as the robot moves in the workspace to cope with unexpected interference and exhibit adaptive behavior for unmodeled systems. However, the above method does not consider the dynamics of the soft robot and is more suitable for static control of pipeline soft robots.
[0007] Compared with the method of estimating the Jacobian matrix, model-free control strategies based on machine learning (ML) technology are more widely used. Almanzor et al. proposed an image-based shape control method for soft robots, which combines the local inverse kinematics formula in image space with convolutional neural networks to achieve precise shape control, but its control loop still runs slowly.
[0008] Reinforcement learning (RL) is a branch of machine learning technology and a general framework for designing nonlinear system controllers. It can learn optimal strategies directly from experience, eliminating the need to select the best action based on a separate control strategy when switching between states. The control strategy obtained using reinforcement learning technology is also more robust to external interference, providing a feasible method for handling complex situations and completing complex tasks. For example, Centuryli et al. used RL to control soft robots to work under different payload conditions. Since it can eliminate prior knowledge of the robot structure, reinforcement learning has received widespread attention in the field of robot control.
[0009] Traditional reinforcement learning is more suitable for discrete states, which is different from the actual application of pipeline soft robots. In order to process continuous state space, deep neural networks are introduced into reinforcement learning technology, which becomes deep reinforcement learning (DRL). Yao et al. used the Cosserat model to simulate the dynamics of magnetic soft robots, and used deep reinforcement learning algorithms to adapt the robots to different magnetization patterns and magnetic field restrictions, and fitted them with experimental results to speed up training. Wu et al. combined the data-driven modeling method with the deep Q network algorithm to realize the motion control task of the end of the cable-driven soft robot. However, the deep Q network algorithm based on the value function updates the strategy by evaluating the action value function, that is, the algorithm needs to calculate all action values at the next moment in the current state and select the optimal action value from them. When the action space is continuous, the computational cost is very high, which also leads to the fact that this algorithm can only realize discrete action control of soft robots.
[0010] In summary, the existing methods have not achieved a good state for the continuous control performance of pipeline pneumatic soft robots, and it is necessary to break through the limitations of traditional control strategies. Summary of the invention
[0011] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the continuous control performance of the pipeline pneumatic soft robot in the prior art has not reached a good state.
[0012] In order to solve the above technical problems, the present invention provides a pipeline pneumatic soft robot control method based on deep reinforcement learning, comprising:
[0013] The improved DDPG algorithm is used to control the movement of the axial actuator in the pipeline pneumatic soft robot to achieve control of the pipeline pneumatic soft robot. Specifically:
[0014] The improved DDPG algorithm is to introduce a prediction reward learning method into the DDPG algorithm;
[0015] The DDPG algorithm includes an actor network and a critic network, wherein the actor network is used to generate the action of the axial actuator, and the critic network is used to score the action generated by the actor network, calculate the action value function Q, and use the calculated action value function Q as the first reward;
[0016] The prediction reward learning method is implemented by a forward model neural network and a reward neural network, wherein the forward model neural network is used to predict the state generated by the axial actuator performing the next action, and the reward neural network is used to predict the corresponding second reward according to the state generated by the next action;
[0017] Combining the first reward corresponding to the DDPG algorithm and the second reward corresponding to the predicted reward learning method to obtain a strategy function for guiding the axial actuator to perform an action;
[0018] The axial actuator performs actions according to the strategy function to achieve control of the pipeline pneumatic soft robot.
[0019] In one embodiment of the present invention, the first reward corresponding to the DDPG algorithm and the second reward corresponding to the prediction reward learning method are combined to obtain a strategy function for guiding the axial actuator to perform an action, and the formula is:
[0020]
[0021] in, is the gradient of the expected return, N is the number of samples collected, λ is the weight factor, is the gradient of the action, Q() is the Q function, s is the state, a is the action, ω is the parameter of the Q function, s i is the current state of the input, μ(si ) is the current strategy of the input, is the state corresponding reward of the next action predicted by the prediction reward learning method, φ is the parameter of the feedforward neural network and the feedforward neural network is used as the forward model neural network, are the parameters of the reward network, is the gradient of parameter θ, and μ is the deterministic strategy to be optimized.
[0022] In one embodiment of the present invention, the learning process of the axial actuator of the pipeline pneumatic soft robot is modeled by using a Markov decision process MDP, including:
[0023] The state space, action space, and reward function of the axial actuator of the pipeline pneumatic soft robot are set as follows:
[0024] The state space: Combined with the bending effect of the pipeline pneumatic soft robot in the finite element analysis, four evenly distributed control points are taken on the axis of the axial actuator. According to the bending motion mode of the pipeline pneumatic soft robot, the end of the axial actuator is fixed, and the position of the control point is expressed by vector X a Indicates that the position of the target point is represented by vector X t Indicated by V a and V t Describe the magnitude and direction of the velocity of the axial actuator tip and the target, respectively, with q a and q t To describe the coordinate system orientation of the axial actuator tip and the target, the state space can be described as S = [X a , V a ,q a , X t , V t ,q t ];
[0025] The action space: According to the mechanical analysis of the pipeline pneumatic soft robot structure deformation, loads are applied to different cavity walls to realize the bending movement of the pipeline pneumatic soft robot axial actuator, and within a certain angle bending range, the inner length of the pipeline pneumatic soft robot axial actuator remains basically unchanged. Similarly, four evenly distributed control points on the axis are taken to apply torque A in the directions of d1(s), d2(s) and d3(s) on the axis. m (s) = α m β m (s), to realize the bending motion of the axial actuator of the pipeline pneumatic soft robot, α m is the maximum torque, β m The selection range of (s) is [-1, 1];
[0026] The reward function is: the distance between the tip position of the axial actuator of the pipeline pneumatic soft robot and the target point is defined as the distance error, which is expressed as:
[0027]
[0028] Where n = ||X n -X t || represents the tip X of the axial actuator of the pipeline soft robot n With the target point X t The distance between them, p is the direction penalty, and d is the reward range distance.
[0029] In one embodiment of the present invention, the forward model neural network is recorded as It is constructed by a feedforward neural network with parameter φ, assuming that the input is the current state s i and the current action a i , the next moment state As the output, construct the loss function of the forward model neural network, the formula is:
[0030]
[0031] Where N is the number of samples collected, by minimizing L f Get the optimal parameter φ;
[0032] Through the loss function L of the forward model neural network f Implement training of the forward model neural network.
[0033] In one embodiment of the present invention, the reward neural network is recorded as It consists of a parameter The feedforward neural network is constructed, and the input is the current state s i ,award For output, construct the loss function of the reward neural network, the formula is:
[0034]
[0035] Where N is the number of samples collected;
[0036] By rewarding the loss function L of the neural network r Implement training of reward neural network.
[0037] In one embodiment of the present invention, the axis of the axial actuator is used as a curve model for dynamic analysis, and the standard orthogonal basis {e1, e2, e3} is assumed to be the global coordinate system of the axis, u(s, t) is the position vector of the axis in the inertial system, wherein s represents the arc length and s∈[0, L], L is the length of the axial actuator, and the local coordinate system of the axis is Q={d1(s), d2(s), d3(s)}; in the local coordinate system, {d1(s), d2(s)} constitutes a cross section of a point on the axis, and d3(s) is the normal vector of this cross section. When shearing occurs at an arc length of s, the direction of d3(s) is different from the direction of the tangent of the axis; when the axial actuator is deformed, Q is the rotation matrix; K(s, t) and σ(s, t) represent the angular strain and linear strain of the axis, respectively, Ω(s, t) represents the angular velocity of the axis, and the linear velocity of the axis is e is Young's modulus, ρ is material density, Α is cross-sectional area, B is stiffness matrix for bending and torsion, S is stiffness matrix for extension and shear, I is second area tensor, c is couple density, f is load force, then the dynamic simulation model of axial actuator is expressed as:
[0038] σ(s,t)=Qu s -d2(s);
[0039]
[0040] in, is the force, is the axis, is the tangent line to the axis.
[0041] In order to solve the above technical problems, the present invention provides a pipeline pneumatic soft robot control system based on deep reinforcement learning, comprising:
[0042] Control module: used to control the movement of the axial actuator in the pipeline pneumatic soft robot through the improved DDPG algorithm to achieve control of the pipeline pneumatic soft robot, specifically:
[0043] The improved DDPG algorithm is to introduce a prediction reward learning method into the DDPG algorithm;
[0044] The DDPG algorithm includes an actor network and a critic network, wherein the actor network is used to generate the action of the axial actuator, and the critic network is used to score the action generated by the actor network, calculate the action value function Q, and use the calculated action value function Q as the first reward;
[0045] The prediction reward learning method is implemented by a forward model neural network and a reward neural network, wherein the forward model neural network is used to predict the state generated by the axial actuator performing the next action, and the reward neural network is used to predict the corresponding second reward according to the state generated by the next action;
[0046] Combining the first reward corresponding to the DDPG algorithm and the second reward corresponding to the predicted reward learning method to obtain a strategy function for guiding the axial actuator to perform an action;
[0047] The axial actuator performs actions according to the strategy function to achieve control of the pipeline pneumatic soft robot.
[0048] In order to solve the above technical problems, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned pipeline pneumatic soft robot control method based on deep reinforcement learning are implemented.
[0049] In order to solve the above technical problems, the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned pipeline pneumatic soft robot control method based on deep reinforcement learning are implemented.
[0050] In order to solve the above technical problems, the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned pipeline pneumatic soft robot control method based on deep reinforcement learning.
[0051] The above technical solution of the present invention has the following advantages compared with the prior art:
[0052] The pipeline pneumatic soft robot control method based on deep reinforcement learning described in the present invention can effectively control the pipeline pneumatic soft robot to perform autonomous continuous movement in three-dimensional space, and can control its tip to reach the target point and target direction more quickly and accurately. In addition, the present invention has achieved better results in convergence speed and performance, and improved the continuous motion control performance of the pipeline pneumatic soft robot. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.
[0054] Figure 1 is a flow chart of the method of the present invention;
[0055] Figure 2 is a schematic diagram of a dynamic model of an axial actuator in an embodiment of the present invention;
[0056] Figure 3 is a schematic diagram of the structure of an actor network in an embodiment of the present invention;
[0057] Figure 4 is a schematic diagram of the structure of a critic network in an embodiment of the present invention;
[0058] Figure 5 (a) is a diagram showing the training results of the PR-DDPG algorithm and the DDPG algorithm with different weight factors in an embodiment of the present invention;
[0059] Figure 5 (b) is a distance error band diagram of the PR-DDPG algorithm and the DDPG algorithm in an embodiment of the present invention;
[0060] Figure 6 (a) is a schematic diagram of two control points on the axis of the axial actuator in an embodiment of the present invention;
[0061] Figure 6 (b) is a schematic diagram of three control points on the axis of the axial actuator in an embodiment of the present invention;
[0062] Figure 6 (c) is a schematic diagram of four control points on the axis of the axial actuator in an embodiment of the present invention;
[0063] Figure 6 (d) is a schematic diagram of eight control points evenly distributed on the axis of the axial actuator in an embodiment of the present invention;
[0064] Figure 6 (e) is a schematic diagram of manually setting eight control points on the axis of the axial actuator in an embodiment of the present invention;
[0065] Figure 7 (a) is a diagram showing the training results of the PR-DDPG algorithm for different control points in an embodiment of the present invention;
[0066] Figure 7 (b) is a comparison chart of the test results of the PR-DDPG algorithm and the DDPG algorithm in an embodiment of the present invention. DETAILED DESCRIPTION
[0067] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.
[0068] Embodiment 1
[0069] Reference Figure 1 The present invention relates to a pipeline pneumatic soft robot control method based on deep reinforcement learning, comprising:
[0070] The improved DDPG algorithm (i.e., PR-DDPG control algorithm) is used to control the motion of the axial actuator in the pipeline pneumatic soft robot, thereby realizing the control of the pipeline pneumatic soft robot. Specifically,
[0071] The improved DDPG algorithm is to introduce a prediction reward learning method into the DDPG algorithm;
[0072] The DDPG algorithm includes an actor network and a critic network, wherein the actor network is used to generate the action of the axial actuator, and the critic network is used to score the action generated by the actor network, calculate the action value function Q, and use the calculated action value function Q as the first reward;
[0073] The prediction reward learning method is implemented by a forward model neural network and a reward neural network, wherein the forward model neural network is used to predict the state generated by the axial actuator performing the next action, and the reward neural network is used to predict the corresponding second reward according to the state generated by the next action;
[0074] Combining the first reward corresponding to the DDPG algorithm and the second reward corresponding to the predicted reward learning method to obtain a strategy function for guiding the axial actuator to perform an action;
[0075] The axial actuator performs actions according to the strategy function to achieve control of the pipeline pneumatic soft robot.
[0076] The following is a detailed introduction to this embodiment:
[0077] 1 Axial actuator dynamics simulation model
[0078] The axial actuator of the pipeline pneumatic soft robot can transform material deformation into structural deformation. By adding different chamber air pipes and controlling the pressure loads of different chambers respectively, the pipeline pneumatic soft robot can complete more complex curved motion. Therefore, the Cosserat theory is used to simulate the bending motion of the axial actuator of the pipeline pneumatic soft robot, and the axis of the axial actuator is used as the curve model for dynamic analysis.
[0079] Assume that the standard orthogonal basis {e1, e2, e3} is the global coordinate system of the axis, u(s, t) is the position vector of the axis in the inertial system, where s represents the arc length and s∈[0, L], L is the length of the axial actuator, and the local coordinate system of the axis is Q={d1(s), d2(s), d3(s)}, as Figure 1As shown. In the local coordinate system, {d1(s), d2(s)} constitutes a cross section at a certain point on the axis, and d3(s) is the normal vector of this cross section. When shear occurs at an arc length of s, the direction of d3(s) is different from the direction of the axis tangent. When the axial actuator is deformed, Q is the rotation matrix. K(s,t) and σ(s,t) represent the angular strain (bending and torsion) and linear strain (shear and extension) of the axis, respectively. Ω(s,t) represents the angular velocity of the axis, and the linear velocity of the axis is e is Young's modulus, ρ is material density, Α is cross-sectional area, B is stiffness matrix for bending and torsion, S is stiffness matrix for extension and shear, I is second area tensor, c is couple density, and f is load force. Then the dynamic equation of the axial actuator can be expressed as:
[0080] σ(s,t)=Qu s -d2(s) (1)
[0081]
[0082] in, is the force, is the axis, is the tangent of the axis. According to the driving principle of the pipeline pneumatic soft robot, pressurizing different air chambers of the axial actuator to make it bend can be regarded as Figure 2 At points S1, S2, etc., forces in different directions are applied to its axis.
[0083] 2 Autonomous continuous motion control algorithm based on reinforcement learning algorithm
[0084] 2.1 Deep Deterministic Policy Gradient Control Algorithm Combined with Predicted Rewards
[0085] The DDPG algorithm is a model-free reinforcement learning algorithm, and the state transition probability of the model-free reinforcement learning algorithm is unknown, so the DDPG algorithm will be affected by sample complexity. In order to improve the exploration efficiency of the DDPG algorithm, the predictive reward learning method containing the environment model is integrated into the DDPG algorithm strategy learning, the transition chain generated during the training algorithm iteration process is constructed to build an environmental dynamic model.
[0086] The goal of the predictive reward learning method is to predict the next state reward given the current state and action, and then use the next state reward for subsequent policy learning. The predictive reward technique is approximated by a parameterized forward model neural network and a reward neural network. The forward model neural network is denoted as It is constructed by a feedforward neural network with a parameter of φ. The input is the state s i and action a i , the next moment state is the output. The Adam optimizer is used to learn the neural network parameters. The forward model neural network loss function is as follows:
[0087]
[0088] In formula (4), N is the number of samples collected from the experience playback set D. By minimizing L f Get the optimal parameter φ.
[0089] The reward prediction function is recorded as It is composed of a parameter The feedforward neural network is constructed with the input being the state s i ,award The Adam optimizer is used to learn the network parameters. By minimizing the loss function L r To train the parameters of the reward neural network:
[0090]
[0091] In formula (5), N is the number of samples collected from the experience replay set D. The output of the forward model neural network is used as the input of the reward neural network, and the two neural networks are connected to predict the reward of the next state given the current state and current action. Prediction Reward Used to guide subsequent strategy learning.
[0092] After learning the dynamic model of the environment, the predicted reward technology estimates the predicted reward value of the next state through the reward neural network. The researchers use the learned dynamic model of the environment as additional training data to supplement the training of the algorithm, or provide information to the agent to help it learn the strategy, so as to apply the predicted reward value of the next state to the algorithm strategy learning. As described above, the DDPG algorithm adopts the actor-critic framework. The critic evaluates the action value of the actor and calculates the optimal action value function Q. The actor updates the strategy according to the gradient of the action value function Q to the action. The update of the strategy is only affected by the optimal action value function Q calculated by the critic network. This function is the discounted sum of all rewards from the start state to the end state, which is called the sum reward. This embodiment proposes a new strategy update method: the gradient of the predicted reward (i.e., the second reward) relative to the action is combined with the gradient of the sum reward (i.e., the first reward) relative to the action to guide the strategy update and the action selection, in which the predicted reward technology provides an exploration direction for dynamic perception. The improved DDPG algorithm is called the PR-DDPG control algorithm. Its strategy learning is affected by the mixed objective of maximizing the total reward and the predicted reward. In order to optimize the strategy, a weight factor is introduced to balance the ratio of the total reward and the predicted reward. According to the derivation of the deterministic policy gradient formula, the strategy update method is as follows:
[0093]
[0094] In formula (6) is the gradient of the expected return, N is the number of samples collected, λ is the weight factor and λ∈(0,1), is the gradient of the action, Q() is the Q function, s is the state, a is the action, ω is the parameter of the Q function, s i is the current state of the input, μ(s i ) is the current strategy of the input, is the state corresponding reward of the next action predicted by the prediction reward learning method, φ is the parameter of the feedforward neural network, are the parameters of the reward network, is the gradient of parameter θ, and μ is the deterministic strategy to be optimized.
[0095] 2.2 Markov Decision Process (MDP) Modeling
[0096] In reinforcement learning, an agent interacts with an environment and learns from repeated trials how to choose optimal actions to maximize cumulative rewards. This learning process is modeled using the RL architecture of Markov decision processes (MDPs).
[0097] The state space, action space, and reward function of the axial actuator of the pipeline pneumatic soft robot are set as follows:
[0098] (1) State space: Combined with the bending effect of the pipeline pneumatic soft robot in the finite element analysis, four control points on the axis of the axial actuator are taken. According to the bending motion mode of the pipeline pneumatic soft robot, the end of the axial actuator is fixed and the position of the control point is expressed by vector X. a Indicates that the position of the target point is represented by vector X t Indicated by V a and V t Describe the magnitude and direction of the velocity of the axial actuator tip and the target, respectively, with q a and q t To describe the coordinate system orientation of the axial actuator tip and the target, the state space can be described as S = [X a , V a ,q a , X t , V t ,q t ], with a total of 24 dimensions.
[0099] (2) Motion space: According to the mechanical analysis of the pipeline pneumatic soft robot structure deformation, it can be known that by applying loads to different cavity walls, the bending motion of the pipeline pneumatic soft robot axial actuator is realized, and within a certain angle bending range, the inner length of the pipeline pneumatic soft robot axial actuator remains basically unchanged. Similarly, four control points on the axis are taken to apply torque A in the directions of d1(s), d2(s), and d3(s) on the axis. m (s) = α m β m (s) to realize the bending motion of the axial actuator of the pipeline pneumatic soft robot, thus having 12 degrees of freedom. m is the maximum torque, β m The selection range of (s) is [-1, 1].
[0100] (3) Reward function: The reward function is a reward agent that can effectively evaluate the subject's actions. The distance between the tip position of the axial actuator of the pipeline pneumatic soft robot and the target point is defined as the distance error. The reward function is defined as follows:
[0101]
[0102] In formula (7), n = ||X n -X t || represents the tip X of the axial actuator of the pipeline soft robot n With the target point X t The distance between them, p is the direction penalty, and d=6mm is the reward range distance, so that its tip keeps approaching the target.
[0103] 2.3 Network structure and hyperparameters
[0104] The relevant hyperparameters are set as follows: the learning rate of the actor network is 0.0001, the learning rate of the critic network is 0.001, the learning rate of the environment model and the reward neural network is 0.001, the discount factor is 0.99, the batch size N is 128, the experience replay set capacity is 2000000, the number of hidden layers of the actor network and the number of hidden layers of the critic network are 2, and the soft update parameter of the target network is 0.001. The dual neural network architecture is adopted to combine the value evaluation ability of the critic network with the real-time control ability of the actor network to synergistically optimize the learning effect. The network structure of the actor network and the critic is as follows Figure 3 , Figure 4 As shown in the figure, the actor network (including input layer, hidden layer, and output layer) is composed of a 24×400×300×12 fully connected neural network (the input layer has 24 neurons, the hidden layer includes two columns, the first column has 400 neurons, the second column has 300 neurons, and the output layer has 12 neurons), and the critic network (including input layer, hidden layer, and output layer) has a structure of 36×400×300×1 (the input layer has 36 neurons, the hidden layer includes two columns, the first column has 400 neurons, the second column has 300 neurons, and the output layer has 1 neuron). The structure of the forward model network is 24×400×300×12, while the reward network has only one hidden layer with 400 nodes and a structure of 24×400×1. The observed state is normalized into a 24-dimensional input to the actor, and the 12-dimensional output action is used to control the motion of the axial actuator.
[0105] 3 Simulation Experiment
[0106] 3.1 Experimental platform setup
[0107] The controller of the pipeline pneumatic soft robot will gradually learn strategies during the control process. During this training process, the amount of data required by the controller of the pipeline pneumatic soft robot is very large, and it takes a lot of time to collect this data in real life. In order to reduce the training cost, a simple model of the axial actuator of the pipeline pneumatic soft robot is created in the simulation environment, and it is controlled to perform autonomous motion control training, and the effect after training is simulated and tested.
[0108] This experiment uses the simulation environment Elastica, which can solve soft mechanics problems. This environment takes into account the physical characteristics of soft robots, and can use high-order continuous elastic effects and their related dynamics to give full play to the ability of soft robots to deform patterns. It can be used for distributed drive and control in continuous state action space. Elastica allows users to use a complete control library and define control tasks, variables, drive modes, and physical environments. Therefore, Elastica can be coupled with RL algorithms to complete the simulation of distributed mechanical control strategies and realize dynamic control of soft robots based on simulation learning.
[0109] The model parameters of the pipeline pneumatic soft robot axial actuator in the simulation environment are set as follows: the initial length L of the axial actuator is 100 mm, the radius d of the axial actuator is 6 mm, the elastic modulus E is 8 MPa, the density ρ is 1000 kg / m3, and the maximum torque α is m is 5 N·m, and the control time step t is 1.4 ms. The axial actuator is modeled as a cosserat beam with its bottom fixed, standing vertically in three-dimensional space.
[0110] 3.2 Experiment of reaching the target point
[0111] The first experiment is to reach the target point. In order to optimize the PR-DDPG control algorithm and study the influence of the weight factor on the performance of the PR-DDPG control algorithm, different weights of the weight factor in the PR-DDPG control algorithm are used to control the pipeline pneumatic soft robot to complete the motion task of reaching the target point through different PR-DDPG algorithms and DDPG algorithms, and the experimental results of the PR-DDPG control algorithm and the DDPG algorithm are compared and analyzed.
[0112] In order to optimize the PR-DDPG control algorithm and evaluate the impact of the proportional change of the introduction of the prediction reward technology on the performance of the PR-DDPG control algorithm, that is, to compare the impact of the weight factor on the performance of the PR-DDPG control algorithm, four different weights were selected: PR-DDPG-λ=0.2, PR-DDPG-λ=0.6, PR-DDPG-λ=0.8 and PR-DDPG-λ=1, and four different PR-DDPG algorithms were obtained, which were trained separately with the DDPG algorithm in the previous article.
[0113] After all algorithms completed 1000 steps of training per round and a total of 5000 rounds of training, the learning curves obtained are as follows Figure 5(a). The solid line represents the average reward of each algorithm round, and the shaded area is the standard deviation. The algorithm PR-DDPG-λ=1 cannot converge because its policy gradient depends only on the predicted reward, while other algorithms can find a control strategy to touch the target point. The reward of the algorithm PR-DDPG-λ=0.2 is similar to that of the DDPG algorithm, but the convergence speed is slower. The performance of the algorithm PR-DDPG-λ=0.6 is slightly better than that of the DDPG algorithm, but the difference between the two in the learning curve is not significant. The algorithm PR-DDPG-λ=0.8 tends to converge after about 2200 rounds of training, while the DDPG algorithm tends to converge after about 3700 rounds of training. The convergence speed of the algorithm PR-DDPG-λ=0.8 is faster, and its performance is significantly better than that of the DDPG algorithm.
[0114] From the learning curve, we can see that the larger the weight of the predicted reward, the faster the algorithm converges. When the weight is too low, the predicted reward part has no obvious effect on the dynamic perception of the environment. However, when the weight factor is 1, the algorithm's strategy learning only depends on the predicted reward. Without the guidance of the total reward, the algorithm cannot converge. Therefore, when the weight factor is too low or set to 1, the PR-DDPG control algorithm is inferior to the DDPG algorithm. When the weight factor is 0.8, the PR-DDPG control algorithm performs better than the DDPG algorithm.
[0115] As learning progresses, the noise signal decreases, and the axial actuator of the pipeline pneumatic soft robot is able to achieve new movements based on its learned experience. The training model tends to be stable, the reward function tends to converge, the control strategy to reach the target is completed, and the controller has performed good learning on the target area in the workspace.
[0116] The trained algorithm PR-DDPG-λ=0.8 was tested in the same way as the DDPG algorithm. After completing 10 test sets, the mean and standard deviation of the error values of different test sets were calculated, and an error band diagram was drawn. The test results of the PR-DDPG control algorithm were compared with those of the DDPG algorithm, as shown in Figure 2. Figure 5 (b) It can be seen that compared with the DDPG algorithm, the tip of the axial actuator of the pipeline soft robot controlled by the PR-DDPG algorithm can autonomously reach the target point in a shorter time and with a smaller distance error.
[0117] 3.3 Experiment of reaching the target point in the target direction
[0118] The second experiment is to reach the target point in the target direction. Taking into account the application scenarios of the pipeline pneumatic soft robot designed in this embodiment in the field of industrial inspection, and the carrier of the inspection equipment is the tip of the pipeline pneumatic soft robot, quickly and autonomously reaching the target direction can improve the detection efficiency of the pipeline pneumatic soft robot, and it is more advantageous when performing pipeline inspection tasks. Therefore, the second experimental task is to control the tip of the axial actuator of the pipeline pneumatic soft robot to reach the target position and target direction in three-dimensional space. The experiment in this section defines the target direction as the tangent direction of the tip of the axial actuator should be vertically upward, and the normal-binormal vector is randomly rotated from the global coordinate system in the range of -90° to 90°.
[0119] In order to effectively evaluate the action of the axial actuator to reach the target point in the target direction, a direction penalty is added to the reward function. The reward function optimized for the experimental task is defined as follows:
[0120]
[0121] In formula (8), d = 6 mm, p = 1-(q a ·q t ) 2 is the direction penalty, which causes the axial actuator tip to rotate to the target direction, q a is the tip direction of the axial actuator of the pipeline soft robot, q t is the direction of the target point.
[0122] In the previous article, the controller learning process was modeled, in which the state space and action space are set according to the control points. The setting of the control points plays a vital role in controlling the axial actuator. The PR-DDPG control algorithm can be optimized by changing the control point settings to control the axial actuator to complete more complex motion tasks. When the number of control points changes, the state space dimension and action space degrees of freedom of the axial actuator change. The network structure of the actor and critic also changes accordingly, as shown in Table 1.
[0123] Table 1 Network structure corresponding to the number of control points
[0124]
[0125] In order to study the influence of control point setting on the PR-DDPG control algorithm, control points of different numbers or positions were taken on the axis of the axial actuator of the pipeline pneumatic soft robot, and the PR-DDPG control algorithm was used to perform the same experimental task. According to the previous article, PR-DDPG-λ=0.8 was taken.
[0126] Simulate different control point settings, such as Figure 6As shown in Figure 2, 3, 4 and 8 uniformly distributed control points and 8 manually set control points training strategies (corresponding to Figure 6 (a) Figure 6 (b) Figure 6 (c) Figure 6 (d) Figure 6 (e)), the corresponding network structures of different PR-DDPG control algorithms are shown in Table 1.
[0127] All algorithms completed 1000 steps of training per round, a total of 5000 rounds of training, the training results are as follows Figure 7 (a) shown. Figure 7 The results in (a) show that the learning performance of 2 and 8 control points is poor. This is because 2 equidistant control points do not provide enough control freedom to reach the goal, while 8 control points have a larger action space, but their extensive exploration leads to slow learning. 3 and 4 control points perform better than 2 and 8 control points because their action space is smaller than 8 control points and their control freedom is higher than 2 control points. Among them, the best performance is 4 control points, which shows that it has achieved a balance between sufficient motivation to complete the task and a small enough action space for rapid exploration. 3 control points also performed well, and its learning performance in 2000 to 2650 rounds was better than that of 4 control points, but the increased controllability of 4 control points allowed it to obtain a higher reward value in the end, indicating that setting 3 control points in simple tasks can improve exploration efficiency.
[0128] In addition, among the 8 manually placed control points and the 8 evenly distributed control points, the 8 manually placed control points perform better because the manually set control point positions can better utilize the compliance of the axial actuator. Unlike the evenly distributed, the control points are manually set at the tip of the axial actuator and at a position close to the middle of the axial actuator from the tip. The control points are set at the tip to control its steering when moving and achieve more precise tip control. The control points are set near the middle to make it produce a large bend, thereby controlling the total deformation of the axial actuator.
[0129] In summary, fewer control points make the action space exploration faster, resulting in more effective sample learning, but too few control points will reduce the controllability and make it impossible to complete the task. Therefore, it is necessary to maximize the controllability of the axial actuator according to the task situation and set as few control points as possible to strike a balance between controllability and exploration speed. For this experimental task, select Figure 6 The four control points shown in (c) make the PR-DDPG control algorithm have the best learning performance.
[0130] After the same training with the DDPG algorithm with 4 control points, 10 test sets were completed with the PR-DDPG algorithm. The mean and standard deviation of the reward values of different test sets were calculated, and the reward values of the PR-DDPG algorithm were compared with those of the DDPG algorithm. Figure 7 (b) After learning, the PR-DDPG algorithm can converge faster, and the reward value obtained is higher than the reward value of the DDPG algorithm, that is, the PR-DDPG algorithm has higher control performance.
[0131] Embodiment 3
[0132] This embodiment provides a pipeline pneumatic soft robot control system based on deep reinforcement learning, including:
[0133] Control module: used to control the movement of the axial actuator in the pipeline pneumatic soft robot through the improved DDPG algorithm to achieve control of the pipeline pneumatic soft robot, specifically:
[0134] The improved DDPG algorithm is to introduce a prediction reward learning method into the DDPG algorithm;
[0135] The DDPG algorithm includes an actor network and a critic network, wherein the actor network is used to generate the action of the axial actuator, and the critic network is used to score the action generated by the actor network, calculate the action value function Q, and use the calculated action value function Q as the first reward;
[0136] The prediction reward learning method is implemented by a forward model neural network and a reward neural network, wherein the forward model neural network is used to predict the state generated by the axial actuator performing the next action, and the reward neural network is used to predict the corresponding second reward according to the state generated by the next action;
[0137] Combining the first reward corresponding to the DDPG algorithm and the second reward corresponding to the predicted reward learning method to obtain a strategy function for guiding the axial actuator to perform an action;
[0138] The axial actuator performs actions according to the strategy function to achieve control of the pipeline pneumatic soft robot.
[0139] Embodiment 3
[0140] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the pipeline pneumatic soft robot control method based on deep reinforcement learning described in Embodiment 1 are implemented.
[0141] Embodiment 4
[0142] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the pipeline pneumatic soft robot control method based on deep reinforcement learning described in Embodiment 1 are implemented.
[0143] Embodiment 4
[0144] This embodiment provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the pipeline pneumatic soft robot control method based on deep reinforcement learning described in Example 1 are implemented.
[0145] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.
[0146] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0147] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0149] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0150] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.
Claims
1. A pipeline pneumatic soft robot control method based on deep reinforcement learning, characterized in that: include: The improved DDPG algorithm is used to control the movement of the axial actuator in the pipeline pneumatic soft robot to achieve control of the pipeline pneumatic soft robot. Specifically: The improved DDPG algorithm is to introduce a prediction reward learning method into the DDPG algorithm; The DDPG algorithm includes an actor network and a critic network, wherein the actor network is used to generate the action of the axial actuator, and the critic network is used to score the action generated by the actor network, calculate the action value function Q, and use the calculated action value function Q as the first reward; The prediction reward learning method is implemented by a forward model neural network and a reward neural network, wherein the forward model neural network is used to predict the state generated by the axial actuator performing the next action, and the reward neural network is used to predict the corresponding second reward according to the state generated by the next action; Combining the first reward corresponding to the DDPG algorithm and the second reward corresponding to the predicted reward learning method to obtain a strategy function for guiding the axial actuator to perform an action; The axial actuator performs actions according to the strategy function to achieve control of the pipeline pneumatic soft robot.
2. The pipeline pneumatic soft robot control method based on deep reinforcement learning according to claim 1 is characterized in that: The first reward corresponding to the DDPG algorithm and the second reward corresponding to the prediction reward learning method are combined to obtain a strategy function for guiding the axial actuator to perform an action, and the formula is: in, is the gradient of the expected return, N is the number of samples collected, λ is the weight factor, is the gradient of the action, Q( ) is the Q function, s is the state, a is the action, ω is the parameter of the Q function, s i is the current state of the input, μ(s i ) is the current strategy of the input, is the state corresponding reward of the next action predicted by the prediction reward learning method, φ is the parameter of the feedforward neural network and the feedforward neural network is used as the forward model neural network, are the parameters of the reward network, is the gradient of parameter θ, and μ is the deterministic strategy to be optimized.
3. The pipeline pneumatic soft robot control method based on deep reinforcement learning according to claim 1 is characterized in that: The learning process of the axial actuator of the pipeline pneumatic soft robot is modeled by using the Markov decision process MDP, including: The state space, action space, and reward function of the axial actuator of the pipeline pneumatic soft robot are set as follows: The state space: Combined with the bending effect of the pipeline pneumatic soft robot in the finite element analysis, four evenly distributed control points are taken on the axis of the axial actuator. According to the bending motion mode of the pipeline pneumatic soft robot, the end of the axial actuator is fixed, and the position of the control point is expressed by vector X a Indicates that the position of the target point is represented by vector X t Indicated by V a and V t Describe the magnitude and direction of the velocity of the axial actuator tip and the target, respectively, with q a and q t To describe the coordinate system orientation of the axial actuator tip and the target, the state space can be described as S = [X a , V a ,q a , X t , V t ,q t ]; The action space: According to the mechanical analysis of the pipeline pneumatic soft robot structure deformation, loads are applied to different cavity walls to realize the bending movement of the pipeline pneumatic soft robot axial actuator, and within a certain angle bending range, the inner length of the pipeline pneumatic soft robot axial actuator remains basically unchanged. Similarly, four evenly distributed control points on the axis are taken to apply torque A in the directions of d1(s), d2(s) and d3(s) on the axis. m (s) = α m β m (s), to realize the bending motion of the axial actuator of the pipeline pneumatic soft robot, α m is the maximum torque, β m The selection range of (s) is [-1, 1]; The reward function is: the distance between the tip position of the axial actuator of the pipeline pneumatic soft robot and the target point is defined as the distance error, which is expressed as: Where n = ||X n -X t || represents the tip X of the axial actuator of the pipeline soft robot n With the target point X t The distance between them, p is the direction penalty, and d is the reward range distance.
4. The pipeline pneumatic soft robot control method based on deep reinforcement learning according to claim 1 is characterized in that: The forward model neural network is denoted as It is constructed by a feedforward neural network with parameter φ, assuming that the input is the current state s i and the current action a i , the next moment state As the output, construct the loss function of the forward model neural network, the formula is: Where N is the number of samples collected, by minimizing L f Get the optimal parameter φ; Through the loss function L of the forward model neural network f Implement training of the forward model neural network.
5. The pipeline pneumatic soft robot control method based on deep reinforcement learning according to claim 1 is characterized in that: The reward neural network is denoted as It consists of a parameter The feedforward neural network is constructed, and the input is the current state s i ,award For output, construct the loss function of the reward neural network, the formula is: Where, N is the number of samples collected; By rewarding the loss function L of the neural network r Implement training of reward neural network.
6. The pipeline pneumatic soft robot control method based on deep reinforcement learning according to claim 1 is characterized in that: The axis of the axial actuator is taken as a curve model for dynamic analysis. The standard orthogonal basis {e1, e2, e3} is assumed to be the global coordinate system of the axis, u(s, t) is the position vector of the axis in the inertial system, where s represents the arc length and s∈[0, L], L is the length of the axial actuator, and the local coordinate system of the axis is Q={d1(s), d2(s), d3(s)}; in the local coordinate system, {d1(s), d2(s)} constitutes a cross section at a point on the axis, and d3(s) is the normal vector of this cross section. When shear occurs at the arc length s, the direction of d3(s) is different from the direction of the axis tangent; when the axial actuator is deformed, Q is the rotation matrix; K(s, t) and σ(s, t) represent the angular strain and linear strain of the axis, respectively, Ω(s, t) represents the angular velocity of the axis, and the linear velocity of the axis is e is Young's modulus, ρ is material density, Α is cross-sectional area, B is stiffness matrix for bending and torsion, S is stiffness matrix for extension and shear, I is second area tensor, c is couple density, f is load force, then the dynamic simulation model of axial actuator is expressed as: σ(s,t)=Qu s -d2(s); in, is the force, is the axis, is the tangent line to the axis.
7. A pipeline pneumatic soft robot control system based on deep reinforcement learning, characterized in that: include: Control module: used to control the movement of the axial actuator in the pipeline pneumatic soft robot through the improved DDPG algorithm to achieve control of the pipeline pneumatic soft robot, specifically: The improved DDPG algorithm is to introduce a prediction reward learning method into the DDPG algorithm; The DDPG algorithm includes an actor network and a critic network, wherein the actor network is used to generate the action of the axial actuator, and the critic network is used to score the action generated by the actor network, calculate the action value function Q, and use the calculated action value function Q as the first reward; The prediction reward learning method is implemented by a forward model neural network and a reward neural network, wherein the forward model neural network is used to predict the state generated by the axial actuator performing the next action, and the reward neural network is used to predict the corresponding second reward according to the state generated by the next action; Combining the first reward corresponding to the DDPG algorithm and the second reward corresponding to the predicted reward learning method to obtain a strategy function for guiding the axial actuator to perform an action; The axial actuator performs actions according to the strategy function to achieve control of the pipeline pneumatic soft robot.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the pipeline pneumatic soft robot control method based on deep reinforcement learning as described in any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the pipeline pneumatic soft robot control method based on deep reinforcement learning as described in any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the pipeline pneumatic soft robot control method based on deep reinforcement learning as described in any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Robot control method based on offline model pre-training learning DDPG algorithm
CN112668235A
Mechanical arm grabbing control method of deep reinforcement learning DDPG algorithm based on visual information
CN115464659A
Continuous body mechanical arm motion control method based on deep reinforcement learning
CN116038691A
Method for predicting remaining life of cross-domain rolling bearing
CN118885793A
Systems and methods for learning reusable options to transfer knowledge between tasks
US20210387330A1
