Control method and system of pipeline pneumatic soft robot based on deep reinforcement learning

Abstract

This invention relates to a control method and system for a pipeline pneumatic soft robot based on deep reinforcement learning. The method includes: controlling the motion of an axial actuator in the pipeline pneumatic soft robot using an improved DDPG algorithm to achieve control of the robot. Specifically, the improved DDPG algorithm incorporates a predictive reward learning method; a first reward is obtained through the DDPG algorithm; a second reward is obtained through the predictive reward learning method; the first reward and the second reward are combined to obtain a policy function that guides the axial actuator to perform actions; the axial actuator performs actions according to the policy function to achieve control of the pipeline pneumatic soft robot. This invention effectively improves the continuous motion control performance of the pipeline pneumatic soft robot.

Description

Technical Field

[0001] This invention relates to the field of pipeline pneumatic soft robot control technology, and in particular to a pipeline pneumatic soft robot control method and system based on deep reinforcement learning. Background Technology

[0002] When designing robot controllers, they can be divided into model-based control strategies and model-free control strategies.

[0003] Model-based control methods can improve the predictability and adaptability of the overall control system. The piecewise constant curvature (PCC) assumption can be used to build models, relying on the assumption that each part of the robot deforms along a circumferential arc and that there is no gravity. This assumption is generally suitable for hyperredundant or rigid continuous robots. Unlike traditional rigid robots with precise analytical kinematics that have been extensively studied for many years, soft robots are mostly composed of continuously deformable materials or mechanisms, possessing more degrees of freedom and high compliance. The PCC assumption is not suitable for the precise control of soft robots. Finite element-based models, however, do not rely on such stringent assumptions as PCC and have been proven suitable for modeling nonlinear soft materials and their interactions with the environment. Wu et al. proposed a finite element-based nonlinear controller for cable-driven soft robots, considering robot hysteresis and friction in the model and achieving good disturbance suppression. However, the huge computational cost prevents the above finite element model from being used in real-time closed-loop control.

[0004] Therefore, soft robots are subject to modeling uncertainties due to internal and external factors, such as nonlinear deformation caused by actuation, material elasticity and interaction with the environment. This poses a significant challenge to accurate analytical modeling, which may reduce the effectiveness of model-based control methods and make it difficult to achieve real-time and fast control effects. Modeling strategies need to make difficult-to-optimize trade-offs between the flexibility and controllability of soft robots.

[0005] Model-free control strategies are independent of the robot's shape and do not require defining parameters in the configuration space or joint space. Due to abundant sample data and sensory noise, complex kinematic models can be created. For highly nonlinear, non-uniform, gravity-affected, or unstructured control systems, model-free control strategies may perform better.

[0006] In model-free control strategies, the Jacobian matrix can be used to directly train neural networks from real data. Lu et al. proposed a data-driven method for 3D shape servo control of soft robots based on proprioceptive feedback. This method first obtains the initial estimate of the Jacobian matrix offline, and then continuously estimates the unknown model of the soft robot online as the robot moves in the workspace to cope with unexpected disturbances and exhibit adaptive behavior for unmodeled systems. However, the above method does not consider the dynamics of the soft robot and is more suitable for the static control of pipeline soft robots.

[0007] Compared to methods that estimate the Jacobian matrix, model-free control strategies based on machine learning (ML) techniques are more widely used. Almanzor et al. proposed an image-based soft robot shape control method that combines local inverse kinematics formulas in image space with convolutional neural networks to achieve precise shape control, but its control loop still operates at a relatively slow speed.

[0008] Reinforcement learning (RL) is a branch of machine learning and a general framework for designing controllers for nonlinear systems. It learns optimal policies directly from experience, eliminating the need to select the best action based on a separate control policy during state transitions. Furthermore, control policies obtained using reinforcement learning are more robust to external disturbances, providing a feasible method for handling complex situations and completing complex tasks. For example, Centurelli et al. used RL to control soft robots to operate under different payload conditions. Because it eliminates prior knowledge about the robot's structure, reinforcement learning has received widespread attention in the field of robot control.

[0009] Traditional reinforcement learning is better suited for discrete states, which differs from the practical applications of pipe-driven soft robots. To handle continuous state spaces, deep neural networks are introduced into reinforcement learning techniques, resulting in Deep Reinforcement Learning (DRL). Yao et al. simulated the dynamics of a magnetic soft robot using a Cosserat model, employing a deep reinforcement learning algorithm to adapt the robot to different magnetization modes and magnetic field constraints, and fitting the results with experimental data to accelerate training. Wu et al. combined data-driven modeling methods with deep Q-network algorithms to achieve motion control of the end effector of a cable-driven soft robot. However, the value function-based deep Q-network algorithm updates the policy by evaluating action value functions; that is, the algorithm needs to calculate the value of all actions in the next time step in the current state and select the optimal action value. When the action space is continuous, the computational cost is very high, which also limits this algorithm to discrete motion control of soft robots.

[0010] In summary, existing methods have not achieved satisfactory continuous control performance for pipeline pneumatic soft robots, and it is necessary to overcome the limitations of traditional control strategies. Summary of the Invention

[0011] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the continuous control performance of pipeline pneumatic soft robots in the prior art has not reached a good state.

[0012] To address the aforementioned technical problems, this invention provides a control method for a pipeline pneumatic soft robot based on deep reinforcement learning, comprising:

[0013] The improved DDPG algorithm is used to control the movement of the axial actuator in the pipeline pneumatic soft robot, thereby achieving control of the pipeline pneumatic soft robot. Specifically:

[0014] The improved DDPG algorithm introduces a predictive reward learning method into the DDPG algorithm;

[0015] The DDPG algorithm includes an actor network and a critic network. The actor network is used to generate the action of the axial actuator, and the critic network is used to score the action generated by the actor network and calculate the action value function Q. The calculated action value function Q is used as the first reward.

[0016] The prediction reward learning method is implemented through a feedforward model neural network and a reward neural network. The feedforward model neural network is used to predict the state generated by the axial actuator performing the next action, and the reward neural network is used to predict the corresponding second reward based on the state generated by the next action.

[0017] The first reward corresponding to the DDPG algorithm and the second reward corresponding to the prediction reward learning method are combined to obtain a policy function for guiding the axial actuator to perform actions.

[0018] The axial actuator performs actions according to the strategy function to achieve control of the pipeline pneumatic soft robot.

[0019] In one embodiment of the present invention, the first reward corresponding to the DDPG algorithm and the second reward corresponding to the prediction reward learning method are combined to obtain a policy function for guiding the axial actuator to perform actions, the formula of which is:

[0020]

[0021] in, Let λ be the gradient of the expected return, N be the number of samples collected, and λ be the weighting factor. Let be the gradient of the action, Q() be the Q-function, s be the state, a be the action, and ω be the parameters of the Q-function. i Given the current input state, μ(s)i () represents the current policy as input. To predict the reward corresponding to the state generated by the next action predicted by the reward learning method, φ is the parameter of the feedforward neural network, and this feedforward neural network is used as the forward model neural network. To reward the network parameters, Let θ be the gradient of the parameter θ, and μ be the deterministic policy to be optimized.

[0022] In one embodiment of the present invention, the learning process of the axial actuator of a pipeline pneumatic soft robot is modeled using a Markov decision process (MDP), including:

[0023] The state space, motion space, and reward function of the axial actuator of the pipeline pneumatic soft robot are set as follows:

[0024] The state space is defined as follows: Based on the bending effect of the pneumatic soft robot in finite element analysis, four evenly distributed control points are selected along the axis of the axial actuator. According to the bending motion mode of the pneumatic soft robot, the end of the axial actuator is fixed, and the positions of the control points are represented by vector X. a The position of the target point is represented by vector X. t Indicated by V a and V t Describe the magnitude and direction of the axial actuator tip and the target velocity, respectively, using q. a and q t To describe the coordinate system orientation between the axial actuator tip and the target, the state space can be described as S = [X...]. a V a q a X t V t q t ];

[0025] The action space is defined as follows: Based on the mechanical analysis of the deformation of the pneumatic soft robot structure, loads are applied to different cavity walls to achieve the bending motion of the axial actuator of the pneumatic soft robot. Within a certain bending angle range, the inner length of the axial actuator of the pneumatic soft robot remains basically unchanged. Similarly, four evenly distributed control points are taken on the axis, and a torque A is applied in the directions d1(s), d2(s), and d3(s) on the axis. m (s)=α m β m (s) is used to realize the bending motion of the axial actuator of the pipeline pneumatic soft robot, α m For the maximum torque, β m The range of selection for (s) is [-1, 1];

[0026] The reward function is defined as follows: the distance error is the distance between the tip of the axial actuator of the pipeline pneumatic soft robot and the target point, expressed as:

[0027]

[0028] Where, n = ||X n -X t || Represents the tip X of the axial actuator of the pipeline soft robot. n With target point X t The distance between them, p is the directional penalty, and d is the reward range distance.

[0029] In one embodiment of the present invention, the feedforward model neural network is denoted as... It is constructed from a feedforward neural network with parameters φ, where the input is the current state s. i and current action a i The state at the next moment For the output, construct the loss function of the feedforward neural network, as shown in the formula:

[0030]

[0031] Where N is the number of samples collected, and L is minimized. f Obtain the optimal parameter φ;

[0032] The loss function L of the feedforward neural network f To train a feedforward neural network.

[0033] In one embodiment of the present invention, the reward neural network is denoted as... It consists of one parameter. The feedforward neural network is constructed, and the input is the current state s. i ,award To output the reward neural network, construct the loss function as follows:

[0034]

[0035] Where N is the number of samples collected;

[0036] The loss function L of the reward neural network r To train a reward neural network.

[0037] In one embodiment of the present invention, the axis of the axial actuator is used as a curve model for dynamic analysis. The orthonormal basis {e1, e2, e3} is set as the global coordinate system of the axis, and u(s,t) is the position vector of the axis of the inertial frame, where s represents the arc length and s∈[0,L], and L is the length of the axial actuator. The local coordinate system of the axis is Q={d1(s), d2(s), d3(s)}. In the local coordinate system, {d1(s), d2(s)} constitute a cross-section at a point on the axis, and d3(s) is the normal vector of this cross-section. When shearing occurs at arc length s, the direction of d3(s) is different from the direction of the tangent to the axis. When the axial actuator deforms, Q is the rotation matrix. K(s,t) and σ(s,t) represent the angular strain and linear strain of the axis, respectively, and Ω(s,t) represents the angular velocity of the axis. The linear velocity of the axis is... Let e ​​be Young's modulus, ρ be material density, A be cross-sectional area, B be the stiffness matrix for bending and torsion, S be the stiffness matrix for extension and shear, I be the second-region tensor, c be couple density, and f be the load force. Then, the dynamic simulation model of the axial actuator can be expressed as:

[0038] σ(s,t)=Qu s -d2(s);

[0039]

[0040] in, For the action force, As the axis, It is the tangent to the axis.

[0041] To address the aforementioned technical problems, this invention provides a pipeline pneumatic soft robot control system based on deep reinforcement learning, comprising:

[0042] Control module: Used to control the movement of the axial actuator in the pipeline pneumatic soft robot through an improved DDPG algorithm, thereby achieving control of the pipeline pneumatic soft robot. Specifically:

[0043] The improved DDPG algorithm introduces a predictive reward learning method into the DDPG algorithm;

[0044] The DDPG algorithm includes an actor network and a critic network. The actor network is used to generate the action of the axial actuator, and the critic network is used to score the action generated by the actor network and calculate the action value function Q. The calculated action value function Q is used as the first reward.

[0045] The prediction reward learning method is implemented through a feedforward model neural network and a reward neural network. The feedforward model neural network is used to predict the state generated by the axial actuator performing the next action, and the reward neural network is used to predict the corresponding second reward based on the state generated by the next action.

[0046] The first reward corresponding to the DDPG algorithm and the second reward corresponding to the prediction reward learning method are combined to obtain a policy function for guiding the axial actuator to perform actions.

[0047] The axial actuator performs actions according to the strategy function to achieve control of the pipeline pneumatic soft robot.

[0048] To address the aforementioned technical problems, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned deep reinforcement learning-based pipeline pneumatic soft robot control method.

[0049] To address the aforementioned technical problems, the present invention provides a computer-readable storage medium storing a computer program thereon, wherein when the computer program is executed by a processor, it implements the steps of the aforementioned deep reinforcement learning-based pipeline pneumatic soft robot control method.

[0050] To address the aforementioned technical problems, this invention provides a computer program product, comprising a computer program that, when executed by a processor, implements the steps of the aforementioned deep reinforcement learning-based pipeline pneumatic soft robot control method.

[0051] The technical solution of the present invention has the following advantages compared with the prior art:

[0052] The deep reinforcement learning-based control method for pipeline pneumatic soft robots described in this invention can effectively control the pipeline pneumatic soft robot to perform autonomous continuous motion in three-dimensional space, and can control its tip to reach the target point and target direction relatively quickly and accurately. Furthermore, this invention achieves better results in terms of convergence speed and performance, thus improving the continuous motion control performance of the pipeline pneumatic soft robot. Attached Figure Description

[0053] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0054] Figure 1 This is a flowchart of the method of the present invention;

[0055] Figure 2 This is a schematic diagram of the axial actuator dynamics model in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the actor network structure in an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the structure of the critic network in an embodiment of the present invention;

[0058] Figure 5 (a) is a diagram showing the training results of the PR-DDPG algorithm and the DDPG algorithm with different weight factors in the embodiments of the present invention;

[0059] Figure 5 (b) is a distance error band diagram of the PR-DDPG algorithm and the DDPG algorithm in the embodiments of the present invention;

[0060] Figure 6 (a) is a schematic diagram of two control points on the axis of the axial actuator in an embodiment of the present invention;

[0061] Figure 6 (b) is a schematic diagram of three control points on the axis of the axial actuator in an embodiment of the present invention;

[0062] Figure 6 (c) is a schematic diagram of four control points on the axis of the axial actuator in an embodiment of the present invention;

[0063] Figure 6 (d) is a schematic diagram of eight control points evenly distributed on the axis of the axial actuator in an embodiment of the present invention;

[0064] Figure 6 (e) is a schematic diagram of manually setting 8 control points on the axis of the axial actuator in an embodiment of the present invention;

[0065] Figure 7 (a) is a graph showing the training results of the PR-DDPG algorithm at different control points in an embodiment of the present invention;

[0066] Figure 7 (b) is a comparison chart of the test results of the PR-DDPG algorithm and the DDPG algorithm in the embodiments of the present invention. Detailed Implementation

[0067] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0068] Example 1

[0069] Reference Figure 1 As shown, this invention relates to a control method for a pipeline pneumatic soft robot based on deep reinforcement learning, comprising:

[0070] The motion of the axial actuator in the pipeline pneumatic soft robot is controlled by an improved DDPG algorithm (i.e., PR-DDPG control algorithm), thereby achieving control of the pipeline pneumatic soft robot. Specifically:

[0071] The improved DDPG algorithm introduces a predictive reward learning method into the DDPG algorithm;

[0072] The DDPG algorithm includes an actor network and a critic network. The actor network is used to generate the action of the axial actuator, and the critic network is used to score the action generated by the actor network and calculate the action value function Q. The calculated action value function Q is used as the first reward.

[0073] The prediction reward learning method is implemented through a feedforward model neural network and a reward neural network. The feedforward model neural network is used to predict the state generated by the axial actuator performing the next action, and the reward neural network is used to predict the corresponding second reward based on the state generated by the next action.

[0074] The first reward corresponding to the DDPG algorithm and the second reward corresponding to the prediction reward learning method are combined to obtain a policy function for guiding the axial actuator to perform actions.

[0075] The axial actuator performs actions according to the strategy function to achieve control of the pipeline pneumatic soft robot.

[0076] The following is a detailed description of this embodiment:

[0077] 1. Axial actuator dynamics simulation model

[0078] The axial actuator of a pneumatic soft robot in a pipeline can transform material deformation into structural deformation. By adding air ducts with different chambers and controlling the pressure load in each chamber, the pneumatic soft robot in a pipeline can perform more complex curved motions. Therefore, the bending motion of the axial actuator of the pneumatic soft robot in a pipeline is simulated using Cosserat theory, and the dynamics are analyzed using the axis of the axial actuator as a curved model.

[0079] Let the orthonormal basis {e1, e2, e3} be the global coordinate system of the axis, u(s,t) be the position vector of the axis in the inertial frame, where s represents the arc length and s∈[0,L], L is the length of the axial actuator, and the local coordinate system of the axis is Q={d1(s), d2(s), d3(s)}, as follows Figure 1As shown. In the local coordinate system, {d1(s), d2(s)} constitutes a cross-section at a point on the axis, and d3(s) is the normal vector of this cross-section. When shearing occurs at an arc length of s, the direction of d3(s) is different from the direction of the tangent to the axis. When the axial actuator deforms, Q is the rotation matrix. K(s,t) and σ(s,t) represent the angular strain (bending and torsion) and linear strain (shear and extension) of the axis, respectively, Ω(s,t) represents the angular velocity of the axis, and the linear velocity of the axis is... Let e ​​be Young's modulus, ρ be the material density, A be the cross-sectional area, B be the stiffness matrix for bending and torsion, S be the stiffness matrix for extension and shear, I be the second-region tensor, c be the couple density, and f be the applied load. Then the dynamic equation of the axial actuator can be expressed as:

[0080] σ(s,t)=Qu s -d2(s) (1)

[0081]

[0082] in, For the action force, As the axis, The axis is tangent. Based on the driving principle of the pneumatic soft robot, pressurizing different air chambers of the axial actuator to cause bending motion can be considered as different points (e.g., ...). Figure 2 Forces in different directions are applied to the axis at points S1, S2, etc.

[0083] 2. Autonomous Continuous Motion Control Algorithm Based on Reinforcement Learning

[0084] 2.1 Deep Deterministic Policy Gradient Control Algorithm Combining Predictive Reward

[0085] The DDPG algorithm is a model-free reinforcement learning algorithm. Since the state transition probabilities of model-free reinforcement learning algorithms are unknown, the DDPG algorithm is affected by sample complexity. To improve the exploration efficiency of the DDPG algorithm, a prediction reward learning method incorporating an environment model is integrated into the DDPG algorithm's policy learning. This trains the transition chains generated during the algorithm's iteration process, constructing a dynamic environment model.

[0086] The goal of predictive reward learning methods is to predict the reward for the next state, given the current state and action, and then use this reward for subsequent policy learning. Predictive reward techniques are approximated using parameterized feedforward neural networks and reward neural networks. The feedforward neural network is denoted as... It is constructed from a feedforward neural network with parameters φ, where the input is the state s. i and action a i The state at the next moment This is the output. The Adam optimizer is used to learn the neural network parameters. The loss function for the feedforward model neural network is as follows:

[0087]

[0088] In equation (4), N is the number of samples collected from the experience replay set D. By minimizing L... f The optimal parameter φ is obtained.

[0089] The reward prediction function is denoted as It is determined by a parameter of It is constructed using a feedforward neural network, where the input is the state s. i ,award The output is obtained by using the Adam optimizer to learn the network parameters. This is achieved by minimizing the loss function L. r To train the parameters of the reward neural network:

[0090]

[0091] In equation (5), N represents the number of samples collected from the experience replay set D. The output of the feedforward neural network is used as the input to the reward neural network, connecting the two networks to predict the reward for the next state given the current state and action. That is, predicting rewards Used to guide subsequent strategy learning.

[0092] After learning the dynamic model of the environment, the predictive reward technique estimates the predicted reward value for the next state through a reward neural network. Researchers use the learned dynamic model of the environment as additional training data to supplement the algorithm's training, or to provide information to the agent to help it learn its policy, thus applying the predicted reward value for the next state to the algorithm's policy learning. As introduced earlier, the DDPG algorithm uses an actor-critic framework. The critic evaluates the value of the actor's actions and calculates the optimal action value function Q. The actor updates its policy based on the gradient of the action value function Q with respect to the action. The policy update is only affected by the optimal action value function Q calculated by the critic network. This function is the discounted sum of all rewards from the start state to the end state, called the total reward. This embodiment proposes a new policy update method: combining the gradient of the predicted reward (i.e., the second reward mentioned above) with respect to the action with the gradient of the total reward (i.e., the first reward mentioned above) with respect to the action to guide policy update and action selection. The predictive reward technique provides a direction for dynamic perception exploration. The improved DDPG algorithm is called the PR-DDPG control algorithm. Its policy learning is influenced by a mixed objective of maximizing the total reward and the predicted reward. To optimize the policy, a weighting factor is introduced to balance the ratio of the total reward to the predicted reward. Based on the derivation of the deterministic policy gradient formula, the policy update method is as follows:

[0093]

[0094] In formula (6) Let N be the gradient of the expected return, N be the number of samples collected, and λ be the weight factor ∈ (0, 1). Let be the gradient of the action, Q() be the Q-function, s be the state, a be the action, and ω be the parameters of the Q-function. i Given the current input state, μ(s) i () represents the current policy as input. The reward corresponding to the state predicted by the reward prediction learning method for the next action is φ, where φ is the parameter of the feedforward neural network. To reward the network parameters, Let θ be the gradient of the parameter θ, and μ be the deterministic policy to be optimized.

[0095] 2.2 Markov Decision Process (MDP) Modeling

[0096] In reinforcement learning, the agent interacts with the environment and learns from repeated trials how to choose the optimal action to maximize cumulative reward. This learning process is modeled using a Markov Decision Process (MDP) RL architecture.

[0097] The state space, motion space, and reward function of the axial actuator for the pipeline pneumatic soft robot are defined as follows:

[0098] (1) State Space: Based on the bending effect of the pneumatic soft robot in the finite element analysis, four control points are selected on the axis of the axial actuator. According to the bending motion mode of the pneumatic soft robot, the end of the axial actuator is fixed, and the position of the control points is represented by vector X. a The position of the target point is represented by vector X. t Indicated by V a and V t Describe the magnitude and direction of the axial actuator tip and the target velocity, respectively, using q. a and q t To describe the coordinate system orientation between the axial actuator tip and the target, the state space can be described as S = [X...]. a V a q a X t V t q t There are 24 dimensions in total.

[0099] (2) Action Space: According to the mechanical analysis of the deformation of the pipeline pneumatic soft robot structure, applying loads to different cavity walls can realize the bending motion of the axial actuator of the pipeline pneumatic soft robot. Within a certain bending angle range, the inner length of the axial actuator of the pipeline pneumatic soft robot remains basically unchanged. Similarly, taking 4 control points on the axis, applying torque A in the directions of d1(s), d2(s), and d3(s) on the axis. m (s)=α m β m (s) is used to realize the bending motion of the axial actuator of the pipeline pneumatic soft robot, thus having 12 degrees of freedom. α m For the maximum torque, β m The range of selection for (s) is [-1, 1].

[0100] (3) Reward Function: The reward function, also known as the reward agent, effectively evaluates the actions of the agent. The distance error is defined as the distance between the tip of the axial actuator of the pipeline pneumatic soft robot and the target point. The reward function is defined as follows:

[0101]

[0102] In equation (7), n = ||X n -X t || Represents the tip X of the axial actuator of the pipeline soft robot. n With target point X t The distance between them, p is the directional penalty, and d = 6mm is the reward range distance, so that its tip keeps getting closer to the target.

[0103] 2.3 Network Structure and Hyperparameters

[0104] The relevant hyperparameters are set as follows: the learning rate of the Actor network is 0.0001, the learning rate of the Critic network is 0.001, the learning rate of the environment model and the reward neural network is 0.001, the discount factor is 0.99, the batch size N is 128, the capacity of the experience replay set is 2,000,000, the number of hidden layers in both the Actor and Critic networks is 2, and the soft update parameter of the target network is 0.001. A dual neural network architecture is adopted, combining the value evaluation capability of the critic network with the real-time control capability of the actor network to collaboratively optimize the learning effect. The network structures of the actor network and the critic network are as follows. Figure 3 , Figure 4 As shown, the actor network (including input, hidden, and output layers) consists of a 24×400×300×12 fully connected neural network (24 neurons in the input layer, two columns of hidden layers with 400 neurons in the first column and 300 neurons in the second column, and 12 neurons in the output layer). The critic network (including input, hidden, and output layers) has a structure of 36×400×300×1 (36 neurons in the input layer, two columns of hidden layers with 400 neurons in the first column and 300 neurons in the second column, and 1 neuron in the output layer). The feedforward model network has a structure of 24×400×300×12, while the reward network has only one hidden layer with 400 nodes and a structure of 24×400×1. The observed state is normalized into a 24-dimensional input to the actor, and the 12-dimensional output action is used to control the movement of the axial actuator.

[0105] 3 Simulation Experiment

[0106] 3.1 Experimental Platform Setup

[0107] The pneumatic soft robot controller for pipelines gradually learns strategies during the control process. This training requires a massive amount of data, and collecting this data physically is time-consuming. To reduce training costs, a simplified model of the axial actuator of the pneumatic soft robot is created in a simulation environment. This model is then used for autonomous motion control training, and the training results are simulated and tested.

[0108] This experiment utilizes Elastica, a simulation environment capable of solving soft robot mechanics problems. This environment considers the physical characteristics of soft robots, leveraging higher-order continuous elastic effects and their associated dynamics to fully utilize the deformation modes of soft robots. It can be used for distributed actuation and control in continuous state action space. Elastica allows users to utilize a comprehensive control library and define control tasks, variables, actuation modes, and physical environments. Therefore, Elastica can be coupled with RL algorithms to simulate distributed mechanical control strategies, achieving simulation-based dynamic control of soft robots.

[0109] The model parameters of the axial actuator of the pipeline pneumatic soft robot in the simulation environment are set as follows: initial length L of the axial actuator is 100mm, radius d of the axial actuator is 6mm, elastic modulus E is 8MPa, density ρ is 1000kg / m3, and maximum torque α is... m The torque is 5 N·m, and the control time step t is 1.4 ms. The axial actuator is modeled as a cosserat beam, with its bottom fixed and standing vertically in three-dimensional space.

[0110] 3.2 Experiment on Reaching the Target Point

[0111] The first experiment was a target point arrival experiment. To optimize the PR-DDPG control algorithm and study the impact of weight factors on its performance, different weights were used in the PR-DDPG control algorithm. Different PR-DDPG and DDPG algorithms were used to control the pipeline pneumatic soft robot to complete the motion task of reaching the target point. The experimental results of the PR-DDPG control algorithm and the DDPG algorithm were compared and analyzed.

[0112] To optimize the PR-DDPG control algorithm and evaluate the impact of changes in the proportion of the introduced predictive reward technique on the performance of the PR-DDPG control algorithm, i.e., to compare the impact of weight factors on the performance of the PR-DDPG control algorithm, four different weights were selected: PR-DDPG-λ = 0.2, PR-DDPG-λ = 0.6, PR-DDPG-λ = 0.8 and PR-DDPG-λ = 1, resulting in four different PR-DDPG algorithms. These algorithms were then trained separately from the DDPG algorithm mentioned earlier.

[0113] After all algorithms completed 1000 steps per round of training, for a total of 5000 rounds, the resulting learning curves are as follows: Figure 5As shown in (a), the solid line represents the average reward for each round of the algorithm, and the shaded area represents the standard deviation. Algorithm PR-DDPG-λ=1 fails to converge because its policy gradient depends only on the predicted reward; other algorithms can find control policies that reach the target point. Algorithm PR-DDPG-λ=0.2 has a similar reward to the DDPG algorithm, but converges more slowly. Algorithm PR-DDPG-λ=0.6 performs slightly better than the DDPG algorithm, but the difference in their learning curves is not significant. Algorithm PR-DDPG-λ=0.8 converges after approximately 2200 rounds of training, while the DDPG algorithm converges after approximately 3700 rounds. Algorithm PR-DDPG-λ=0.8 converges faster and significantly outperforms the DDPG algorithm.

[0114] The learning curves show that the larger the weight of the predicted reward, the faster the algorithm converges. When the weight is too low, the predicted reward component has little effect on the dynamic perception of the environment. However, when the weight factor is 1, the algorithm's policy learning relies solely on the predicted reward, lacking guidance from the total reward, and the algorithm fails to converge. Therefore, when the weight factor is too low or set to 1, the PR-DDPG control algorithm is inferior to the DDPG algorithm; when the weight factor is 0.8, the PR-DDPG control algorithm outperforms the DDPG algorithm.

[0115] As learning progresses, the noise signal decreases, and the axial actuator of the pipeline pneumatic soft robot can achieve new movements based on its learned experience. The training model tends to stabilize, the reward function approaches convergence, and the control strategy to reach the target is completed. The controller has learned well in the target area of ​​the workspace.

[0116] The trained algorithm PR-DDPG-λ=0.8 was subjected to the same tests as the DDPG algorithm. After completing 10 test sets, the mean and standard deviation of the error values ​​for different test sets were calculated, and an error band plot was drawn. The test results of the PR-DDPG control algorithm were compared with those of the DDPG algorithm. Figure 5 As shown in (b), it can be seen that, compared with the DDPG algorithm, the axial actuator tip of the pipe soft robot controlled by the PR-DDPG algorithm can reach the target point autonomously in a shorter time and with a smaller distance error.

[0117] 3.3 Experiment on reaching the target point in the target direction

[0118] The second experiment focuses on reaching the target point from the target direction. Considering the application scenario of the pipeline pneumatic soft robot designed in this embodiment in the field of industrial inspection, and since the carrier of the inspection equipment is the tip of the pipeline pneumatic soft robot, rapidly and autonomously reaching the target direction can improve the inspection efficiency of the pipeline pneumatic soft robot, giving it a greater advantage when performing pipeline inspection tasks. Therefore, the second experimental task is to control the tip of the pipeline pneumatic soft robot's axial actuator to reach the target position and target direction in three-dimensional space. In this experiment, the target direction is defined as the tangent direction of the axial actuator tip should be perpendicularly upward, while the normal-binormal vector undergoes a random rotation within the range of -90° to 90° from the global coordinate system.

[0119] To effectively evaluate the action of the axial actuator reaching the target point in the target direction, a direction penalty is added to the reward function. The optimized reward function for the experimental task is defined as follows:

[0120]

[0121] In equation (8), d = 6 mm, p = 1 - (q) a ·q t ) 2 As a directional penalty, the tip of the axial actuator is rotated to the target direction, q a For the axial actuator tip direction of the pipeline soft robot, q t The direction of the target point.

[0122] The controller learning process was modeled earlier, where the state space and motion space are defined based on control points. The setting of control points plays a crucial role in controlling the axial actuator. The PR-DDPG control algorithm can be optimized by changing the control point settings, allowing the axial actuator to perform more complex motion tasks. When the number of control points changes, the state space dimension and motion space degrees of freedom of the axial actuator change. The network structures of the actor and critic also change accordingly, as shown in Table 1.

[0123] Table 1. Network Structure Corresponding to the Number of Control Points

[0124]

[0125] To investigate the impact of control point settings on the PR-DDPG control algorithm, control points of different numbers or positions were selected on the axis of the axial actuator of the pipeline pneumatic soft robot, and the same experimental task was performed using the PR-DDPG control algorithm. Based on the previous text, PR-DDPG-λ = 0.8 was selected.

[0126] Simulate different control point settings, such as Figure 6As shown. Training strategies for 2, 3, 4, and 8 uniformly distributed control points, and 8 manually set control points (corresponding to...). Figure 6 (a) Figure 6 (b) Figure 6 (c) Figure 6 (d) Figure 6 (e)), the network structures of the corresponding different PR-DDPG control algorithms are shown in Table 1.

[0127] After all algorithms completed 1000 steps per round of training, for a total of 5000 rounds, the training results are as follows: Figure 7 As shown in (a). Figure 7 The results in (a) show that 2 and 8 control points have poor learning performance. This is because 2 equidistant control points do not provide enough degrees of freedom to achieve the goal, while 8 control points have a larger action space, but extensive exploration leads to slow learning. 3 and 4 control points perform better than 2 and 8 control points because their action space is smaller than 8 control points and their degrees of freedom are higher than 2 control points. The best performing control point is 4 control points, indicating that it strikes a balance between sufficient motivation to complete the task and a sufficiently small action space for rapid exploration. 3 control points also perform well, outperforming 4 control points in the learning performance range of 2000 to 2650 rounds, but the increased controllability of 4 control points ultimately leads to a higher reward value, suggesting that setting 3 control points can improve exploration efficiency in simple tasks.

[0128] Furthermore, the manually placed 8 control points performed better than the uniformly distributed 8 control points because the manually set control point positions better utilize the compliance of the axial actuator. Unlike the uniformly distributed control points, the control points were manually set at the tip of the axial actuator and at positions near the middle of the axial actuator. Setting the control points at the tip is to control its steering during movement, achieving more precise tip control, while setting the control points near the middle is to induce a larger bend, thereby controlling the overall deformation of the axial actuator.

[0129] In summary, fewer control points allow for faster exploration of the action space, resulting in more effective sample learning. However, too few control points reduce controllability, potentially making task completion impossible. Therefore, it is necessary to maximize the controllability of the axial actuator based on the task requirements and set as few control points as possible, striking a balance between controllability and exploration speed. For this experimental task, the following was selected: Figure 6 (c) shows four control points that optimize the learning performance of the PR-DDPG control algorithm.

[0130] After training with the DDPG algorithm using 4 control points, both the PR-DDPG algorithm and the DDPG algorithm were tested on 10 test sets. The mean and standard deviation of the reward values ​​for different test sets were calculated, and the reward values ​​of the PR-DDPG algorithm were compared with those of the DDPG algorithm. Figure 7 As shown in (b), after learning, the PR-DDPG algorithm converges faster and obtains a higher reward value than the DDPG algorithm, meaning that the PR-DDPG algorithm has higher control performance.

[0131] Example 3

[0132] This embodiment provides a pipeline pneumatic soft robot control system based on deep reinforcement learning, including:

[0133] Control module: Used to control the movement of the axial actuator in the pipeline pneumatic soft robot through an improved DDPG algorithm, thereby achieving control of the pipeline pneumatic soft robot. Specifically:

[0134] The improved DDPG algorithm introduces a predictive reward learning method into the DDPG algorithm;

[0135] The DDPG algorithm includes an actor network and a critic network. The actor network is used to generate the action of the axial actuator, and the critic network is used to score the action generated by the actor network and calculate the action value function Q. The calculated action value function Q is used as the first reward.

[0136] The prediction reward learning method is implemented through a feedforward model neural network and a reward neural network. The feedforward model neural network is used to predict the state generated by the axial actuator performing the next action, and the reward neural network is used to predict the corresponding second reward based on the state generated by the next action.

[0137] The first reward corresponding to the DDPG algorithm and the second reward corresponding to the prediction reward learning method are combined to obtain a policy function for guiding the axial actuator to perform actions.

[0138] The axial actuator performs actions according to the strategy function to achieve control of the pipeline pneumatic soft robot.

[0139] Example 3

[0140] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the pipeline pneumatic soft robot control method based on deep reinforcement learning described in Embodiment 1.

[0141] Example 4

[0142] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the pipeline pneumatic soft robot control method based on deep reinforcement learning described in Embodiment 1.

[0143] Example 4

[0144] This embodiment provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the deep reinforcement learning-based pipeline pneumatic soft robot control method described in Embodiment 1.

[0145] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0146] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0149] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0150] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A control method for a pipeline pneumatic soft robot based on deep reinforcement learning, characterized in that: include: The improved DDPG algorithm is used to control the movement of the axial actuator in the pipeline pneumatic soft robot, thereby achieving control of the pipeline pneumatic soft robot. Specifically: The improved DDPG algorithm introduces a predictive reward learning method into the DDPG algorithm; The DDPG algorithm includes an actor network and a critic network. The actor network is used to generate the action of the axial actuator, and the critic network is used to score the action generated by the actor network and calculate the action value function Q. The calculated action value function Q is used as the first reward. The prediction reward learning method is implemented through a feedforward model neural network and a reward neural network. The feedforward model neural network is used to predict the state generated by the axial actuator performing the next action, and the reward neural network is used to predict the corresponding second reward based on the state generated by the next action. The first reward corresponding to the DDPG algorithm and the second reward corresponding to the prediction reward learning method are combined to obtain a policy function for guiding the axial actuator to perform actions. The axial actuator performs actions according to the strategy function to achieve control of the pipeline pneumatic soft robot.

2. The control method for a pipeline pneumatic soft robot based on deep reinforcement learning according to claim 1, characterized in that: The first reward corresponding to the DDPG algorithm and the second reward corresponding to the prediction reward learning method are combined to obtain a policy function for guiding the axial actuator to perform actions, as shown in the formula: in, Let λ be the gradient of the expected return, N be the number of samples collected, and λ be the weighting factor. Let be the gradient of the action, Q() be the Q-function, s be the state, a be the action, and ω be the parameters of the Q-function. i Given the current input state, μ(s) i () represents the current policy as input. To predict the reward corresponding to the state generated by the next action predicted by the reward learning method, φ is the parameter of the feedforward neural network, and this feedforward neural network is used as the forward model neural network. To reward the network parameters, Let θ be the gradient of the parameter θ, and μ be the deterministic policy to be optimized.

3. The control method for a pipeline pneumatic soft robot based on deep reinforcement learning according to claim 1, characterized in that: The learning process of the axial actuator of the pipeline pneumatic soft robot is modeled using Markov Decision Process (MDP), including: The state space, motion space, and reward function of the axial actuator of the pipeline pneumatic soft robot are set as follows: The state space is defined as follows: Based on the bending effect of the pneumatic soft robot in finite element analysis, four evenly distributed control points are selected along the axis of the axial actuator. According to the bending motion mode of the pneumatic soft robot, the end of the axial actuator is fixed, and the positions of the control points are represented by vector X. a The position of the target point is represented by vector X. t Indicated by V a and V t Describe the magnitude and direction of the axial actuator tip and the target velocity, respectively, using q. a and q t To describe the coordinate system orientation between the axial actuator tip and the target, the state space can be described as S = [X...]. a V a q a X t V t q t ]; The action space is defined as follows: Based on the mechanical analysis of the deformation of the pneumatic soft robot structure, loads are applied to different cavity walls to achieve the bending motion of the axial actuator of the pneumatic soft robot. Within a certain bending angle range, the inner length of the axial actuator of the pneumatic soft robot remains basically unchanged. Similarly, four evenly distributed control points are taken on the axis, and torques A are applied in the directions d1(s), d2(s), and d3(s) on the axis. m (s)=α m β m (s) is used to realize the bending motion of the axial actuator of the pipeline pneumatic soft robot, α m For the maximum torque, β m The range of selection for (s) is [-1, 1]; The reward function is defined as follows: the distance error is the distance between the tip of the axial actuator of the pipeline pneumatic soft robot and the target point, expressed as: Where, n = ||X n -X t || Represents the tip X of the axial actuator of the pipeline soft robot. n With target point X t The distance between them, p is the directional penalty, and d is the reward range distance.

4. The control method for a pipeline pneumatic soft robot based on deep reinforcement learning according to claim 1, characterized in that: Let the feedforward model neural network be denoted as... It is constructed from a feedforward neural network with parameters φ, where the input is the current state s. i and current action a i The state at the next moment For the output, construct the loss function of the feedforward neural network, as shown in the formula: Where N is the number of samples collected, and L is minimized. f Obtain the optimal parameter φ; The loss function L of the feedforward neural network f To train a feedforward neural network.

5. The control method for a pipeline pneumatic soft robot based on deep reinforcement learning according to claim 1, characterized in that: Let the reward neural network be denoted as... It consists of one parameter. The feedforward neural network is constructed, and the input is the current state s. i ,award To output the reward neural network, construct the loss function as follows: Where N is the number of samples collected; The loss function L of the reward neural network r To train a reward neural network.

6. The control method for a pipeline pneumatic soft robot based on deep reinforcement learning according to claim 1, characterized in that: The dynamic analysis is performed using the axis of the axial actuator as a curve model. Let the orthonormal basis {e1, e2, e3} be the global coordinate system of the axis, and u(s,t) be the position vector of the axis in the inertial frame, where s represents the arc length and s∈[0,L], and L is the length of the axial actuator. The local coordinate system of the axis is Q={d1(s), d2(s), d3(s)}. In the local coordinate system, {d1(s), d2(s)} constitute a cross-section at a point on the axis, and d3(s) is the normal vector of this cross-section. When shearing occurs at arc length s, the direction of d3(s) is different from the direction of the tangent to the axis. When the axial actuator deforms, Q is the rotation matrix. K(s,t) and σ(s,t) represent the angular strain and linear strain of the axis, respectively, Ω(s,t) represents the angular velocity of the axis, and the linear velocity of the axis is... Let e ​​be Young's modulus, ρ be material density, A be cross-sectional area, B be the stiffness matrix for bending and torsion, S be the stiffness matrix for extension and shear, I be the second-region tensor, c be couple density, and f be the load force. Then, the dynamic simulation model of the axial actuator can be expressed as: σ(s,t)=Qu s -d2(s); in, For the action force, As the axis, It is the tangent to the axis.

7. A control system for a pipeline pneumatic soft robot based on deep reinforcement learning, characterized in that: include: Control module: Used to control the movement of the axial actuator in the pipeline pneumatic soft robot through an improved DDPG algorithm, thereby achieving control of the pipeline pneumatic soft robot. Specifically: The improved DDPG algorithm introduces a predictive reward learning method into the DDPG algorithm; The DDPG algorithm includes an actor network and a critic network. The actor network is used to generate the action of the axial actuator, and the critic network is used to score the action generated by the actor network and calculate the action value function Q. The calculated action value function Q is used as the first reward. The prediction reward learning method is implemented through a feedforward model neural network and a reward neural network. The feedforward model neural network is used to predict the state generated by the axial actuator performing the next action, and the reward neural network is used to predict the corresponding second reward based on the state generated by the next action. The first reward corresponding to the DDPG algorithm and the second reward corresponding to the prediction reward learning method are combined to obtain a policy function for guiding the axial actuator to perform actions. The axial actuator performs actions according to the strategy function to achieve control of the pipeline pneumatic soft robot.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the deep reinforcement learning-based pipeline pneumatic soft robot control method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the deep reinforcement learning-based pipeline pneumatic soft robot control method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the deep reinforcement learning-based control method for a pipe pneumatic soft robot as described in any one of claims 1 to 6.

Images