A continuum robotic arm motion control method based on deep reinforcement learning

Abstract

The present invention discloses a continuum robot motion control method based on deep reinforcement learning, which applies the deep reinforcement learning DDPG algorithm to the continuum robot motion control, and trains the control strategy by making the robot continuously interact with the environment. The present invention realizes the autonomous motion control of the continuum robot by constructing a virtual training environment for the continuum robot, establishing a DDPG algorithm model, constructing a reward function calculation model, accumulating an experience replay pool, training a deep reinforcement learning network, and deploying the trained model to the actual robot system. The DDPG algorithm in the present invention adopts a deterministic strategy, has good network convergence, reduces excessive Q estimation values ​​by using a target network, and adds random action noise, which is conducive to exploring the optimal strategy; priority experience replay is used during training, which can speed up the training speed and improve the training effect; and through the domain randomization method, the deviation between the training environment and the real environment is reduced, and the success rate of model deployment is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot arm control, and in particular to a continuum robot arm motion control method based on deep reinforcement learning. Background Art

[0002] Continuum manipulators are a type of manipulator developed based on bionics. They have the advantages of small size, light weight, and good flexibility, and can be used in various narrow and complex working environments. However, continuum manipulators have problems such as high system dimension, obvious nonlinear characteristics, and difficulty in accurate modeling. Traditional control methods mostly rely on system models, making it difficult to achieve accurate control.

[0003] Reinforcement learning is a "trial and error search" method that does not rely on the precise mathematical model of the agent. It enables the agent to learn action strategies through continuous interaction with the environment. Deep reinforcement learning combines the fitting ability of deep neural networks with the decision-making ability of reinforcement learning. The two complement each other and provide solutions to the perception and decision-making problems of complex systems. In recent years, with the emergence of deep reinforcement learning algorithms such as DDPG, TD3, and SAC, reinforcement learning has been able to solve the problem of continuous state action space and can be well applied to the robotic arm environment, especially bringing new research directions to the control problem of continuum robotic arms.

[0004] At present, the robot arm control problem based on deep reinforcement learning is represented by the Actor-Critic algorithm, which mostly uses uncertain action strategies and uniform experience replay. There are problems such as slow network convergence, long training time, and poor control effect. Moreover, most methods adopt the strategy of "training in a virtual environment first and then deploying in the actual system". Since the constructed virtual environment is often different from the actual environment, the trained model cannot have a good effect on the actual system. Summary of the invention

[0005] The purpose of the present invention is to provide a continuum robotic arm control method based on deep reinforcement learning to address the problems existing in the above-mentioned prior art.

[0006] The technical solution to achieve the purpose of the present invention is: a continuum robot arm motion control method based on deep reinforcement learning, the method comprising the following steps:

[0007] Step 1, constructing a continuum robotic arm virtual training environment;

[0008] Step 2: Establish a deep reinforcement learning DDPG algorithm model;

[0009] Step 3, construct a reward function calculation model;

[0010] Step 4, accumulate experience replay pool;

[0011] Step 5, perform one-stage training of the deep reinforcement learning network;

[0012] Step 6, randomization of virtual environment domain;

[0013] Step 7, perform two-stage training of the deep reinforcement learning network;

[0014] Step 8: Deploy the trained deep reinforcement learning model to the actual robotic arm system.

[0015] Furthermore, step 1 of constructing a continuum robotic arm virtual training environment specifically includes:

[0016] Step 1.1, draw a three-dimensional model of the continuum robotic arm;

[0017] Step 1.2: Initialize the virtual environment, set the coordinate system O-XYZ, define the robot working space, and adjust the camera position;

[0018] Step 1.3, import the 3D model of the continuum robotic arm, add dynamic and kinematic properties to it, adjust the initial state of the robotic arm, and render the environment in which the robotic arm is located;

[0019] Step 1.4, specify the observation state and action information of the robot arm, and set the robot arm update steps;

[0020] Step 1.5, set up the communication connection between the virtual environment and the external algorithm model.

[0021] Furthermore, step 2 establishes a deep reinforcement learning DDPG algorithm model, specifically including:

[0022] Step 2.1, establish the DDPG algorithm neural network: including the main network and the target network, where the main network includes the Actor network and the Critic network, and the target network includes the Target Actor network and the Target Critic network. The main network and the target network have the same network structure, and only the network parameters are different; the Actor network adopts a deterministic strategy, given an input state, and outputs a certain action information, expressed as π(s; θ), where s is the observation state of the robot arm, and θ is the Actor network parameter; the Critic network is an evaluation network, which is used to calculate the Q value to evaluate the quality of the strategy adopted by the Actor network, expressed as Q(s, a; w), where a is the action information output by the Actor network, and w is the Critic network parameter; the target network is used to calculate the target Q value to assist in updating the main network parameters. The TargetActor and Target Critic networks are expressed as π(s; θ - ) and Q(s, a; w -), where θ - and w - are the parameters of the TargetActor network and the Target Critic network respectively; randomly initialize the parameters of the Actor network to θ = θ0 and the Critic network to w = w0, and initialize the parameters of the TargetActor network to θ - =θ and the parameter of TargetCritic network is w - =w;

[0023] Step 2.2, establish the experience replay pool: the data stored in the experience pool is in the form of five-tuple (s t , a t , r t ,s t+1 , δ t ), where s t represents the observed state at time t, a t Indicates state s t The action output obtained under the input, r t Indicates execution of action a t The reward value obtained, s t+1 Indicates execution of action a t The next state obtained after δ t is the time difference error, which is used to determine the priority of experience playback; if the experience pool capacity is C, the observation state dimension is dim s , the action information dimension is dim a , then use a two-dimensional matrix with C rows and D columns to store it, where D = 2·dim s +dim a +1+1, two numbers 1 represent r t and δ t Each occupies one storage space; and initializes each element of the two-dimensional matrix to 0.

[0024] Furthermore, the calculation of the reward function in step 3 follows the following rules:

[0025] Rule 1: If the robot moves beyond the working space, a penalty of -10 will be imposed;

[0026] Rule 2: The reward value is related to the distance from the end of the robot to the target position. The closer the new end position is to the target position relative to the previous end position, the greater the reward value.

[0027] Rule 3: If the distance from the end position to the target position is less than the set value d min , it is considered that the end has reached the target position and a reward of +10 is given;

[0028] According to the above rules, the reward value r tThe calculation process is as follows: If the initial position of the end of the robot at time t is (x0 t ,y0 t , z0 t ), the target position is (x1 t , y1 t , z1 t ), the robot arm is based on the current input state s t Execute action a t After updating, the new end position is (x0 t+1 ,y0 t+1 , z0 t+1 ), then the reward function is as shown in formula (1):

[0029]

[0030] Furthermore, the accumulated experience replay pool in step 4 specifically includes:

[0031] Step 4.1, observe the current virtual robot state s t , input to the Actor network to get action output When accumulating experience replay pool, in action Add random noise as the actual action, that is Where N t The mean is 0 and the variance is σ 2 Gaussian random noise;

[0032] Step 4.2: The virtual environment robot performs action a t , update the robot state to get the next moment observation state s t+1 , and calculate the reward value r t ;

[0033] Step 4.3, change the state s t and action a t Input into the Critic network to get the Q estimate value Q eval =Q(s t , a t ; w), the state s t+1 Input into the Target Actor network to get the next action output a' of the network t+1 =π(s t+1 θ - ), and then the state s t+1 and action a′ t+1 Input into the Target Critic network to get Q(s t+1 , a′ t+1 ;w - ), we can further get the target value of Q as Q target=r t +γ·Q(s t+1 , a′ t+1 ;w - ), where γ is the discount factor, then the time difference error is calculated according to formula (2):

[0034] δ t =Q eval -Q target =Q(s t , a t ;w)-(r t +γ·Q(s t+1 , a′ t+1 ;w - )) (2)

[0035] Step 4.4, convert the quintuple (s t , a t , r t ,s t+1 , δ t ) is stored in the experience replay pool;

[0036] Step 4.5: Set the next state s t+1 As the current observation state of the robot arm s t , repeat steps 4.1 to 4.4;

[0037] Step 4.6: Repeat step 4.5 until the experience replay pool is full.

[0038] Furthermore, the first-stage training of the deep reinforcement learning network described in step 5 specifically includes:

[0039] Step 5.1, using the priority experience replay method to take N samples from the experience replay pool for training; specifically including:

[0040] Step 5.1.1, according to |δ i The size of | gets the rank (i) of each sample in the experience replay pool, |δ i The larger the |, the smaller the value of rank(i);

[0041] Step 5.1.2, extract N samples according to the sampling probability P(i) defined by formula (3):

[0042]

[0043] in, The parameter α controls the degree of priority sorting, α∈(0,1);

[0044] Step 5.1.3, calculate the importance sampling weights of the N samples sampled according to formula (4):

[0045]

[0046] Where C is the capacity of the experience replay pool, P(i) is the sampling probability of sample i, and the parameter β controls the degree of using importance sampling weights, β∈(0,1);

[0047] Step 5.2, update the Actor network parameters, specifically: set the state s i and action a i Input into the Critic network to get the Q estimate value Q(s i , a i ; w), calculate the average policy gradient g of the sampled N experiences according to formula (5):

[0048]

[0049] Update the parameters θ of the Actor network based on gradient ascent, as shown in formula (6):

[0050] θ←θ+ρ·g (6)

[0051] Where ρ is the Actor network learning rate, ρ∈(0,1);

[0052] Step 5.3, update the critic network parameters, specifically:

[0053] The goal of updating the critic network is to minimize the temporal difference error, and the mean square temporal difference error weighted by the importance sampling weight is used as the loss function L, as shown in formula (7):

[0054]

[0055] Update the critic network parameter w based on gradient descent, as shown in formula (8):

[0056]

[0057] Where v is the Critic network learning rate, v∈(0,1);

[0058] Step 5.4, update the Target Actor network parameters and Target Critic network parameters, specifically:

[0059] Every K rounds, the target network is soft-updated using the Actor network and Critic network parameters, as shown in formula (9):

[0060]

[0061] Where τ is the soft update parameter, τ∈(0,1);

[0062] Step 5.5, update the experience replay pool: discard the oldest five-tuple in the experience replay pool, and execute steps 4.1 to 4.4 to accumulate a new set of experience;

[0063] Step 5.6, loop steps 5.1 to 5.5 in the total iteration step range, and the total number of loops is total_step1=step1*epoch 1, where step1 is the number of iteration steps per round, and epoch1 is the number of training rounds.

[0064] Furthermore, the virtual environment domain randomization in step 6 is specifically as follows:

[0065] The action strategy trained in step 5 is applied to the virtual robotic arm and the actual robotic arm respectively to obtain the virtual observation trajectory and the actual observation trajectory. The difference between the two trajectories is minimized by adjusting the virtual environment system parameters.

[0066] Furthermore, the virtual environment domain randomization in step 6 specifically includes the following process:

[0067] Step 6.1, define the virtual robot system parameters to follow Gaussian distribution The distribution is determined by the parameter φ = (μ, ∑), where ξ represents the virtual robot system parameters, μ and ∑ represent the mean and covariance of the Gaussian distribution, respectively. The probabilistic system dynamics of the virtual robot is expressed as And initialize φ=φ0;

[0068] Step 6.2, resetting both the virtual robot arm and the actual robot arm to their initial states;

[0069] Step 6.3: The action strategy π trained in step 5 * (s; θ) are applied to the virtual robot arm and the actual robot arm respectively, and the current states of the virtual robot arm and the actual robot arm are observed as o i，ξ and i，real , according to the action strategy, the action outputs are a i，ξ =π * (o i，ξ ;θ) and a i，real =π * (o i，real ;θ), after executing the action, the states at the next moment are o i+1，ξ and i+1，real ;

[0070] Step 6.4, the next state o i+1，ξ and i+1，real , respectively as the current state o i，ξ and i，real , repeat step 6.3;

[0071] Step 6.5: Repeat step 6.4 within the total iteration number T steps to obtain the virtual observation trajectory. and real-world observation trajectories The deviation function of the two trajectories is calculated using the 2-norm as shown in (10):

[0072]

[0073] Where W is the importance weight matrix of each observation dimension;

[0074] Step 6.6, virtual environment parameter update: minimize the objective function shown in formula (11):

[0075]

[0076] The virtual environment parameters are optimized by gradient descent, as shown in formula (12):

[0077]

[0078] Among them, λ is the virtual environment parameter update step, λ∈(0,1);

[0079] Step 6.7: Repeat steps 6.2 to 6.6 within the total number of optimization rounds. The total number of rounds is E. Finally, the optimal parameter of the virtual environment is φ. * .

[0080] Furthermore, the two-stage training of the deep reinforcement learning network described in step 7 includes the following specific processes:

[0081] Step 7.1, re-accumulate the experience replay pool. The specific process is the same as step 4, the difference is: when selecting an action, use a t =π * (s t ; θ), and no random action noise is added;

[0082] Step 7.2, network retraining, the process is the same as step 5, the difference is: the number of training rounds epoch2 and the number of iterations per round step2 are reduced compared to the first stage training.

[0083] Furthermore, step 8 deploys the trained deep reinforcement learning model to the actual robotic arm system. The specific process includes:

[0084] The actual robot arm system includes an observation module, a decision module, an execution module and a continuum robot arm;

[0085] Step 8.1, deploy the deep reinforcement learning model trained in step 7 to the decision module of the actual robotic arm system;

[0086] Step 8.2, obtain the current input state s of the robot through the observation module t , the deep reinforcement learning model in the decision module is based on the input state s t Output action command a t , the execution module performs action a t , get the next input state s t+1 ;

[0087] Step 8.3, the next input state s t+1 As the current input state s t , repeat step 8.2 until the end of the robot arm reaches the target position.

[0088] Compared with the prior art, the present invention has the following significant advantages:

[0089] (1) The DDPG algorithm uses a deterministic strategy, and the network has better convergence. By adding random action noise to the output of the policy network, the robot arm can better explore the environment and discover the potential optimal strategy.

[0090] (2) By using two target networks, TargetActor and Target Critic, to calculate the temporal difference target value, the influence of "bootstrapping" is eliminated, the overestimation value is reduced, and the effectiveness of strategy learning is improved.

[0091] (3) By using the priority experience replay method, valuable samples are given priority for training, making the strategy training more efficient, accelerating the convergence of the model, and improving the success rate of the control strategy.

[0092] (4) By performing domain randomization on the virtual training environment, the difference between the virtual environment and the real environment is greatly reduced, thereby improving the success rate of the algorithm model in actual system deployment.

[0093] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 Schematic diagram of the continuum robotic arm structure.

[0095] Figure 2 This is a flow chart of the continuum robotic arm motion control method based on deep reinforcement learning of the present invention.

[0096] Figure 3 Schematic diagram of the DDPG algorithm structure in the present invention.

[0097] Figure 4 Schematic diagram of deploying the deep reinforcement learning model to the actual robotic arm system. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0099] like Figure 1 The figure shows a schematic diagram of the structure of a continuum robot arm in a specific embodiment of the present invention. The robot arm is composed of a central rod, a plurality of spacer disks and four driving ropes. Each spacer disk is fixedly connected to the central rod, and the driving rope passes through each spacer disk. One end of the driving rope is connected to a motor, and the other end is connected to the end spacer disk. The motor controls the retraction and extension of the four driving ropes to complete the control of the robot arm.

[0100] like Figure 2 As shown, it is a flow chart of a continuum robot arm motion control method based on deep reinforcement learning according to a specific embodiment of the present invention, which includes the following steps:

[0101] Step 1: Build a virtual training environment for the continuum robotic arm

[0102] This embodiment uses Unity software to build a continuum robotic arm virtual environment. The main steps are as follows:

[0103] Step 1.1, draw a three-dimensional model of the continuum robotic arm using SolidWorks software;

[0104] Step 1.2, initialize the Unity virtual environment, set the coordinate system O-XYZ, specify the robot arm workspace, and adjust the camera position;

[0105] Step 1.3, import the 3D model of the continuum robot arm, add dynamics, kinematics and other properties to it, adjust the initial state of the robot arm, and render the environment in which the robot arm is located;

[0106] Step 1.4: define the observation state of the continuum manipulator as the position coordinate of the end of the manipulator, that is, s t =(x t ,y t , z t ); The action amount of the continuum robot arm is defined as the change in the length of the driving rope, i.e., a t =(Δl1, Δl2, Δl3, Δl4); and set the robot arm update steps;

[0107] Step 1.5, setting the communication connection between the virtual environment and the external algorithm model. This specific embodiment adopts TCP communication.

[0108] Step 2: Press Figure 3 The algorithm structure shown establishes the deep reinforcement learning DDPG algorithm model

[0109] Step 2.1, establish the algorithm neural network: the neural network of the DDPG algorithm consists of a main network and a target network, where the main network consists of an Actor network and a Critic network, and the target network consists of a TargetActor network and a Target Critic network. The main network and the target network have the same network structure, but only the network parameters are different; the Actor network adopts a deterministic strategy, given an input state, and outputs a certain action information, expressed as π(s; θ), where s is the observed state of the robot, and θ is the Actor network parameter; the Critic network is an evaluation network, which is used to calculate the Q value to evaluate the quality of the strategy adopted by the Actor network, expressed as Q(s, a; w), where s is the observed state of the robot, a is the action information output by the Actor network, and w is the Critic network parameter; the main function of the target network is to calculate the target Q value to assist in updating the main network parameters. The two networks are expressed as π(s; θ - ) and Q(s, a; w - ), where θ - and w - are the parameters of the TargetActor network and the Target Critic network respectively; the parameters of the Actor network are randomly initialized to θ=θ0 and the parameters of the Critic network are randomly initialized to w=w0, and the parameters of the TargetActor network are initialized to θ - =θ and initialize the parameters of the Target Critic network to w - =w;

[0110] Step 2.2, establish the experience replay pool: the data stored in the experience pool is in the form of five-tuple (s t , a t , r t ,s t+1 , δ t ), where s t represents the observed state at time t, a t Indicates state s t The action output obtained under the input, r t Indicates execution of action a t The reward value obtained, s t+1 Indicates execution of action a t The next state obtained after δ t is the time difference error, which is calculated as shown in formula (1) and is used to determine the experience playback priority; if the experience pool capacity is C and the observation state dimension is dim s , the action information dimension is dim a , then use a two-dimensional matrix with C rows and D columns to store it, where D = 2·dim s +dima +1+1, two numbers 1 represent r t and δ t Each occupies one storage space; and initializes each element of the two-dimensional matrix to 0.

[0111] Step 3: Construct a reward function calculation model, following the following rules:

[0112] Rule 1: If the robot moves beyond the working space, a large penalty of -10 will be imposed;

[0113] Rule 2: The reward value is related to the distance from the end of the robot to the target position. The closer the new end position is to the target position relative to the previous end position, the greater the reward value will be.

[0114] Rule 3: If the distance from the end position to the target position is less than d min , it is considered that the end has reached the target position and a larger reward of +10 is given;

[0115] According to the above rules, the reward value r t The calculation process is as follows: If the initial position of the end of the robot at time t is (x0 t ,y0 t , z0 t ), the target position is (x1 t , y1 t , z1 t ), the robot arm is based on the current input state s t Execute action a t After updating, the new end position is (x0 t+1 ,y0 t+1 , z0 t+1 ), then the reward function is as shown in formula (13):

[0116]

[0117] Step 4: Accumulate experience replay pool

[0118] Step 4.1, observe the current virtual robot state s t , input to the Actor network to get action output When accumulating experience replay pool, in order to enhance the exploration ability of the robot arm, Add random noise as the actual action, that is Where N t The mean is 0 and the variance is σ 2 Gaussian random noise;

[0119] Step 4.2: The virtual environment robot performs action a t , update the robot state to get the next moment observation state st+1 , and calculate the reward value r t ;

[0120] Step 4.3, change the state s t and action a t Input into the Critic network to get the Q estimate value Q eval =Q(s t , a t ; w), the state s t+1 Input into the Target Actor network to get the next action output a' of the network t+1 =π(s t+1 θ - ), and then the state s t+1 and action a′ t+1 Input into the Target Critic network to get Q(s t+1 , a′ t+1 ;w - ), we can further get the target value of Q as Q target =r t +γ·Q(s t+1 , a′ t+1 ;w - ), where γ is the discount factor, then the time difference error is calculated according to formula (14):

[0121] δ t =Q eval -Q target =Q(s t , a t ;w)-(r t +γ·Q(s t+1 , a′ t+1 ;w - )) (14)

[0122] Step 4.4, convert the quintuple (s t , a t , r t ,s t+1 , |δ t |)Store to the experience replay pool;

[0123] Step 4.5: Set the next state s t+1 As the current observation state of the robot arm s t , repeat steps 4.1 to 4.4;

[0124] Step 4.6: Repeat step 4.5 until the experience replay pool is full.

[0125] Step 5: Perform one-stage training of the deep reinforcement learning network

[0126] Step 5.1: Use priority experience replay to take N samples from the experience replay pool for training. Specifically:

[0127] Step 5.1.1, according to |δ i The size of | gets the rank (i) of each sample in the experience replay pool, |δ i The larger the |, the smaller the value of rank(i);

[0128] Step 5.1.2, extract N samples according to the sampling probability P(i) defined by equation (15):

[0129]

[0130] in, The parameter α controls the degree of priority sorting, α∈(0,1);

[0131] In step 5.1.3, the importance sampling weights of the N samples are calculated according to formula (16). The weights are used in step 5.3 to update the critic network parameters to correct the changes in the original sample distribution caused by priority sampling:

[0132]

[0133] Where C is the capacity of the experience replay pool, P(i) is the sampling probability of sample i, and the parameter β controls the degree of using importance sampling weights, β∈(0,1);

[0134] Step 5.2, update the Actor network parameters, specifically:

[0135] The state s i and action a i Input into the Critic network to get the Q estimate value Q(s i , a i; w), which is used to evaluate the quality of the action strategy of the Actor network. The training goal of the Actor network is to maximize the Q estimate. First, the average policy gradient of the N sampled experiences is calculated according to formula (17):

[0136]

[0137] Then the parameters θ of the Actor network are updated based on the gradient ascent, as shown in formula (18):

[0138] θ←θ+ρ·g (18)

[0139] Where ρ is the Actor network learning rate, ρ∈(0,1);

[0140] Step 5.3, update the critic network parameters, specifically:

[0141] In step 4.3, the time difference error of each sample in the experience replay pool has been calculated. The time difference error represents the deviation between the Q estimate and the Q target value. The goal of the critic network update is to minimize this deviation. The mean square time difference error weighted by the importance sampling weight is used as the loss function, as shown in formula (19):

[0142]

[0143] Then, the critic network parameter w is updated based on gradient descent, as shown in formula (20):

[0144]

[0145] Where v is the Critic network learning rate, v∈(0,1);

[0146] Step 5.4, update the Target Actor network parameters and Target Critic network parameters, specifically:

[0147] Every K rounds, the target network is soft-updated using the Actor network and Critic network parameters, as shown in formula (21):

[0148]

[0149] Where τ is the soft update parameter, τ∈(0,1);

[0150] Step 5.5, update the experience replay pool: discard the oldest five-tuple in the experience replay pool, and execute steps 4.1 to 4.4 to accumulate a new set of experience;

[0151] Step 5.6, loop steps 5.1 to 5.5 in the total iteration step range, and the total number of loops is total_step1=step1*epoch 1, where step1 is the number of iteration steps per round, and epoch1 is the number of training rounds.

[0152] Step 6: Virtual environment domain randomization, including:

[0153] Step 6.1, Step 6.1, define the virtual robot system parameters to follow Gaussian distribution The distribution is determined by the parameter φ = (μ, ∑), where ξ represents the virtual robot system parameters, μ and ∑ represent the mean and covariance of the Gaussian distribution, respectively. The probabilistic system dynamics of the virtual robot is expressed as And initialize φ=φ0;

[0154] Step 6.2, resetting both the virtual robot arm and the actual robot arm to their initial states;

[0155] Step 6.3: The action strategy π trained in step 5 * (s; θ) are applied to the virtual robot arm and the actual robot arm respectively, and the current states of the virtual robot arm and the actual robot arm are observed as o i，ξ and i，real , according to the action strategy, the action outputs are a i，ξ =π * (o i，ξ ;θ) and a i，real =π * (o i，real ;θ), after executing the action, the states at the next moment are o i+1，ξ and i+1，real ;

[0156] Step 6.4, the next state o i+1，ξ and i+1，real , respectively as the current state o i，ξ and i，real , repeat step 6.3;

[0157] Step 6.5: Repeat step 6.4 within the total iteration number T steps to obtain the virtual observation trajectory. and real-world observation trajectories The deviation function of the two trajectories is calculated using the 2-norm as shown in (22):

[0158]

[0159] Where W is the importance weight matrix of each observation dimension;

[0160] Step 6.6, virtual environment parameter update: The goal of virtual environment domain randomization is to minimize the deviation between the virtual observation trajectory and the real world observation trajectory when the same control strategy is adopted, that is, to minimize the objective function shown in (23):

[0161]

[0162] The virtual environment parameters are optimized by gradient descent, as shown in formula (24):

[0163]

[0164] Among them, λ is the virtual environment parameter update step, λ∈(0,1);

[0165] Step 6.7: Repeat steps 6.2 to 6.6 within the total number of optimization rounds. The total number of rounds is E. Finally, the optimal parameter of the virtual environment is φ. * .

[0166] Step 7: Perform two-stage training of the deep reinforcement learning network

[0167] Step 7.1 Reaccumulate the experience replay pool: basically the same as step 4, except that a is used when selecting actions t =π * (s t ; θ), and no random action noise is added;

[0168] Step 7.2 Network retraining: It is basically the same as step 5, except that the number of training rounds epoch2 and the number of iterations per round step2 are reduced compared to the first stage of training.

[0169] Step 8: Deploy the trained deep reinforcement learning model to the actual robotic arm system.

[0170] The actual robotic arm system of this specific embodiment is composed of a motion capture system, a driver and a motor, an edge computing device, and a continuum robotic arm. Figure 4 As shown in the figure, the trained deep reinforcement learning model controls the actual robotic arm in the following steps:

[0171] Step 8.1, deploy the deep reinforcement learning model trained in step 7 to the edge computing device of the actual robotic arm system;

[0172] Step 8.2, obtain the current end position input state s of the robot arm through the motion capture system t , the deep reinforcement learning model in the edge computing device is based on the input state s t Output rope length change action command a t , the driver and motor perform action a t Control the movement of the robot arm and get the next input state s t+1 ;

[0173] Step 8.3, the next input state s t+1 As the current input state s t , repeat step 8.2 until the end of the continuum robot arm reaches the target position.

[0174] The present invention applies the deep reinforcement learning DDPG algorithm to the motion control of a continuum robot arm, by training the control strategy of the robot arm in continuous interaction with the environment. The present invention realizes the autonomous motion control of a continuum robot arm by constructing a virtual training environment for the continuum robot arm, establishing a deep reinforcement learning DDPG algorithm model, constructing a reward function calculation model, accumulating an experience replay pool, training a deep reinforcement learning network, and deploying the trained model to an actual robot arm system. The DDPG algorithm used in the present invention adopts a deterministic strategy, has good network convergence, reduces excessive Q estimates by using a target network, and adds random action noise, which is conducive to exploring the optimal strategy; using priority experience replay during training can speed up the training speed and improve the training effect; and through the domain randomization method, reduces the deviation between the training environment and the real environment, and improves the success rate of model deployment.

[0175] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A continuum robot arm motion control method based on deep reinforcement learning, characterized in that: The method comprises the following steps: Step 1, constructing a continuum robotic arm virtual training environment; Step 2: Establish a deep reinforcement learning DDPG algorithm model; specifically include: Step 2.1, establish the DDPG algorithm neural network: including the main network and the target network, where the main network includes the Actor network and the Critic network, and the target network includes the Target Actor network and the Target Critic network. The main network and the target network have the same network structure, and only the network parameters are different; the Actor network adopts a deterministic strategy, given an input state, and outputs a certain action information, expressed as π(s; θ), where s is the observation state of the robot arm, and θ is the Actor network parameter; the Critic network is an evaluation network, which is used to calculate the Q value to evaluate the quality of the strategy adopted by the Actor network, expressed as Q(s, a; w), where a is the action information output by the Actor network, and w is the Critic network parameter; the target network is used to calculate the target Q value to assist in updating the main network parameters. The Target Actor and Target Critic networks are expressed as π(s; θ - ) and Q(s,a;w - ), where θ - and w - are the parameters of the Target Actor network and the Target Critic network respectively; randomly initialize the parameters of the Actor network to θ = θ0 and the Critic network to w = w0, and initialize the parameters of the Target Actor network to θ - =θ and the parameter of the Target Critic network is w - =w; Step 2.2, establish the experience replay pool: the data stored in the experience pool is in the form of five-tuple (s t ,a t ,r t ,s t+1 ,δ t ), where s t represents the observed state at time t, a t Indicates state s t The action output obtained under the input, r t Indicates execution of action a t The reward value obtained, s t+1 Indicates execution of action a t The next state obtained after δ t is the time difference error, which is used to determine the priority of experience playback; if the experience pool capacity is C, the observation state dimension is dim s , the action information dimension is dim a , then use a two-dimensional matrix with C rows and D columns to store it, where D = 2·dim s +dim a +1+1, two numbers 1 represent r t and δ t Each occupies a storage space; and initializes each element of the two-dimensional matrix to 0; Step 3: Construct a reward function calculation model; the calculation of the reward function follows the following rules: Rule 1: If the robot moves beyond the working space, a penalty of -10 will be imposed; Rule 2: The reward value is related to the distance from the end of the robot to the target position. The closer the new end position is to the target position relative to the previous end position, the greater the reward value. Rule 3: If the distance from the end position to the target position is less than the set value d min , the end is considered to have reached the target position and a reward of +10 is given; According to the above rules, the reward value r t The calculation process is as follows: If the initial position of the end of the robot at time t is (x0 t ,y0 t ,z0 t ), the target position is (x1 t ,y1 t ,z1 t ), the robot arm is based on the current input state s t Execute action a t After updating, the new end position is (x0 t+1 ,y0 t+1 ,z0 t+1 ), then the reward function is as shown in formula (1): Step 4: Accumulate experience replay pool; specifically include: Step 4.1, observe the current virtual robot state s t , input to the Actor network to get action output When accumulating experience replay pool, in action Add random noise as the actual action, that is Where N t The mean is 0 and the variance is σ 2 Gaussian random noise; Step 4.2: The virtual environment robot performs action a t , update the robot state to get the next moment observation state s t+1 , and calculate the reward value r t ; Step 4.3, change the state s t and action a t Input into the Critic network to get the Q estimate value Q eval =Q(s t ,a t ; w), the state s t+1 Input into the Target Actor network to get the next action output a' of the network t+1 =π(s t+1 θ - ), and then the state s t+1 and action a′ t+1 Input into the Target Critic network to get Q(s t+1 ,a′ t+1 ;w - ), we can further get the target value of Q as Q target =r t +γ·Q(s t+1 ,a′ t+1 ;w - ), where γ is the discount factor, then the time difference error is calculated according to formula (2): δ t =Q eval -Q target =Q(s t ,a t ;w)-(r t +γ·Q(s t+1 ,a t+1 ;w - )) (2) Step 4.4, convert the quintuple (s t ,a t ,r t ,s t+1 ,δ t ) is stored in the experience replay pool; Step 4.5: Set the next state s t+1 As the current observation state of the robot arm s t , repeat steps 4.1 to 4.4; Step 4.6: Repeat step 4.5 until the experience replay pool is full. Step 5, perform one-stage training of the deep reinforcement learning network; Step 6, randomization of virtual environment domain; Step 7, perform two-stage training of the deep reinforcement learning network; Step 8: Deploy the trained deep reinforcement learning model to the actual robotic arm system.

2. The method for controlling the motion of a continuum manipulator based on deep reinforcement learning according to claim 1, characterized in that: Step 1 of constructing a continuum robotic arm virtual training environment specifically includes: Step 1.1, draw a three-dimensional model of the continuum robotic arm; Step 1.2: Initialize the virtual environment, set the coordinate system O-XYZ, define the robot working space, and adjust the camera position; Step 1.3, import the 3D model of the continuum robotic arm, add dynamic and kinematic properties to it, adjust the initial state of the robotic arm, and render the environment in which the robotic arm is located; Step 1.4, specify the observation state and action information of the robot arm, and set the robot arm update steps; Step 1.5, set up the communication connection between the virtual environment and the external algorithm model.

3. The method for controlling the motion of a continuum manipulator based on deep reinforcement learning according to claim 1, characterized in that: Step 5 describes the first phase of deep reinforcement learning network training, which specifically includes: Step 5.1, using the priority experience replay method to take N samples from the experience replay pool for training; specifically including: Step 5.1.1, according to |δ i The size of | gets the rank (i) of each sample in the experience replay pool, |δ i The larger the |, the smaller the value of rank(i); Step 5.1.2, extract N samples according to the sampling probability P(i) defined by formula (3): in, The parameter α controls the degree of priority sorting, α∈(0,1); Step 5.1.3, calculate the importance sampling weights of the N samples sampled according to formula (4): Where C is the capacity of the experience replay pool, P(i) is the sampling probability of sample i, and the parameter β controls the degree of using importance sampling weights, β∈(0,1); Step 5.2, update the Actor network parameters, specifically: set the state s i and action a i Input into the Critic network to get the Q estimate value Q(s i ,a i ; w), calculate the average policy gradient g of the sampled N experiences according to formula (5): Update the parameters θ of the Actor network based on gradient ascent, as shown in formula (6): θ←θ+ρ·g (6) Where ρ is the Actor network learning rate, ρ∈(0,1); Step 5.3, update the critic network parameters, specifically: The goal of updating the critic network is to minimize the temporal difference error, and the mean square temporal difference error weighted by the importance sampling weight is used as the loss function L, as shown in formula (7): Update the critic network parameter w based on gradient descent, as shown in formula (8): Where v is the Critic network learning rate, v∈(0,1); Step 5.4, update the Target Actor network parameters and Target Critic network parameters, specifically: Every K rounds, the target network is soft-updated using the Actor network and Critic network parameters, as shown in formula (9): Where τ is the soft update parameter, τ∈(0,1); Step 5.5, update the experience replay pool: discard the oldest five-tuple in the experience replay pool, and execute steps 4.1 to 4.4 to accumulate a new set of experience; Step 5.6, loop steps 5.1 to 5.5 in the total iteration step range, and the total number of loops is total_step1=step1*epoch1, where step1 is the number of iteration steps per round, and epoch1 is the number of training rounds.

4. The method for controlling the motion of a continuum manipulator based on deep reinforcement learning according to claim 3, characterized in that: The virtual environment domain randomization described in step 6 is specifically as follows: The action strategy trained in step 5 is applied to the virtual robotic arm and the actual robotic arm respectively to obtain the virtual observation trajectory and the actual observation trajectory. The difference between the two trajectories is minimized by adjusting the virtual environment system parameters.

5. The method for controlling the motion of a continuum manipulator based on deep reinforcement learning according to claim 4, characterized in that: The virtual environment domain randomization described in step 6 specifically includes the following process: Step 6.1, define the virtual robot system parameters to follow Gaussian distribution The distribution is determined by the parameters φ = (μ, Σ), where ξ represents the virtual robot system parameters, μ and Σ represent the mean and covariance of the Gaussian distribution, respectively. The probabilistic system dynamics of the virtual robot is expressed as And initialize φ=φ0; Step 6.2, resetting both the virtual robotic arm and the actual robotic arm to their initial states; Step 6.3: The action strategy π trained in step 5 * (s; θ) are applied to the virtual robot arm and the actual robot arm respectively, and the current states of the virtual robot arm and the actual robot arm are observed as o i,ξ and i,real , according to the action strategy, the action outputs are a i,ξ =π * (o i,ξ ;θ) and a i,real =π * (o i,real ;θ), after executing the action, the states at the next moment are o i+1,ξ and i+1,real ; Step 6.4, the next state o i+1,ξ and i+1,real , respectively as the current state o i,ξ and i,real , repeat step 6.3; Step 6.5: Repeat step 6.4 within the total iteration number T steps to obtain the virtual observation trajectory. and real-world observation trajectories The deviation function of two trajectories is calculated using the 2-norm as shown in formula (10): Where W is the importance weight matrix of each observation dimension; Step 6.6, virtual environment parameter update: minimize the objective function shown in formula (11): The virtual environment parameters are optimized by gradient descent, as shown in formula (12): Among them, λ is the virtual environment parameter update step, λ∈(0,1); Step 6.7: Repeat steps 6.2 to 6.6 within the total number of optimization rounds. The total number of rounds is E. Finally, the optimal parameter of the virtual environment is φ. * .

6. The method for controlling the motion of a continuum manipulator based on deep reinforcement learning according to claim 5, characterized in that: Step 7 describes the two-stage training of the deep reinforcement learning network. The specific process includes: Step 7.1, re-accumulate the experience replay pool. The specific process is the same as step 4, the difference is: when selecting an action, use a t =π * (s t ; θ), and no random action noise is added; Step 7.2: Network retraining. The process is the same as step 5, except that the number of training rounds epoch2 and the number of iterations per round step2 are reduced compared to the first stage of training.

7. The method for controlling the motion of a continuum manipulator based on deep reinforcement learning according to claim 6, characterized in that: Step 8 describes deploying the trained deep reinforcement learning model to the actual robotic arm system. The specific process includes: The actual robot arm system includes an observation module, a decision module, an execution module and a continuum robot arm; Step 8.1, deploy the deep reinforcement learning model trained in step 7 to the decision module of the actual robotic arm system; Step 8.2, obtain the current input state s of the robot through the observation module t , the deep reinforcement learning model in the decision module is based on the input state s t Output action command a t , the execution module performs action a t , get the next input state s t+1 ; Step 8.3, the next input state s t+1 As the current input state s t , repeat step 8.2 until the end of the robot arm reaches the target position.

Images