A deep reinforcement learning robotic grasping method for sparse rewards

Abstract

The invention discloses a deep reinforcement learning manipulator grasping method for sparse rewards, comprising: firstly analyzing the characteristics of the manipulator grasping task, modeling it as a Markov decision problem, designing a binary sparse reward, reducing the complexity of the reward function design, and reducing the design cost; secondly, taking the DDPG algorithm as the main deep reinforcement learning training algorithm framework, building an Actor-Critic structure network, and processing the continuous state action space; then, designing a post-experience playback mechanism, using the G-HGG algorithm for auxiliary target generation, using a pre-trained action network for action screening and adding exploration noise and energy functions to process the accumulated experience pool, enhancing the experience utilization rate, and improving the training efficiency and grasping success rate; finally, building a manipulator model and scene information, using interactive data for optimization training, and realizing manipulator target grasping. The method can solve the grasping learning problem of the manipulator under sparse rewards.

Description

Technical Field

[0001] The present invention relates to the field of robot arm grasping, and in particular to a deep reinforcement learning robot arm grasping method for sparse rewards. Background Art

[0002] Robotic grasping technology is one of the basic directions in the field of robotics. It plays a key role in the field of industrial automation. It can improve production efficiency, reduce labor costs, and realize automated production. Existing grasping control can be mainly divided into two categories: classical analysis method and data-driven method. The classical analysis method is a traditional target grasping method that requires the establishment of geometric physical models, contact models, and rigid body models, and kinematic, dynamic, and mechanical analysis and calculation. Its limitation is that the mathematical and physical modeling in some scenarios is very complex, and the modeling process is difficult to complete. In recent years, data-driven methods represented by deep reinforcement learning have developed rapidly, and scholars at home and abroad have begun to devote themselves to the research of data-driven robotic grasping methods.

[0003] Deep reinforcement learning combines the powerful feature extraction ability of deep learning with the excellent decision-making ability of reinforcement learning. It has the advantages of being able to cope with large-scale complex environments, strong generalization ability, and end-to-end learning. The emergence of Actor-Critic structure algorithms such as DDPG, SAC, and PPO can well solve the problem of continuous state action space, especially in grasping tasks, enabling the robot arm to accurately perform grasping actions in continuous space. In the field of robot grasping learning based on deep reinforcement learning, the agent optimizes decisions through rewards. For more complex scenarios, it is difficult for the agent to obtain positive rewards. Sparse reward signals will lead to slow network convergence and even training failure. There is still a lot of room for exploration on how to solve the problems caused by sparse rewards. Therefore, there is an urgent need to develop deep reinforcement learning robot grasping methods for sparse rewards. Summary of the invention

[0004] In view of the above-mentioned technical deficiencies, the purpose of the present invention is to provide a deep reinforcement learning robotic arm grasping method for sparse rewards, which uses the DDPG algorithm to explore deterministic strategies, can process the continuous state action space of the robotic arm, add random exploration noise to the action, enhance the exploration ability of the robotic arm, simplify the learning problem, reduce the design cost, have strong interpretability, improve the experience utilization rate, improve the grasping success rate, and enhance the network convergence speed and training efficiency.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] The present invention provides a deep reinforcement learning manipulator grasping method for sparse rewards, comprising the following steps:

[0007] Step 1: Analyze the characteristics of the robot grasping task, model it as a Markov decision problem, and design a binary sparse reward function;

[0008] Step 2: Based on the deep reinforcement learning DDPG algorithm, build the main network and target network of the Actor-Critic structure;

[0009] Step 3: Design a post-experience replay mechanism, use the G-HGG algorithm to generate auxiliary targets, use the pre-trained action network for action screening, add exploration noise and energy functions, and accumulate an experience replay pool;

[0010] Step 4: Build the robot arm model and scene information, use the interaction data to estimate the value function and update the strategy to obtain a deterministic strategy to complete the target grasping task.

[0011] Preferably, step 1 specifically includes the following contents:

[0012] According to the characteristics of the robot arm grasping task, define the quadruple Specifically:

[0013] S1-1, is the state space, which represents the set of information observed by the agent, specifically the position and speed of all joints of the robot, the position of the end effector, the target position, and the position and direction of the object;

[0014] S1-2, is the action space, which represents the set of operations performed by the agent, specifically represented by the three-dimensional coordinate increment of the end effector and the opening and closing of the gripper;

[0015] S1-3, r is a reward function, which is used to evaluate the effect of the agent in executing actions, so as to guide the agent to learn to achieve the expected goal. The reward function adopts the following binary sparse reward function:

[0016]

[0017] Where g is the auxiliary target generated by the algorithm, d is the Euclidean distance from the target object to the auxiliary target position, and d th is the minimum threshold of the distance;

[0018] S1-4, γ is the discount factor, which determines the importance the agent attaches to future rewards.

[0019] Preferably, step 3 specifically includes the following contents:

[0020] S3-1. Create an experience replay pool, add an energy function to the standard experience pool, and store data in the experience pool in the form of seven tuples (s t ||g,a t ,r t ,st+1 ||g,done,p,E traj ), train the main network and the target network according to batch sampling in the experience pool, where s t ||g is the state s of the agent at time t t The joint representation of a and target g, t represents the action taken by the agent at time t, r t Indicates that the agent is taking action a t The reward obtained after t+1 Indicates that the agent is in state s t Take action a t The state transferred at time t+1, done represents the task identification bit, which is a bool value, indicating whether the current round of tasks is completed or not, p is the playback priority of the current trajectory, E traj The total energy value of the experience track, which is used to determine the priority of experience playback;

[0021] S3-2. Create a graph model G = (V, E) for environmental information, where V is the set of nodes in the graph, and E is the set of edges in the graph. Where p1 and p2 are two possible nodes, and ω is the weight;

[0022] S3-3. Use Dijkstra algorithm to pre-calculate two nodes (p1, p2)∈P 2 The shortest distance between

[0023] S3-4. Load a set of pre-training strategies Define L goals Corresponding to K pre-training strategies;

[0024] S3-5. Based on the G-HGG algorithm, construct a set of M auxiliary targets

[0025] S3-6. Use the auxiliary target generated in step S3-5 Replace the original target (s0,g);

[0026] S3-7, at time t according to target o l Adjust the state and target (s′, g′) from the pre-trained policy Determine candidate actions And store the action in the action collection middle;

[0027] S3-8, repeat step S4-6 L times;

[0028] S3-9, repeat step S4-7 K times;

[0029] S3-10, the observed state s of the robot arm at time t t Input to the Actor network, output action according to the deterministic strategy π(s; θ) And add a mean of 0 and a variance of σ to the action 2 Gaussian noise N t , that is, the action at time t And store the action in the action collection middle;

[0030] S3-11. Select the optimal action according to formula (2) Execute the action and obtain the state s at time t+1 t+1 ;

[0031]

[0032] S3-12, repeat steps S4-8, S4-9 and S4-10 for a total of T times;

[0033] S3-13. Define the potential energy function E p (s t )=mgz t , where m is the mass of the object, g≈9.81m / s 2 represents the acceleration due to gravity, z t is the z-axis coordinate of the object at time t; the definition formula (3) is the kinetic energy function,

[0034]

[0035] Among them, (x t ,y t ,z t ) and (x t-1 ,y t-1 ,z t-1 ) are the three-dimensional coordinates of the target object at time t and t-1, respectively, and Δt is the sampling time interval of the robot arm;

[0036] The state transfer energy function is defined as formula (4):

[0037]

[0038] in, is the upper limit of the transfer energy, which is a set value, E(s t ) is the state energy function, defined as E(s t )=E p (s t )+E k (s t );

[0039] Calculate the empirical trajectory energy E according to the energy function traj , defined as formula (5);

[0040]

[0041] S3-14. Calculate the experience trajectory playback priority according to formula (6)

[0042]

[0043] S3-15. Calculate the reward r at time t t , and convert the sextuple Store in experience pool Sample a set of auxiliary targets g′ from the auxiliary target set;

[0044] S3-16. Calculate the auxiliary reward r at time t t ′, and auxiliary experience Store in experience pool

[0045] S3-17, looping through S4-15 for the auxiliary target g′;

[0046] S3-18, loop steps S4-14 and S4-16 for a total of T times.

[0047] Preferably, step 4 specifically includes the following contents:

[0048] S4-1, set the environment coordinate system O-XYZ, set the initial state of the robot arm, and determine the working space of the robot arm;

[0049] S4-2, setting scene information such as obstacles and target objects;

[0050] S4-3, determining the observation state information and action information of the robot arm, and determining the update time interval of the robot arm;

[0051] S4-4, interact with the environment based on the DDPG algorithm, and use the experience pool processing method in step 3 to accumulate the experience replay pool;

[0052] S4-5. From the Experience Replay Pool Minibatch with sampling size of batchsize

[0053] S4-6, according to p The samples are sorted in the , the larger the p value is, the higher the playback priority is;

[0054] S4-7, according to the playback priority The experience input network is trained, the state-action pair (s, a) is input into the Critic network, and the Q value Q(s, a; μ) of the state-action pair is calculated. μ is the Critic network parameter and is updated by minimizing the loss function.

[0055] S4-8, input the state s into the Actor network, and obtain the output action a based on the deterministic strategy π(s; θ), where θ is the Actor network parameter, updated by gradient descent;

[0056] S4-9. Soft update the parameters of the target network according to the parameters of the main network:

[0057] θ′=τ·θ+(1-τ)·θ′ (7)

[0058] μ′=τ·μ+(1-τ)·μ′ (8)

[0059] Where θ′ and μ′ are the network parameters of the target network, including the Target Actor network and the TargetCritic network, respectively, τ is the soft update amplitude of the network parameters, τ∈(0,1);

[0060] S4-10, update the experience replay pool, discard the oldest experience track in the experience pool, and accumulate a new set of experience;

[0061] S4-11, the training is repeated until the set number of rounds is reached, and then the training ends.

[0062] The beneficial effects of the present invention are:

[0063] 1) Use the DDPG algorithm to explore deterministic strategies, which can handle the continuous state action space of the robot arm, add random exploration noise to the action, and enhance the exploration ability of the robot arm;

[0064] 2) The reward function signal uses a binary sparse reward form, which reduces the complexity of reward function design, simplifies learning problems, reduces design costs, and has strong interpretability;

[0065] 3) Based on post-experience playback, the G-HGG algorithm is used for auxiliary target generation, the pre-trained action network is used for action screening and the energy function is added to process the experience pool to improve the experience utilization rate, improve the grasping success rate, and enhance the network convergence speed and training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 It is a framework diagram of a deep reinforcement learning manipulator grasping method for sparse rewards provided by the present invention;

[0068] Figure 2 It is a mujoco simulation environment scene information graph constructed in the embodiment;

[0069] Figure 3 is a schematic diagram of an environmental information graph model provided in an embodiment;

[0070] Figure 4 is a schematic diagram of the shortest path between nodes provided in the embodiment;

[0071] Figure 5 is a curve of the robot arm grasping training result in the embodiment. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] See also Figure 1 , a deep reinforcement learning robotic arm grasping method for sparse rewards, comprising the following steps:

[0074] Step 1: Analyze the characteristics of the robot grasping task, model it as a Markov decision problem, and design a binary sparse reward function;

[0075] Step 2: Based on the deep reinforcement learning DDPG algorithm, build the main network and target network of the Actor-Critic structure;

[0076] Step 3: Design a post-experience replay mechanism, use the G-HGG algorithm to generate auxiliary targets, use the pre-trained action network for action screening, add exploration noise and energy functions, and accumulate an experience replay pool;

[0077] Step 4: Build the robot arm model and scene information, use the interaction data to estimate the value function and update the strategy to obtain a deterministic strategy to complete the target grasping task.

[0078] Step 1 specifically includes the following:

[0079] According to the characteristics of the robot arm grasping task, define the quadruple Specifically:

[0080] S1-1, is the state space, which represents the set of information observed by the agent, specifically the position and speed of all joints of the robot, the position of the end effector, the target position, and the position and direction of the object;

[0081] S1-2, is the action space, which represents the set of operations performed by the agent, specifically represented by the three-dimensional coordinate increment of the end effector and the opening and closing of the gripper;

[0082] S1-3, r is a reward function, which is used to evaluate the effect of the agent in executing the action, so as to guide the agent to learn to achieve the expected goal. The reward function in the present invention adopts the following binary sparse reward function:

[0083]

[0084] Where g is the auxiliary target generated by the algorithm, d is the Euclidean distance from the target object to the auxiliary target position, and d th is the minimum threshold of the distance;

[0085] S1-4, γ is the discount factor, which determines the importance the agent attaches to future rewards.

[0086] Step 3 specifically includes the following:

[0087] S3-1. Create an experience replay pool, add an energy function to the standard experience pool, and store data in the experience pool in the form of seven tuples (s t ||g,a t ,r t ,s t+1 ||g,done,p,E traj ), train the main network and the target network according to batch sampling in the experience pool, where s t ||g is the state s of the agent at time t t The joint representation of a and target g, t represents the action taken by the agent at time t, r t Indicates that the agent is taking action a t The reward obtained after t+1 Indicates that the agent is in state s t Take action a tThe state transferred at time t+1, done is the task identification bit, which is a bool value, indicating whether the current round of tasks is completed or not, p is the playback priority of the current trajectory, E traj The total energy value of the experience track, which is used to determine the experience replay priority.

[0088] S3-2. Create a graph model G = (V, E) for environmental information, where V is the set of nodes in the graph, and E is the set of edges in the graph. Where p1 and p2 are two possible nodes, and ω is the weight;

[0089] S3-3. Use Dijkstra algorithm to pre-calculate two nodes (p1, p2)∈P 2 The shortest distance between

[0090] S3-4. Load a set of pre-training strategies Define L goals Corresponding to K pre-training strategies;

[0091] S3-5. Based on the G-HGG algorithm, construct a set of M auxiliary targets

[0092] S3-6. Use the auxiliary target generated in S3-5 Replace the original target (s0,g);

[0093] S3-7, at time t according to target o l Adjust the state and target (s′, g′) from the pre-trained policy Determine candidate actions And store the action in the action collection middle;

[0094] S3-8, repeat step S4-6 L times;

[0095] S3-9, repeat step S4-7 K times;

[0096] S3-10, the observed state s of the robot arm at time t t Input to the Actor network, output action according to the deterministic strategy π(s; θ) And add a mean of 0 and a variance of σ to the action 2 Gaussian noise N t , that is, the action at time t And store the action in the action collection middle;

[0097] S3-11. Select the optimal action according to formula (2) Execute the action and obtain the state s at time t+1 t+1 ;

[0098]

[0099] S3-12, repeat steps S4-8, S4-9 and S4-10 for a total of T times;

[0100] S3-13. Define the potential energy function E p (s t )=mgz t , where m is the mass of the object, g≈9.81m / s 2 represents the acceleration due to gravity, z t is the z-axis coordinate of the object at time t; the definition formula (3) is the kinetic energy function,

[0101]

[0102] Among them, (x t ,y t ,z t ) and (x t-1 ,y t-1 ,z t-1 ) are the three-dimensional coordinates of the target object at time t and t-1, respectively, and Δt is the sampling time interval of the robot arm;

[0103] The state transfer energy function is defined as formula (4):

[0104]

[0105] in, is the upper limit of the transfer energy, which is a set value, E(s t ) is the state energy function, defined as E(s t )=E p (s t )+E k (s t );

[0106] Calculate the empirical trajectory energy E according to the energy function traj , defined as formula (5);

[0107]

[0108] S3-14. Calculate the experience trajectory playback priority according to formula (6)

[0109]

[0110] S3-15. Calculate the reward r at time t t , and convert the sextuple Store in experience pool Sample a set of auxiliary targets g′ from the auxiliary target set;

[0111] S3-16. Calculate the auxiliary reward r at time t t ′, and auxiliary experience Store in experience pool

[0112] S3-17, looping through S4-15 for the auxiliary target g′;

[0113] S3-18, loop steps S4-14 and S4-16 for a total of T times.

[0114] Step 4 specifically includes the following:

[0115] S4-1, set the environment coordinate system O-XYZ, set the initial state of the robot arm, and determine the working space of the robot arm;

[0116] S4-2, setting scene information such as obstacles and target objects;

[0117] S4-3. Determine the observation status information and action information of the robotic arm, and determine the update time interval of the robotic arm.

[0118] S4-4, interact with the environment based on the DDPG algorithm, and use the experience pool processing method in step 3 to accumulate the experience replay pool;

[0119] S4-5. From the Experience Replay Pool Minibatch with sampling size of batchsize

[0120] S4-6, according to p The samples are sorted in the , the larger the p value is, the higher the playback priority is;

[0121] S4-7, according to the playback priority The experience input network is trained, the state-action pair (s, a) is input into the Critic network, and the Q value Q(s, a; μ) of the state-action pair is calculated. μ is the Critic network parameter and is updated by minimizing the loss function.

[0122] S4-8, input the state s into the Actor network, and obtain the output action a based on the deterministic strategy π(s; θ), where θ is the Actor network parameter, updated by gradient descent;

[0123] S4-9. Soft update the parameters of the target network according to the parameters of the main network:

[0124] θ′=τ·θ+(1-τ)·θ′ (7)

[0125] μ′=τ·μ+(1-τ)·μ′ (8)

[0126] Where θ′ and μ′ are the network parameters of the target network, including the Target Actor network and the TargetCritic network, respectively, τ is the soft update amplitude of the network parameters, τ∈(0,1);

[0127] S4-10, update the experience replay pool, discard the oldest experience track in the experience pool, and accumulate a new set of experience;

[0128] S4-11, the training is repeated until the set number of rounds is reached, and then the training ends.

[0129] In order to enable those skilled in the art to better understand the present invention, the deep reinforcement learning robotic arm grasping method for sparse rewards is described in detail below in combination with specific embodiments.

[0130] The simulation scene built using mujoco is as follows Figure 2 As shown, a 7-DOF robot arm of model KUKA LBR iiwa 7R800 equipped with a two-finger gripper is used to perform the task of grasping objects with obstacles. The main network Actor-Critic structure of the DDPG algorithm adopts the MLP network structure. The Actor network parameters include three fully connected hidden layers, each containing 256 neurons. The ReLU function is used as the nonlinear activation function of the hidden layer. The tanh function is used to output the continuous action strategy in the output layer. The Critic network parameters include three fully connected hidden layers, each containing 256 neurons. The ReLU function is used as the nonlinear activation function of the hidden layer. The output layer has only 1 neuron, and the activation function is not specified. It is used for value function prediction. The Actor-Critic structure of the target network is exactly the same as the main network, only the network parameters are inconsistent. Create an experience pool, and the data stored in the experience pool is in the form of seven tuples (s t ||g,a t ,r t ,s t+1 ||g,done,p,E traj ), train the main network and the target network according to batch sampling in the experience pool, where s t ||g represents the state s of the agent at time t t The joint representation of a and target g, t represents the action taken by the agent at time t, r t Indicates that the agent is taking action a t The reward obtained after t+1 Indicates that the agent is in state s t Take action a tThe state transferred at time t+1, done represents the task identification bit, which is a bool value, indicating whether the current round task is completed or not, and is the playback priority of the current trajectory. traj is the total energy value of the experience trajectory, which is used to determine the priority of experience playback. The graph model created for environmental information is as follows: Figure 3 As shown, Figure 4 is the shortest path between nodes calculated using the Dijkstra algorithm.

[0131] From the Experience Replay Pool Minibatch with sampling size of batchsize According to The samples in the rank order are sorted. The larger the p value, the higher the playback priority. The experience input network is trained. The state-action pair (s, a) is input into the Critic network, the Q value Q(s, a; μ) of the state-action pair is calculated, and the Critic network parameter μ is updated by minimizing the loss function. The state s is input into the Actor network to obtain the output action a based on the deterministic strategy π(s; θ), and the Actor network parameter θ is updated by gradient descent. The parameters of the target network are soft-updated using the main network parameters according to the following formula:

[0132] θ′=τ·θ+(1-τ)·θ′ (7)

[0133] μ′=τ·μ+(1-τ)·μ′ (8)

[0134] Update the experience replay pool, discard the oldest experience track in the experience pool, and accumulate a new set of experience; loop training until the training ends after the set round.

[0135] The training curve obtained after the training is as follows Figure 5 As shown, it can be seen from the training curve that the convergence speed is fast and the success rate is high, the target grasping task is achieved, and the feasibility of the present invention is verified.

[0136] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A deep reinforcement learning robotic arm grasping method for sparse rewards, characterized in that: The following steps are involved: Step 1: Analyze the characteristics of the robot grasping task, model it as a Markov decision problem, and design a binary sparse reward function; Step 2: Based on the deep reinforcement learning DDPG algorithm, build the main network and target network of the Actor-Critic structure; Step 3: Design a post-experience replay mechanism and use the G-HGG algorithm to generate auxiliary targets. The specific steps include: S3-1. Create an experience replay pool, add an energy function to the standard experience pool, and store data in the experience pool in the form of seven tuples (s t ||g,a t ,r t ,s t+1 ||g,done,p,E traj ), train the main network and the target network according to batch sampling in the experience pool, where s t ||g is the state s of the agent at time t t The joint representation of a and target g, t represents the action taken by the agent at time t, r t Indicates that the agent is taking action a t The reward obtained after t+1 Indicates that the agent is in state s t Take action a t The state transferred at time t+1, done represents the task identification bit, which is a bool value, indicating whether the current round of tasks is completed or not, p is the playback priority of the current trajectory, E traj The total energy value of the experience track, which is used to determine the priority of experience playback; S3-2. Create a graph model G = (V, E) for environmental information, where V is the set of nodes in the graph, and E is the set of edges in the graph. Where p1 and p2 are two possible nodes, and ω is the weight; S3-3. Use Dijkstra algorithm to pre-calculate two nodes (p1, p2)∈P 2 The shortest distance between S3-4. Load a set of pre-training strategies Define L goals Corresponding to K pre-training strategies; S3-5. Based on the G-HGG algorithm, construct a set of M auxiliary targets S3-6. Use the auxiliary target generated in step S3-5 Replace the original target (s0,g); S3-7, at time t according to target o l Adjust the state and target (s′, g′) from the pre-trained policy Determine candidate actions in And store the action in action set A t middle; S3-8, repeat step S4-6 L times; S3-9, repeat step S4-7 K times; Then, the pre-trained action network is used for action screening, and exploration noise and energy functions are added to accumulate the experience replay pool; Step 4: Build the robot arm model and scene information, use the interaction data to estimate the value function and update the strategy to obtain a deterministic strategy to complete the target grasping task.

2. The method according to claim 1, characterized in that Step 1 specifically includes the following: According to the characteristics of the robot arm grasping task, define the quadruple Specifically: S1-1, S is the state space, which represents the set of information observed by the agent, specifically the position and speed of all joints of the robot arm, the position of the end effector, the target position, and the position and direction of the object; S1-2, A is the action space, which represents the set of operations performed by the agent, specifically represented by the three-dimensional coordinate increment of the end effector and the opening and closing of the gripper; S1-3, r is a reward function, which is used to evaluate the effect of the agent in executing actions, so as to guide the agent to learn to achieve the expected goal. The reward function adopts the following binary sparse reward function: Where g is the auxiliary target generated by the algorithm, d is the Euclidean distance from the target object to the auxiliary target position, and d th is the minimum threshold of the distance; S1-4, γ is the discount factor, which determines the importance the agent attaches to future rewards.

3. The method according to claim 2, characterized in that In step 3, the pre-trained action network is used for action screening, and exploration noise and energy functions are added to accumulate the experience replay pool, which specifically includes the following: S3-10, the observed state s of the robot arm at time t t Input to the Actor network, output action according to the deterministic strategy π(s; θ) And add a mean of 0 and a variance of σ to the action 2 Gaussian noise N t , that is, the action at time t And store the action in action set A t middle; S3-11. Select the optimal action according to formula (2) Execute the action and obtain the state s at time t+1 t+1 ; a * =argmaxQ π (s t ,g,A t )(2) S3-12, repeat steps S4-8, S4-9 and S4-10 for a total of T times; S3-13. Define the potential energy function E p (s t )=mgz t , where m is the mass of the object, g≈9.81m / s 2 represents the acceleration due to gravity, z t is the z-axis coordinate of the object at time t; the definition formula (3) is the kinetic energy function, Among them, (x t ,y t ,z t ) and (x t-1 ,y t-1 ,z t-1 ) are the three-dimensional coordinates of the target object at time t and t-1, respectively, and Δt is the sampling time interval of the robot arm; The state transfer energy function is defined as formula (4): in, is the upper limit of the transfer energy, which is a set value, E(s t ) is the state energy function, defined as E(s t )=E p (s t )+E k (s t ); Calculate the empirical trajectory energy E according to the energy function traj , defined as formula (5); S3-14, calculating the experience trajectory playback priority p(T) according to formula (6); S3-15. Calculate the reward r at time t t , and convert the sextuple Store in the experience pool R, and sample a set of auxiliary targets g′ from the auxiliary target set; S3-16. Calculate the auxiliary reward r at time t t ′, and auxiliary experience Stored in experience pool R; S3-17, looping through S4-15 for the auxiliary target g′; S3-18, loop steps S4-14 and S4-16 for a total of T times.

4. The method according to claim 1, characterized in that Step 4 specifically includes the following: S4-1, set the environment coordinate system O-XYZ, set the initial state of the robot arm, and determine the working space of the robot arm; S4-2, setting obstacle and target object scene information; S4-3, determining the observation state information and action information of the robot arm, and determining the update time interval of the robot arm; S4-4, interact with the environment based on the DDPG algorithm, and use the experience pool processing method in step 3 to accumulate the experience replay pool; S4-5, sample minibatchB of size batchsize from the experience replay pool R; S4-6, sort the samples in B according to p, the larger the p value, the higher the playback priority; S4-7, input the experience in B into the network for training according to the playback priority, input the state-action pair (s, a) into the Critic network, calculate the Q value Q(s, a; μ) of the state-action pair, μ is the Critic network parameter, and is updated by minimizing the loss function; S4-8, input the state s into the Actor network, and obtain the output action a based on the deterministic strategy π(s; θ), where θ is the Actor network parameter, updated by gradient descent; S4-9. Soft update the parameters of the target network according to the parameters of the main network: θ′=τ·θ+(1-τ)·θ′ (7) μ′=τ·μ+(1-τ)·μ′ (8) Where θ′ and μ′ are the network parameters of the target network, including the TargetActor network and the TargetCritic network, respectively, τ is the soft update amplitude of the network parameters, τ∈(0,1); S4-10, update the experience replay pool, discard the oldest experience track in the experience pool, and accumulate a new set of experience; S4-11, the training is repeated until the set number of rounds is reached, and then the training ends.

Images