Method for training mechanical arm to move and grab target based on deep reinforcement learning
By combining priority ranking and dynamic back-look experience playback techniques with deep reinforcement learning, the problem of low sample sampling efficiency in the training of robotic arms for grasping moving targets was solved, achieving faster training speed and higher success rate.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YANSHAN UNIV
- Filing Date
- 2024-02-28
- Publication Date
- 2026-07-31
AI Technical Summary
In existing training methods for robotic arm moving target grasping, the dynamic back-view experience playback technique uses a random uniform sampling method, which results in low sample sampling efficiency and ignores the importance of transformation tuples, affecting the efficiency and success rate of the training process.
By calculating the priority of each relabeled experience, the experiences in the experience pool are sorted, and samples are taken according to priority during training to improve the efficiency of experience utilization. The training process is optimized by combining priority-based dynamic backward experience replay technology and deep reinforcement learning algorithms.
It accelerated the training speed of the robotic arm's moving target grasping task, improved the convergence speed of the task success rate, and increased the training success rate.
Smart Images
Figure CN118024244B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of robotic arm control and deep reinforcement learning, and in particular to a priority-based dynamic backward experience playback technique combined with a deep reinforcement learning method, thereby enabling a robotic arm to grasp a moving target in a simulated environment. Background Technology
[0002] Reinforcement learning is a process where an agent learns from its environment through trial and error. Rewards and penalties gained through interaction with the environment guide actions. The goal is to maximize the agent's reward. At each moment, the agent selects an action based on its policy and the current state. The environment responds to these actions, transitioning to a new state and generating a reward signal, typically a numerical value. The discounted sum of these rewards is called the reward. The reward is the objective the agent aims to maximize during action selection. To train a reliable agent, a reward function needs to be designed to help the agent complete training according to this function. However, for engineering projects where only the outcome is known but the process is difficult to describe, designing a suitable reward function is challenging. Post-Experience Replay (HER) proposes replacing the expected value of the training trajectory with the achieved goal from failed experiences. This method can additionally utilize failed experiences to enrich the learning repository. With this modification, any failed experience can receive a non-negative reward, allowing the agent to learn even when a task fails. This method performs well in solving problems related to robotic arm manipulation tasks. The later updated DHER method can handle moving object tracking and grasping tasks. However, for HER and DHER, the number of failed experiences gained after multiple training sessions is large, and their degree of closeness to the desired goal varies. These experiences are treated equally when sampled. However, not all failed experiences contribute equally to the training of the environment; some can only provide limited help.
[0003] In existing training methods for moving target grasping in robotic arms, the dynamic back-look experience playback technique cleverly solves the sparse reward problem and achieves tracking and grasping of dynamic targets. However, the method of randomly and uniformly sampling mini-batch from the transformed tuples leads to low sample sampling efficiency, which is not conducive to the subsequent training process.
[0004] In the initial experience collection process, transformation tuples were generated and stored by initializing the state and attaching randomly generated actions. Subsequently, some of the stored transformation tuples were relabeled. However, the number of relabeled transformation tuples is large and contains duplicates. The importance of each transformation tuple also varies. Simply randomly and uniformly sampling all relabeled transformation tuples to train the Q-network ignores the importance of experience. Randomly sampling a large number of transformation tuples mixed together is detrimental to the subsequent training process. Therefore, a method is needed to improve the efficiency of experience utilization, thereby increasing the convergence speed and training success rate. Summary of the Invention
[0005] The technical problem this invention aims to solve is to provide an optimized training method for robotic arm moving target grasping based on prioritized dynamic back-look experience playback. This method calculates the priority of each re-labeled experience, sorts the experiences in the experience pool, and samples them according to priority during training. This improves the efficiency of experience utilization, thereby increasing the convergence speed of the training success rate and accelerating the training process.
[0006] To solve the above-mentioned technical problems, the technical solution adopted by this invention is: an optimized training method for robotic arm moving target grasping based on deep reinforcement learning, the specific steps of which are as follows:
[0007] Step 1: Obtain the state vector through robotic arm environment initialization;
[0008] Step 2: Input the state vector into the neural network to obtain the probability distribution of the action, and select the action based on the probability distribution;
[0009] Step 3: Calculate the state vector of the next state by taking the action as input to the environment; and return the reward obtained by performing this action and the end flag indicating whether the environment has ended;
[0010] Step 4: Store the current state, selected action, next state, reward, and end flag as a transition tuple in an empty list. Determine if the maximum number of training steps has been reached. If the maximum number of training steps has not been reached, return to step 2. If the maximum number of training steps has been reached, stop the current training round.
[0011] Step 5: Sort the transformation tuples stored in the list by priority and store them in the experience pool;
[0012] Step 6: Select a portion of the transformation tuples stored in the list for remarking, then sort the remarked transformation tuples by priority and store them in the experience pool;
[0013] Step 7: Extract a batch of transformed tuples from the experience pool according to priority as parameters that need to be updated later;
[0014] Step 8: Calculate the loss function according to the temporal difference algorithm, and update the neural network parameters through gradient descent to reduce the loss. Return to step 1 until the maximum number of training sets is reached to complete the network model training. Then, test the trained model on the environment.
[0015] A further improvement of the technical solution of the present invention is that: in step 1, the state vector includes the position of the end of the robotic arm, the position and velocity of the operated object, the state of the gripper, and the position and velocity of the target object.
[0016] A further improvement to the technical solution of the present invention is as follows: Step 5 is specifically as follows:
[0017] Step 5.1: Set the current state s t Action a t The next state s t+1 And reward r t Transform it into a tuple and store it in an empty list;
[0018] Step 5.2: When the number of transformed tuples in the list reaches the set value, calculate the priority of the transformed tuples in the list and sort them by priority;
[0019] Step 5.3: Store the sorted transformation tuples in the list into the experience pool.
[0020] A further improvement to the technical solution of the present invention is as follows: the specific method for prioritizing the transformed tuples in step 5.2 is as follows:
[0021] Step 5.21: Calculate the TD-error for each transformed tuple;
[0022] δ t =Q t -(r t +γ·Q t+1 )
[0023] Where δ t For TD-error, γ is the discount factor, and Q t Let s be the current state at time step t. t Action a executed according to strategy π t Estimation of the value function;
[0024] Step 5.22: Set the current state s t Action a t The next state s t+1 And reward r t Substituting into equation 5.21, we obtain the following equation:
[0025] δ t =rt +γ t max a Q(s t+1 ,a)-Q(s t ,a t )
[0026] Step 5.23: Directly based on |δ t The sampling probability of each transformed tuple is determined by the following formula:
[0027] p t ∝|δ t |+ε
[0028] Where ε is a small constant greater than 0 used to guarantee p t >0.
[0029] A further improvement to the technical solution of the present invention is as follows: Step 6 is specifically as follows:
[0030] Step 6.1: Select the transformed tuples that need to be re-marked using the "future" method;
[0031] Step 6.2: Re-label the selected transformed tuples;
[0032] Step 6.3: Given a failure event, for the target trajectory achieved by the failure event, find an ideal target trajectory that can be matched from other events and then relabel the experience of the moving target task;
[0033] Step 6.4: Prioritize the re-marked transformed tuples and store them in the experience pool.
[0034] A further improvement to the technical solution of the present invention is that: in step 6.1, the "future" method specifically involves randomly selecting the targets corresponding to the k states after the current transformation tuple in the trajectory each time as the re-labeled targets.
[0035] A further improvement to the technical solution of this invention lies in the following: Step 6.3, finding a matching ideal target trajectory, specifically involves assembling a new event by matching the realized target trajectory of a given event with the expected target trajectory of the created event; Let... This represents the goal achieved by the agent at time step q in scenario i. Let E represent the desired objective at time step p in episode j; given a set of failed experiences {E1, E2, E3, E4, ...}, search for two failed episodes E i and E j (i≠j), satisfying use Replace E iThe expected goal in the equation is used to unite these two experiences, where j represents E. j , t≤min{p,q}; ultimately based on E i Assemble new experience E i It has a new "imagined" target trajectory. Where t≤min{p,q}.
[0036] A further improvement to the technical solution of the present invention is as follows: the specific method for prioritizing the transformed tuples in step 6.4 is as follows:
[0037] Step 6.41: Calculate the TD-error for each transformed tuple;
[0038] δ t =Q t -(r t +γ·Q t+1 )
[0039] Where δ t For TD-error, γ is the discount factor, and Q t Let s be the current state at time step t. t Action a executed according to strategy π t Estimation of the value function;
[0040] Step 6.42: Re-mark the state s t ||g', action a t The next state s t+1 ||g' and reward r t Substituting into equation 6.41, we obtain the following equation:
[0041] δ t =r t +γ t max a Q(S t+1 ||g',a)-Q(S t ||g',a t )
[0042] Step 6.43: Directly based on |δ t The sampling probability of each transformed tuple is determined by the following formula:
[0043] p t ∝|δ t |+ε
[0044] Where ε is a small constant greater than 0 used to guarantee p t >0.
[0045] A further improvement of the technical solution of the present invention is that: the network model trained in step 8 is used as an Actor-Critic network to train the robotic arm task operation environment.
[0046] The technological advancements achieved by this invention, due to the adoption of the aforementioned technical solution, are as follows: By establishing a reinforcement learning environment model for the problem of a robotic arm grasping a moving target, the current state information of the moving target is acquired using a visual sensor. This acquired state information is used as input to a neural network, which outputs a probability distribution of selectable actions and stores the state-action transition tuples in an experience pool. Subsequently, a priority-based dynamic back-look experience replay technique is used to sort and expand the experience pool and train the Actor-Critic network. Through continuous training iterations, the optimal strategy for the robotic arm grasping the moving target can be obtained. By using a training model based on the robotic arm grasping the moving target, considering that different transition tuples have their own priorities during the training process, sampling can be performed according to priority when the neural network samples before training, improving the efficiency of experience utilization and increasing the training speed of the robotic arm grasping task. By combining with deep reinforcement learning algorithms, a priority-based approach is introduced into the dynamic back-look experience replay technique, which accelerates the training speed of the robotic arm grasping the moving target, improves the convergence speed of the task success rate, and learns the optimal control strategy more quickly, providing a powerful tool for achieving higher levels of automated control. By introducing a non-uniform sampling strategy, the relabeled transformed tuples are labeled and sampled with priority. Attached Figure Description
[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0048] Figure 1 This is a training flowchart of the optimized training method for robotic arm moving target grasping according to the present invention;
[0049] Figure 2 This is a comparison chart of dynamic target tracking training results;
[0050] Figure 3 This is a comparison chart of dynamic target tracking and push training results. Detailed Implementation
[0051] The present invention will be further described in detail below with reference to embodiments:
[0052] like Figure 1The diagram shows a flowchart of an optimized training method for robotic arm moving target grasping based on prioritized dynamic back-look experience playback. The specific steps of the optimization training method based on deep reinforcement learning are as follows:
[0053] Step 1: Obtain the state vector through robotic arm environment initialization; initialize the environment and obtain the observed state vector through robotic arm environment initialization. The state vector includes the position of the robotic arm end effector, the position and velocity of the manipulated object, the gripper state, and the position and velocity of the target object. This state vector consists of a dictionary, which includes three keys: observation value, desired goal, and goal already achieved.
[0054] Step 2: Input the state vector into the neural network to obtain the probability distribution of the action, and select the action according to the probability distribution; in this embodiment, the neural network corresponding to the DDPG algorithm proposed by Lillicrap, Timothy P., et al. "Continuous control with deep reinforcement learning." arXiv preprint arXiv:1509.02971 (2015) is used.
[0055] Step 3: Calculate the state vector of the next state by taking the action as input to the environment; and return the reward obtained by performing this action and the end flag indicating whether the environment has ended; where the reward is a sparse reward.
[0056] Step 4: Store the current state, selected action, next state, reward, and end flag as a transition tuple in an empty list. Determine if the maximum number of training steps has been reached. If the maximum number of training steps has not been reached, return to step 2. If the maximum number of training steps has been reached, stop this round of training.
[0057] Step 5: Sort the transformation tuples stored in the list by priority and store them in the experience pool;
[0058] Step 5.1: Set the current state s t Action a t The next state s t+1 And reward r t Transform it into a tuple and store it in an empty list;
[0059] Step 5.2: When the number of transformed tuples in the list reaches the set value, calculate the priority of the transformed tuples in the list and sort them by priority;
[0060] Step 5.21: Calculate the TD-error for each transformed tuple;
[0061] δ t =Q t-(r t +γ·Q t+1 )
[0062] Where δ t For TD-error, γ is the discount factor, and Q t Let s be the current state at time step t. t Action a executed according to strategy π t Estimation of the value function;
[0063] Step 5.22: Set the current state s t Action a t The next state s t+1 And reward r t Substituting into equation 5.21, we obtain the following equation:
[0064] δ t =r t +γ t max a Q(s t+1 ,a)-Q(s t ,a t )
[0065] Step 5.23: Directly based on | δt| determines the sampling probability of each transformed tuple, and the calculation formula is shown below:
[0066] p t ∝|δ t |+ε
[0067] Where ε is a small constant greater than 0 used to guarantee p t >0.
[0068] Step 5.3: Store the sorted transformation tuples in the list into the experience pool.
[0069] Step 6: Select a portion of the transformation tuples stored in the list for remarking, then sort the remarked transformation tuples by priority and store them in the experience pool;
[0070] Step 6.1: Select the transformation tuples that need to be remarked using the "future" method; the "future" method specifically means that each time, the k targets corresponding to the current transformation tuple in the trajectory are randomly selected as the targets to be remarked.
[0071] Step 6.2: Remark the selected transformation tuples; when remarking the transformation tuples of dynamic targets, since dynamic targets cannot be directly regarded as completed targets, it is necessary to select a new set of failed task trajectories as completed targets. After experiencing a certain trajectory Then, each transformed tuple (s) of this segment t ,a t ,r t ,s t+1 ) is stored in the replay buffer, where s t Represents the state s at time step t. t a t Indicates an action, r t This represents a reward. Therefore, before t, there is a series of transformation tuples {(s0,a0,r0,s1),(s0,a0,r0,s1),...,(s t ,a t ,r t ,s t+1 The state consists of three parts: observation o t Expected goals and achieved goals That is
[0072] Step 6.3: Given a failure event, for the target trajectory achieved by the failure event, find a matching ideal target trajectory from other events to re-label the experience of the moving target task; the specific way to find a matching ideal target trajectory is to assemble a new event by matching the achieved target trajectory of the given event with the expected target trajectory of the created event; let... This represents the goal achieved by the agent at time step q in scenario i. Let E represent the desired objective at time step p in episode j; given a set of failed experiences {E1, E2, E3, E4, ...}, search for two failed episodes E i and E j (i≠j), satisfying use Replace E i The expected goal in the equation is used to unite these two experiences, where j represents E. j , t≤min{p,q}; ultimately based on E i Assemble new experience E i It has a new "imagined" target trajectory. Where t≤min{p,q}.
[0073] Step 6.4: Prioritize the re-marked transformed tuples and store them in the experience pool.
[0074] Step 6.41: Calculate the TD-error for each transformed tuple;
[0075] δ t =Q t -(r t +γ·Qt+1 )
[0076] Where δ t For TD-error, γ is the discount factor, and Q t Let s be the current state at time step t. t Action a executed according to strategy π t Estimation of the value function;
[0077] Step 6.42: Re-mark the state s t ||g', action a t The next state s t+1 ||g' and reward r t Substituting into equation 6.41, we obtain the following equation:
[0078] δ t =r t +γ t max a Q(S t+1 ||g',a)-Q(S t ||g',a t )
[0079] Step 6.43: Directly based on |δ t The sampling probability of each transformed tuple is determined by the following formula:
[0080] p t ∝|δ t |+ε
[0081] Where ε is a small constant greater than 0 used to guarantee p t >0.
[0082] Step 7: Extract a batch of transformed tuples from the experience pool according to priority as parameters that need to be updated later;
[0083] Step 8: Calculate the loss function according to the temporal difference algorithm, and update the neural network parameters through gradient descent to reduce the loss. Return to step 1 until the maximum number of training sets is reached to complete the network model training. Then, test the trained model on the environment.
[0084] This invention proposes a prioritized dynamic back-look experience replay technique combined with a deep reinforcement learning algorithm. This method improves the utilization efficiency of experience in the experience pool during training for robotic arm moving target grasping tasks, thereby increasing the training success rate. In this embodiment, the method is trained in a simulation environment and compared with unprioritized dynamic back-look experience replay. The final experimental results are as follows: Figure 2 As shown in Figure 3, Figure 2 For comparison of training results for dynamic target tracking, Figure 3 This study compares dynamic target tracking with improved training results. The results show that adding priority ranking improves training efficiency and experimental success rate.
[0085] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.
Claims
1. An optimized training method for robotic arm moving target grasping based on deep reinforcement learning, characterized in that: The specific steps are as follows: Step 1: Obtain the state vector through robotic arm environment initialization. The state vector includes the position of the robotic arm end effector, the position and velocity of the manipulated object, the gripper state, and the position and velocity of the target object. Step 2: Input the state vector into the neural network to obtain the probability distribution of actions, and select actions based on the probability distribution; Step 3: Calculate the state vector of the next state by taking the action as input to the environment; and return the reward obtained by performing this action and the end flag indicating whether the environment has ended; Step 4: Store the current state, selected action, next state, reward, and end flag as a transition tuple in an empty list. Determine if the maximum number of training steps has been reached. If the maximum number of training steps has not been reached, return to step 2. If the maximum number of training steps has been reached, stop the current training round. Step 5: When the number of transformation tuples in the list reaches the set value, calculate the TD-error of each transformation tuple in the list; determine the priority of each transformation tuple based on the TD-error, sort the transformation tuples stored in the list by priority and store them in the experience pool; Step 6: Select a portion of the transformation tuples stored in the list for remarking, then prioritize the remarked transformation tuples and store them in the experience pool; the "future" method is used to select transformation tuples that need to be remarked; given a failure event, for the target trajectory achieved by the failure event, find an ideal target trajectory that can be matched from other events, assemble a new event by matching the achieved target trajectory of the given event with the expected target trajectory of the creation event, and form an experience with a new "imagined" target trajectory based on the assembled new experience, thereby remarking the experience of the moving target task; calculate the TD-error of each remarked transformation tuple, determine the priority of each remarked transformation tuple according to the TD-error of the remarked transformation tuple, prioritize the remarked transformation tuples and store them in the experience pool; Step 7: Extract a batch of transformed tuples from the experience pool according to priority as parameters that need to be updated later; Step 8: Calculate the loss function according to the temporal difference algorithm, and update the neural network parameters through gradient descent to reduce the loss. Return to step 1 until the maximum number of training sets is reached to complete the network model training. Then, test the trained model on the environment.
2. The optimized training method for robotic arm moving target grasping based on deep reinforcement learning according to claim 1, characterized in that: Step 5 is detailed below: Step 5.1: Calculate the TD-error for each transformed tuple; in For TD-error, As a discount factor, The current state at time step t. According to the strategy Actions performed Estimation of the value function; Step 5.2: Set the current state ,action Next state and rewards Substituting into equation 5.1, we obtain the following equation: Step 5.3: Directly based on The sampling probability of each transformed tuple is determined by the following formula: in It is a small constant greater than 0 used to guarantee .
3. The optimized training method for robotic arm moving target grasping based on deep reinforcement learning according to claim 1, characterized in that: In step 6, the "future" method specifically involves randomly selecting the k targets corresponding to the current transition tuple in the trajectory as the relabeled targets each time.
4. The optimized training method for robotic arm moving target grasping based on deep reinforcement learning according to claim 1, characterized in that: Step 6 involves finding a matching ideal target trajectory by assembling a new event by matching the realized target trajectory of a given event with the expected target trajectory of the created event; let... This represents the goal achieved by the agent at time step q in scenario i. Represent the desired objective at time step p in plot j; given a set of failure experiences. By searching two failed plots and ,satisfy ;use replace The expected goal in the context is to unite these two experiences, where j represents... , Ultimately based on Assemble new experiences It has a new "imagined" target trajectory. ,in .
5. The optimized training method for robotic arm moving target grasping based on deep reinforcement learning according to claim 1, characterized in that: The specific method for prioritizing the re-tagged transformed tuples in step 6 is as follows: Step 6.1: Calculate the TD-error for each remarked transformation tuple; in For TD-error, As a discount factor, The current state at time step t. According to the strategy Actions performed Estimation of the value function; Step 6.2: Re-mark the state ,action Next state and rewards Substituting into equation 6.1, we obtain the following equation: Step 6.3: Directly based on The sampling probability of each relabeled tuple is determined by the following formula: in It is a small constant greater than 0 used to guarantee .
6. The optimized training method for robotic arm moving target grasping based on deep reinforcement learning according to claim 1, characterized in that: The trained network model in step 8 is used as the Actor-Critic network to train the robotic arm's task operation environment.