Dynamic target fast grasping planning method and system based on deep reinforcement learning
By constructing a neural network model through deep reinforcement learning and combining dense rewards and obstacle avoidance penalties, the problems of long planning time and low success rate in dynamic target grasping in traditional methods are solved, and the robotic arm can quickly and intelligently grasp dynamic targets.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2023-10-31
- Publication Date
- 2026-07-21
AI Technical Summary
Traditional grasping planning methods based on modeling and analysis have long planning times and low success rates in dynamic target grasping, and fail to take into account grasping posture and human behavioral standards.
A neural network model based on deep reinforcement learning is constructed. The robotic arm is trained to plan its grasping motion by using reward functions of dense reward, sparse reward, and obstacle avoidance penalty, taking into account grasping position and posture, to achieve six degrees of freedom grasping.
It enables robotic arms to quickly and intelligently grasp dynamic targets, improving the success rate of grasping and the ability of posture to conform to human behavioral standards.
Smart Images
Figure CN117245666B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of task planning technology, specifically a dynamic target fast grasping planning method and system based on deep reinforcement learning. Background Technology
[0002] Highly intelligent object grasping technology, achieving human-like dexterity in object grasping, has always been a key goal in robotics development. Robots have a wide range of needs for grasping dynamic targets, such as grabbing objects on a conveyor belt, picking up a rolling ball, picking up a bottle handed to them, or capturing a target in space. Therefore, research on methods for grasping dynamic targets is a crucial research area for realizing intelligent industrial production lines, intelligent service robots, and intelligent on-orbit space services. However, traditional grasping planning methods based on modeling and analysis rely on precise modeling of the controlled system, resulting in long planning times. These methods are unsuitable for rapid grasping planning of dynamic targets and have a low success rate. They often only consider how to grab the target, without considering how to grasp it effectively or how to grasp it in a way that better conforms to human behavioral standards. Summary of the Invention
[0003] To address the shortcomings of the existing technology, this invention provides a dynamic target fast grasping planning method and system based on deep reinforcement learning, which can effectively realize six-degree-of-freedom grasping of dynamic targets, thereby achieving intelligent dynamic target grasping by robotic arms.
[0004] To achieve the above objectives, this invention provides a dynamic target fast grasping and planning method based on deep reinforcement learning, comprising the following steps:
[0005] A neural network model is constructed for motion planning during the process of a robotic arm grasping a dynamic target. The input of the neural network model is the current pose of the robotic arm and the expected grasping pose of the dynamic target, and the output is the angle increment of each joint of the robotic arm.
[0006] In the simulation environment, the neural network model is trained based on a dynamic target with random motion speed and trajectory, and a reward function consisting of dense reward, sparse reward and obstacle avoidance penalty is constructed based on the pose relationship between the robotic arm and the target during the training process.
[0007] After training the neural network model, the robotic arm is used to plan dynamic target grasping actions based on the neural network model.
[0008] In one embodiment, the training process of the neural network model includes several training rounds, and each training round includes several training steps;
[0009] The termination condition for training the neural network model is reaching the maximum number of rounds, and the termination condition for each training round is reaching the maximum number of training steps or satisfying the grasping condition N times consecutively.
[0010] In one embodiment, the training process of the neural network model specifically includes the following steps:
[0011] Step 1: Let the number of training step iterations i = 1, the number of round iterations j = 1, and the number of times the grabbing condition is satisfied k = 0. Let the maximum number of training steps in the j-th training round be I. MAX，j The maximum number of rounds is J. MAX ;
[0012] Step 2: Initialize the robotic arm by rotating each joint to its initial position;
[0013] Step 3: Input the current pose of the robotic arm and its end effector and the desired grasping pose corresponding to the dynamic target into the neural network model, and control the robotic arm to execute the action output by the neural network model, wherein the action is the angle increment of each joint of the robotic arm;
[0014] Step 4: After the robotic arm executes the output action, calculate the reward function for the current training step and determine whether the current training step meets the grasping conditions.
[0015] If so, let k = k + 1 and then proceed to step 5;
[0016] Otherwise, set k=0 and proceed to step 5;
[0017] Step 5, determine whether k≥n holds true:
[0018] If so, update the reward function of the current training step, and store the task execution status of the previous training step, the action performed by the robotic arm in the previous training step, the task execution status of the current training step, and the reward function of the current training step into a set of experience data and store them in the experience pool, and then proceed to step 7.
[0019] Otherwise, the task execution status in step 3, the actions performed by the robotic arm, the task execution status of the current training step, and the reward function of the training step are combined into a set of experience data and stored in the experience pool, and then step 6 is performed.
[0020] Step 6, determine if i > J MAX，j Is it true or false?
[0021] If so, proceed to step 7;
[0022] Otherwise, let i = i + 1 and return to step 3;
[0023] Step 7: Let j = j + 1, and determine if j > J.MAX Is it true or false?
[0024] If so, terminate the training of the neural network model;
[0025] Otherwise, set i = 1 and return to step 2;
[0026] During steps 2 to 7, the network parameters of the neural network model are updated online based on the experience data collected in the experience pool.
[0027] In one embodiment, step 4, the process of determining whether the current training step meets the grasping conditions, is as follows:
[0028] After the robotic arm executes the action output by the neural network model, the distance between the end effector of the robotic arm and the target is calculated. If the distance is less than the preset grasping distance threshold, the grasping condition is met; otherwise, the grasping condition is not met.
[0029] In one embodiment, during steps 2 to 7, the network parameters of the neural network model are updated online based on the experience pool, specifically including:
[0030] The number of experience data in the experience pool is detected in real time. When the number of experience data reaches a preset threshold, a batch of experience data is extracted from the experience pool to update the network parameters of the neural network model.
[0031] In one embodiment, the reward function is:
[0032] R = λ1·R dense +λ2·R sparse +λ3·R avoid obstacle
[0033] Where R is the reward function, R dense For dense rewards, R sparse For sparse rewards, R zvoid obstacle For obstacle avoidance penalties, λ1, λ2, and λ3 are weighting coefficients.
[0034] In one embodiment, the dense reward specifically refers to:
[0035] R dense =W dist ·r dist +W air ·r dir
[0036] Where, r dist This is the proximity reward, representing the degree to which the robotic arm approaches the target in Euclidean distance after planning; r dirThis is the attitude approach reward, representing the degree to which the robotic arm, after planning, approaches the target in the grasping posture of the gripper at the end of the robotic arm; W dist and W dir r dist and r dir The corresponding weighting coefficients;
[0037] Specifically, r dist The Euclidean distance d between the end effector and the target pre-grab point after the robotic arm performs the action dist Divide by the Euclidean distance d between the robotic arm end effector and the target pre-grab point at the start of the round. initial_dist To obtain, that is:
[0038]
[0039] Where (x1,y1,z1) represents the three-dimensional coordinates of the center point of the end effector of the robotic arm in Cartesian space, and (x2,y2,z2) represents the three-dimensional coordinates of the pre-grabbing pose of the grasped target in Cartesian space at a future time.
[0040] r dir The directional difference d between the direction of the robotic arm's end effector and the desired grasping pose of the target after the robotic arm performs the action. dir The result is obtained by taking the absolute value and dividing by π, i.e.:
[0041]
[0042] Where (w1,x1,y1,z1) represents the quaternion of the direction of the end effector of the robotic arm, and (w2,x2,y2,z2) represents the quaternion of the expected pre-grasp posture of the grasped target at a future time.
[0043] In one embodiment, the sparse reward is specifically:
[0044]
[0045] Here, "self collision" represents interference or collision between joints of the robotic arm during the execution of the action in the current training step; "task successful" indicates that the current training step is the Nth training step that meets the grasping conditions, i.e., the target grasp is successfully completed; l1 is the first distance threshold, δ is the angle threshold, r1 represents the negative reward when interference or collision occurs during the execution of the action in the current training step, and r2 represents the negative reward after the robotic arm performs the action in the current training step. dist Positive reward when ≤l1, r3 represents the reward after r when the robotic arm performs an action in the current training step. dir Positive reward when ≤δ, r4 represents the positive reward after the robotic arm performs the action in the current training step. dist≤l1 and r dir The positive reward when ≤δ, r5 represents the positive reward when the Nth grab condition is met.
[0046] In one embodiment, the obstacle avoidance penalty is specifically as follows:
[0047]
[0048] Where, d m r is the minimum distance between each link of the robotic arm and the target. avoid_obstacle l1 is a scalar representing the weighting coefficient of obstacle avoidance reward, l2 is the second distance threshold, and l3 is the third distance threshold.
[0049] To achieve the above objectives, the present invention also provides a dynamic target fast grasping planning system based on deep reinforcement learning, which uses the above method for dynamic target fast grasping planning.
[0050] Compared with the prior art, the present invention has the following beneficial technical effects:
[0051] This invention constructs a neural network model for motion planning during the grasping of dynamic targets by a robotic arm. In a simulation environment, the neural network model is trained based on dynamic targets with random motion speeds and trajectories. During training, a reward function consisting of dense rewards, sparse rewards, and obstacle avoidance penalties is constructed based on the pose relationship between the robotic arm and the target. By simultaneously considering the grasping position and grasping posture in the dynamic target grasping task, the invention can effectively achieve six-degree-of-freedom grasping of dynamic targets, thereby realizing the intelligent grasping of dynamic targets by the robotic arm. Attached Figure Description
[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.
[0053] Figure 1 This is a flowchart of the dynamic target fast grasping and planning method in an embodiment of the present invention;
[0054] Figure 2 This is a flowchart illustrating the training process of the neural network model in an embodiment of the present invention.
[0055] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0057] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.
[0058] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0059] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two elements or the interaction between two elements, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0060] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.
[0061] like Figure 1 The above describes a dynamic target fast grasping and planning method based on deep reinforcement learning disclosed in this embodiment, which specifically includes the following steps:
[0062] A neural network model is constructed for motion planning during the process of a robotic arm grasping a dynamic target. The input of the neural network model is the current pose of the robotic arm and the expected grasping pose of the dynamic target, and the output is the angle increment of each joint of the robotic arm.
[0063] In the simulation environment, a neural network model is trained based on a dynamic target with random motion speed and trajectory. During the training process, a reward function consisting of dense reward, sparse reward and obstacle avoidance penalty is constructed based on the pose relationship between the robotic arm and the target.
[0064] After training the neural network model, the robotic arm is used to plan dynamic target grasping actions based on the neural network model.
[0065] It is worth noting that both the current pose of the robotic arm and the expected grasping pose of the dynamic target refer to the pose in the world coordinate system.
[0066] In this embodiment, during the training of the neural network model, the pose of the robotic arm can be obtained by acquiring images of the robotic arm through an external camera and then measuring them using conventional target localization methods. The dynamic target consists of the target object and a wooden board carrying the target object, which acts as a conveyor belt. The desired grasping pose of the dynamic target can be directly obtained through a pre-designed dynamic target motion trajectory. For example, a dataset containing a sequence of planar linear, circular, and sinusoidal path points can be created as the preset trajectory of the dynamic target. During training, the target pose can be obtained directly from the dataset based on the timestamp. In the simulation environment, the built-in `pybullet.createConstriaint()` function in the simulation software is used to create constraints between the conveyor belt and a point in the simulation environment space. Each time the conveyor belt moves, the position of the constraint is changed to the desired position using `pybullet.changeConstriaint()`, thus simulating the movement of the conveyor belt. During the simulation, the movement of the robotic arm is controlled by the `pybullet.setJointMotorControlArray()` function in the simulation software, which controls the rotation of the joint motors of the robotic arm. This function performs PD control at the underlying level to ensure that the target position given to the robotic arm is actually reached. In the specific application stage after the neural network model is trained, the pose of the robotic arm can also be obtained by acquiring images of the robotic arm through an external camera and then measuring it using conventional target localization methods; the desired pose of the dynamic target can be measured based on the historical trajectory of the dynamic target combined with conventional trajectory prediction methods. Among these, the target localization method and the trajectory prediction method are relatively common existing technologies and are not the focus of this embodiment, so they will not be described in detail here.
[0067] The basic principle of neural network model training in this embodiment is as follows: the neural network model makes decisions based on perceptual information obtained from the environment (i.e., the current pose of the robotic arm and the expected grasping pose of the dynamic target), the robotic arm performs actions according to the decisions, the environment passively changes due to the robotic arm's actions, the robotic arm receives rewards for its actions, and continuously improves its capabilities under the guidance of the rewards. During the interaction between the neural network model and the robotic arm environment, the two influence each other, and under the guidance of the reward function, the neural network model can progressively and freely control the movement of the robotic arm.
[0068] In practice, the training of the neural network model includes several training rounds, each of which consists of several training steps. The training of the neural network model terminates when the maximum number of rounds is reached, and the training round terminates when the maximum number of training steps is reached or the grasping condition is met N times consecutively. At the beginning of each training round, the robotic arm joint angles are reset to their initial positions. The conveyor belt carrying the target object randomly selects the initial position of its trajectory and transmits the current state (i.e., the current pose of the robotic arm and the desired pose of the dynamic target) to the neural network model. The neural network model takes the state as input and outputs the increment of the robotic arm joint angles. If the end effector of the robotic arm meets the grasping condition for N consecutive training steps, the grasping condition is determined to be met, and the grasping action is executed.
[0069] refer to Figure 2 The training process of the neural network model in this embodiment specifically includes the following steps:
[0070] Step 1: Let the number of training step iterations i = 1, the number of round iterations j = 1, and the number of times the capture condition is satisfied k = 0. Let the maximum number of training steps in the j-th round be I. MAX，j The maximum number of rounds is J MAX ;
[0071] Step 2: Initialize the robotic arm by rotating each joint to its initial position and randomly initializing the position of the dynamic target within the workspace of the robotic arm.
[0072] Step 3: Input the current pose of the robotic arm and its end effector and the expected grasping pose corresponding to the dynamic target into the neural network model. The neural network model outputs the action and controls the robotic arm to execute the action output by the neural network model. The action is the angle increment of each joint of the robotic arm.
[0073] Step 4: After the robotic arm performs the output action, calculate the reward function for the current training step and determine whether the current training step meets the grasping conditions.
[0074] If so, let k = k + 1 and then proceed to step 5;
[0075] Otherwise, set k=0 and proceed to step 5;
[0076] Step 5, determine whether k≥N holds true:
[0077] If so, update the reward function for the current training step, that is, add the reward for satisfying the grasping condition N times consecutively (i.e., the reward corresponding to task successful in sparse rewards) to the original reward function for the current training step. Then, take the task execution state of the previous training step (i.e., the current pose of the robotic arm and the target grasping pose input by the neural network), the action performed by the robotic arm in the previous training step (i.e., the angle increment of each joint of the robotic arm output by the neural network), the task execution state of the current training step (i.e., the current pose of the robotic arm and the target grasping pose after performing the action) and the reward function to form a set of empirical data and store it in the experience pool, and then proceed to step 7.
[0078] Otherwise, the task execution status of the previous training step, the action performed by the robotic arm in the previous training step, the task execution status of the current training step, and the reward function are combined to form a set of experience data and stored in the experience pool, and then step 6 is performed.
[0079] Step 6, determine if i > J MAX，j Is it true or false?
[0080] If so, proceed to step 7;
[0081] Otherwise, let i = i + 1 and return to step 3;
[0082] Step 7: Let j = j + 1, and determine if j > J. MAX Is it true or false?
[0083] If so, end the training of the neural network model;
[0084] Otherwise, set i = 1 and return to step 2;
[0085] During steps 2 to 7, the network parameters of the neural network model are updated online based on the experience data collected in the experience pool. Specifically, this includes:
[0086] The system detects the number of new experience data points added to the experience pool after the last network parameter update in real time. When the number of new experience data points reaches a preset threshold, the network parameters of the neural network model are updated based on all the new experience data points.
[0087] In the specific implementation process, the process of determining whether the current training step meets the grasping conditions is as follows: after the robotic arm executes the action output by the neural network model, the distance between the end effector of the robotic arm and the target is calculated. If the distance is less than the preset grasping distance threshold, the grasping conditions are met; otherwise, the grasping conditions are not met.
[0088] To enable the robotic arm to consider the target's posture during grasping and achieve six-DOF grasping of dynamic targets, unlike previous robotic arms that always grasped from top to bottom, the end effector of this embodiment explores various positions and directions simultaneously. Therefore, each link of the robotic arm may collide with the target object. Such unexpected collisions can lead to two problems: first, damage to both the robotic arm and the target object in the real-world environment; and second, changes in the target object's state beyond the grasping action. These unexpected state changes can cause significant fluctuations in rewards, drastically impacting policy learning and potentially leading to convergence problems. Therefore, this embodiment uses links 1 to 12 to represent the poses of the target object and each link of the robotic arm in the world coordinate system. M All of these are input as state variables to the neural network model, allowing the model to quickly understand its own posture. Here, M represents the number of links in the robotic arm.
[0089] In addition, there are some common parameters in reinforcement learning control of robotic arms, such as the 6D pose of the robotic arm's end effector in the world coordinate system. M+1 The 6D pose of the target object in the world coordinate system object 6D pose of pre-captured position pre_grasp The distance d between the end effector and the pre-grab position dist The absolute quaternion distance d in the direction dir And the rotation angles p1 to p of each joint of the robotic arm. M and angular velocity w1~w M Among them, link1~link M+1 There are M+1 one-dimensional arrays, each representing the 6D pose of the centroid of each link in the world coordinate system, i.e., the relationship between position and rotation. Rotation is represented by quaternions. Each array contains M+1 variables, as follows:
[0090] s={d dist ,d dir ,p1,…,p M pose object pose pre_grasp ,link1,…,link M ,link M+1}
[0091] The neural network simulates the motion as an increment of the joint angle of the robotic arm. Compared to directly setting the motion as the joint angle value of the robotic arm, this approach offers several advantages: it enables continuous control of the robotic arm, allows for fine-tuning of motion, increases the stability and smoothness of the robotic arm, and provides excellent interpretability, allowing for an intuitive understanding of the movement changes of the corresponding joints. This is extremely helpful for debugging, understanding, and adjusting the behavior of the robotic arm.
[0092] During training, to generate grasping poses for dynamic targets, this embodiment pre-computes a grasping pose database while all target objects remain static. This database is collected and evaluated in the pure grasping simulation software Graspit. First, simulated annealing is used to densely generate 5000 stable grasps for each target object. Then, all grasping poses are evaluated in Graspit, with each grasping action involving 50 grasps and lifting of the object. Random noise is added to the object during each grasp to simulate a real-world scenario and increase the reliability of the results. The success rate is used as a measure of the robustness of all grasping poses, and the top 100 reliable grasping poses are selected as the end effector grasping poses for the target's expected pose in subsequent training. In practice, a pre-computed reachability space can also be used to quickly rank grasping poses for a given predicted future pose of the target object.
[0093] In this embodiment, the reward function is:
[0094] R = λ1·R dense +λ2·R sparse +λ3·R avoid obstacle
[0095] Where R is the reward function, R dense For dense rewards, R sparse For sparse rewards, R avoid obstacle To avoid penalties, λ1, λ2, and λ3 are weighting coefficients.
[0096] Dense rewards are:
[0097] R dense =W dist ·r dist +W dir ·r dir
[0098] Where, r dist This is the proximity reward, representing the degree to which the robotic arm approaches the target in Euclidean distance after planning; r dir This is the attitude approach reward, representing the degree to which the robotic arm, after planning, approaches the target in the grasping posture of the gripper at the end of the robotic arm; W dist and W dir rdist and r dir The corresponding weighting coefficients;
[0099] Specifically, r dist The Euclidean distance d between the end effector and the target pre-grab point after the robotic arm performs the action dist Divide by the Euclidean distance d between the robotic arm end effector and the target pre-grab point at the start of the round. initial_dist To obtain, that is:
[0100]
[0101] Where (x1,y1,z1) represents the three-dimensional coordinates of the center point of the end effector of the robotic arm in Cartesian space, and (x2,y2,z2) represents the three-dimensional coordinates of the pre-grabbing pose of the grasped target in Cartesian space at a future time.
[0102] r dir The directional difference d between the direction of the robotic arm's end effector and the desired grasping pose of the target after the robotic arm performs the action. dir (Using absolute quaternion distance, ranging from [-π, π]) The absolute value is then divided by π to obtain the distance, i.e.:
[0103]
[0104] Where (w1,x1,y1,z1) represents the quaternion of the direction of the end effector of the robotic arm, and (w2,x2,y2,z2) represents the quaternion of the expected pre-grasp posture of the grasped target at a future time.
[0105] In this embodiment, the proportions of position and orientation in the dense reward are normalized. Placing the differences in position and orientation on the same order of magnitude helps to balance their contributions to the reward, preventing one factor from dominating other factors during training. It also ensures that continuous rewards maintain a consistent scale under different environments, such as different moving target trajectories, different initial positions, or different target objects, thereby enhancing the stability and convergence of the training process.
[0106] In addition to dense rewards, this embodiment also sets some discrete rewards (i.e., sparse rewards). These sparse rewards can help the neural network model further clarify the specific task. In this embodiment, if the distance error between the robotic arm end effector pose and the pre-grasping pose is within l1 (e.g., l1 = 1.5 cm) and the orientation error is within δ (e.g., δ = 15°), it is considered a successful tracking step in this training step, and a positive reward is given. If the robotic arm meets the grasping condition N times consecutively (e.g., N = 5 times), it is considered a successful task. If it successfully tracks N or more training steps consecutively, it is considered a successful task in meeting the grasping condition, and a large positive reward is given. This large positive reward should be greater than the sum of all other rewards that can be obtained in a training round; otherwise, the training may get stuck in a local extremum, thus failing to find the optimal solution. Because the tracking success condition involves two variables, the position difference and the orientation difference of the end effector, a positive reward is given separately when the position error is within l1 or the orientation difference is within δ. That is, the sparse reward in this embodiment is:
[0107]
[0108] Here, "self collision" represents interference or collision between joints of the robotic arm during the execution of the action in the current training step, and "task successful" indicates that the current training step is the Nth training step that meets the grasping conditions, i.e., the target grasping is successfully completed. l1 is the first distance threshold, δ is the angle threshold, r1 represents the negative reward when interference or collision occurs during the execution of the action in the current training step, and r2 represents the reward after the robotic arm performs the action in the current training step. dist Positive reward when ≤l1, r3 represents the reward after r when the robotic arm performs an action in the current training step. dir Positive reward when ≤δ, r4 represents the positive reward after the robotic arm performs the action in the current training step. dist ≤l1 and r dir The positive reward when ≤δ, r5 represents the positive reward when the Nth grab condition is met.
[0109] In practical applications, r1, r2, r3, r4, r5, l1, and δ can all be set according to the specific scenario. The key is to ensure that r5 is significantly greater than r1, r2, r3, and r4. For example, the sparse reward can be set as follows:
[0110]
[0111] Sparse rewards help provide more feedback signals to the task, and during training, the neural network model is not limited to satisfying a single condition but pays attention to more conditions. This helps improve the flexibility and adaptability of the neural network model, accelerates the learning speed, guides the neural network model to discover valuable actions more quickly, and can adapt to handle more tasks and scenarios.
[0112] Furthermore, to ensure the safety of the robotic arm and the target object, whenever one of the links of the robotic arm is close enough to the target object, and the minimum distance d between them is... m When the distance falls below a set threshold, an obstacle avoidance penalty is triggered. Specifically, this embodiment defines two types of space around the target object. The first type is a warning zone; if any link of the robotic arm is less than l3 (e.g., l3 = 2 cm) from the target object, a negative reward is received. The absolute value of this negative reward increases with d. m The decrease in d leads to an increase. m When the obstacle falls below l2 (e.g., l2 = 1 cm) by 1 cm, the robotic arm enters the danger zone, and the absolute value of the obstacle avoidance penalty reaches its maximum. The obstacle avoidance penalty in this embodiment is specifically as follows:
[0113]
[0114] Where, d m r is the minimum distance between each link of the robotic arm and the target. avoid_bostacle l1 is a scalar representing the weighting coefficient of obstacle avoidance reward, l2 is the second distance threshold, and l3 is the third distance threshold.
[0115] In summary, the dynamic target fast grasping planning method in this embodiment constructs a neural network model for motion planning during the robotic arm's grasping of dynamic targets. In a simulation environment, the neural network model is trained based on dynamic targets with random motion speeds and trajectories. During training, a reward function consisting of dense rewards, sparse rewards, and obstacle avoidance penalties is constructed based on the pose relationship between the robotic arm and the target. By simultaneously considering the grasping pose and grasping posture in the dynamic target grasping task, the method can effectively achieve six-degree-of-freedom grasping of dynamic targets, thereby realizing the intelligent grasping of dynamic targets by the robotic arm.
[0116] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.
Claims
1. A dynamic target fast grasping planning method based on deep reinforcement learning, characterized in that, Includes the following steps: A neural network model is constructed for motion planning during the process of a robotic arm grasping a dynamic target. The input of the neural network model is the current pose of the robotic arm and the expected grasping pose of the dynamic target, and the output is the angle increment of each joint of the robotic arm. In the simulation environment, the neural network model is trained based on a dynamic target with random motion speed and trajectory, and a reward function consisting of dense reward, sparse reward and obstacle avoidance penalty is constructed based on the pose relationship between the robotic arm and the target during the training process. After training the neural network model, the robotic arm is used to plan dynamic target grasping actions based on the neural network model. The training process of the neural network model includes several training rounds, and each training round includes several training steps; wherein, the termination condition for the training of the neural network model is reaching the maximum number of rounds, and the termination condition for each training round is reaching the maximum number of training steps or consecutive training rounds. The crawling condition is met once; the training process of the neural network model specifically includes the following steps: Step 1, let the number of training iterations be... Number of rounds / iterations Number of times the crawling conditions are met , order the The maximum number of training steps in a training round is The maximum number of rounds is ; Step 2: Initialize the robotic arm by rotating each joint to its initial position; Step 3: Input the current pose of the robotic arm and its end effector and the desired grasping pose corresponding to the dynamic target into the neural network model, and control the robotic arm to execute the action output by the neural network model, wherein the action is the angle increment of each joint of the robotic arm; Step 4: After the robotic arm executes the output action, calculate the reward function for the current training step and determine whether the current training step meets the grasping conditions. If so, let Then proceed to step 5; Otherwise, let Then proceed to step 5; Step 5, Determine Is it true or false? If so, update the reward function of the current training step, and store the task execution status of the previous training step, the action performed by the robotic arm in the previous training step, the task execution status of the current training step, and the reward function of the current training step into a set of experience data and store them in the experience pool, and then proceed to step 7. Otherwise, the task execution status in step 3, the actions performed by the robotic arm, the task execution status of the current training step, and the reward function of the training step are combined into a set of experience data and stored in the experience pool, and then step 6 is performed. Step 6, Determine Is it true or false? If so, proceed to step 7; Otherwise, let Then return to step 3; Step 7, let and judge Is it true or false? If so, terminate the training of the neural network model; Otherwise, let Then return to step 2; During steps 2 to 7, the network parameters of the neural network model are updated online based on the experience data collected in the experience pool.
2. The dynamic target fast grasping planning method based on deep reinforcement learning according to claim 1, characterized in that, In step 4, the process of determining whether the current training step meets the grasping conditions is as follows: After the robotic arm executes the action output by the neural network model, the distance between the end effector of the robotic arm and the target is calculated. If the distance is less than the preset grasping distance threshold, the grasping condition is met; otherwise, the grasping condition is not met.
3. The dynamic target fast grasping planning method based on deep reinforcement learning according to claim 1, characterized in that, During steps 2 to 7, the network parameters of the neural network model are updated online based on the experience pool, specifically including: The number of experience data in the experience pool is detected in real time. When the number of experience data reaches a preset threshold, a batch of experience data is extracted from the experience pool to update the network parameters of the neural network model.
4. The dynamic target fast grasping planning method based on deep reinforcement learning according to any one of claims 1 to 3, characterized in that, The reward function is: in, For the reward function, For dense rewards, For sparse rewards, To avoid the penalty, , , These are the weighting coefficients.
5. The dynamic target fast grasping planning method based on deep reinforcement learning according to claim 4, characterized in that, The dense reward specifically refers to: in, It is the position proximity reward, which represents the degree to which the robotic arm approaches the target in Euclidean distance after planning; It is the attitude approach reward, which represents the degree to which the robotic arm approaches the target in the grasping posture of the gripper at the end of the robotic arm after planning; and They are respectively and The corresponding weighting coefficients; Specifically, The Euclidean distance between the end effector and the target pre-grab point after the robotic arm performs the action Divide by the Euclidean distance between the robotic arm's end effector and the target's pre-grab point at the start of the round. To obtain, that is: in, This represents the three-dimensional coordinates of the center point of the robotic arm's end effector in Cartesian space. The three-dimensional coordinates of the target's pre-grabbing pose in Cartesian space at a future moment; The directional difference between the direction of the robotic arm's end effector after the robotic arm performs the action and the desired grasping pose of the target. Take the absolute value and divide by To obtain, that is: in, A quaternion representing the direction of the robotic arm's end effector. The quaternion represents the expected pre-grabbing posture of the target at a future moment.
6. The dynamic target fast grasping planning method based on deep reinforcement learning according to claim 5, characterized in that, The sparse reward specifically refers to: in, This indicates that during the current training step, interference or collisions occurred between the joints of the robotic arm. This indicates that the current training step is the Nth training step that meets the grasping condition, meaning that the target grasp has been successfully completed. The first distance threshold, For angle threshold, This represents the negative reward when the robotic arm interferes or collides during the execution of its actions in the current training step. This indicates the action performed by the robotic arm during the current training step. Timely rewards This indicates the action performed by the robotic arm during the current training step. Timely rewards This indicates the action performed by the robotic arm during the current training step. and Timely rewards This represents the positive reward when the Nth capture condition is met.
7. The dynamic target fast grasping planning method based on deep reinforcement learning according to claim 5, characterized in that, The specific penalties for obstacle avoidance are as follows: in, This represents the minimum distance between each link of the robotic arm and the target. The scalar represents the weighting coefficient of the obstacle avoidance reward. The second distance threshold, This is the third distance threshold.
8. A dynamic target fast grasping and planning system based on deep reinforcement learning, characterized in that, The method described in any one of claims 1 to 7 is used for dynamic target rapid grasping planning.