Mobile object robot arm grasping method based on multi-agent deep reinforcement learning

Abstract

The application discloses a kind of mobile object mechanical arm grabbing method based on multi-agent deep reinforcement learning.The present application includes the following steps: first, the image of mobile object on mobile object grabbing platform is captured in real time using camera, then the real-time target pose is obtained after target state extraction of mobile object image;Then a deep reinforcement learning model for mechanical arm to grab mobile object is established, and the real-time target pose is constantly controlled to continuously try to grab the target object, the model parameters are updated according to the grabbing result, so as to improve the success rate of grabbing, until the model training is completed;Finally, the real-time target pose of the mobile object to be grabbed is sent to the trained deep reinforcement learning model, and then the mechanical arm gripper is controlled to approach the mobile object and complete the grabbing task.The mobile object mechanical arm grabbing method proposed in the present application can successfully complete grabbing under different moving trajectories for different grabbing targets.

Description

Technical Field

[0001] This invention relates to a method for a robotic arm to grasp objects in the field of robotic arm control, and more particularly to a method for a robotic arm to grasp moving objects based on multi-agent deep reinforcement learning. Background Technology

[0002] Today, service robots are widely used in various fields such as healthcare, transportation, and machinery manufacturing. In the assembly field, robotic arms help workers grasp parts and tools, significantly improving assembly efficiency. Currently, the robotic arm grasping problem is a classic problem that has been extensively studied in recent years. However, most research focuses on grasping static objects, with very little research on grasping arbitrarily moving objects in irregular environments.

[0003] Robotic arm grasping methods can be categorized into analytical methods and data-driven methods. Early researchers employed analytical methods to perform robotic arm object grasping tasks. However, because analytical methods require extensive and precise prior knowledge of the moving object and its environment, data-driven methods have become increasingly prevalent in recent years. Data-driven methods train a control model to perform grasping tasks by building a dataset about robotic arm grasping. To avoid the drawbacks of deep learning, such as the significant effort required to build labeled datasets, limitations in model grasping performance due to the inability to accurately represent grasping postures as output values, and susceptibility to environmental influences, reinforcement learning has emerged as a new approach for moving object grasping. Classical reinforcement learning models learn through trial and error, using rewards gained from interactions with the environment to guide subsequent actions, aiming to maximize the agent's reward. Reinforcement learning-based moving object grasping models continuously attempt to grasp objects, seeking higher rewards and ultimately training to successfully grasp objects and obtain the highest reward. For example, one method (patent application publication number: CN112975977A) uses a depth camera to identify the position of the target object and the gripper of the robotic arm and transmits this information to a computer. The computer uses a weighted sum of the distance the robotic arm's end effector moves relative to the object, the number of steps the robotic arm takes, the sum of the rotation angles of each degree of freedom of the robotic arm's servo motors, and whether the target object was successfully grasped, as the reward mechanism for the DDPG deep deterministic policy gradient network. However, this method makes the reinforcement learning model prone to getting stuck in local optima and has poor exploration capabilities. Another method (patent application publication number: CN113752255A) uses a binocular camera to acquire images of the object on the control panel, performs object detection using the YOLO algorithm, and then trains the robotic arm through a reinforcement learning model to enable the robotic arm to complete the grasping action. This method is easy to implement, but it does not consider the training value of the samples for the model, which can easily reduce the training speed and may fail to converge.

[0004] In summary, research on moving object grasping based on reinforcement learning still faces many challenges, such as low real-time performance and poor applicability of control models, difficulty in deploying simulation results to real-world environments, and long training times. Summary of the Invention

[0005] To address the problems and needs in the background technology, this invention provides a method for grasping moving objects using a robotic arm based on multi-agent deep reinforcement learning. This method collects object information using an RGBD camera, finds the target's position in the image using visual detection algorithms and image processing techniques, calculates the target's three-dimensional position in the robotic arm's base coordinate system, and finally outputs the three-dimensional movement speed of the end effector through the MA-TD3H model to complete the moving object grasping task.

[0006] The technical solution of the present invention is as follows:

[0007] Step S1: Construct a moving object grasping platform, use an RGBD camera to capture images of moving objects on the moving object grasping platform in real time, and then extract the target state from the moving object images to obtain the real-time target pose.

[0008] Step S2: Establish a deep reinforcement learning model MA-TD3H for the robotic arm to grasp moving objects;

[0009] Step S3: The deep reinforcement learning model MA-TD3H continuously controls the robotic arm to try to grasp the target object based on the real-time target pose. The deep reinforcement learning model MA-TD3H updates the model parameters based on the grasping results, thereby improving the grasping success rate until the model training is completed.

[0010] Step S4: Send the real-time target pose of the moving object to be grasped to the trained deep reinforcement learning model MA-TD3H, and then control the robotic arm gripper to approach the moving object and complete the grasping task.

[0011] In step S1, after extracting the target state from the moving object image, the real-time target pose is obtained, specifically as follows:

[0012] S11: The moving object image includes an RGB object image and a depth image. The target region and contour are extracted from the RGB image to obtain the target contour map.

[0013] S12: Calculate the two-dimensional coordinates of the grasping center of the moving object in the camera coordinate system and the grasping angle of the robotic arm gripper based on the target contour map.

[0014] S13: Based on the coordinate transformation matrix from camera to robotic arm base, the two-dimensional coordinates of the gripping center in the camera coordinate system are transformed, and then combined with the depth image, the three-dimensional coordinates of the gripping center in the base coordinate system are obtained and recorded as the three-dimensional position of the moving object. The real-time target pose is composed of the three-dimensional position of the moving object and the gripping angle of the robotic arm gripper.

[0015] In step S11, the target image region in the RGB image is extracted using the YOLOv3 algorithm, and then the contour of the target image region is extracted using the Canny operator to obtain the target contour map.

[0016] Specifically, S12 is:

[0017] S121: Use Hough transform to extract straight lines in the target contour map, count the number of slopes corresponding to each straight line in each angle interval, and determine the gripping angle of the robotic arm gripper based on the angle interval where the number of slopes corresponding to the straight lines is the largest.

[0018] S122: Calculate the lengths from each point within the target contour map along the grasping angle and the angle perpendicular to the grasping angle to the edge of the contour, and then calculate the grasping confidence T for each point. The point with the highest grasping confidence T is taken as the grasping center. The formula for calculating the grasping confidence T for each point is as follows:

[0019]

[0020] Where l1 and l2 are the lengths from each point to the two edges of the contour along the same angle, respectively, and l3 and l4 are the lengths from each point to the two edges of the contour along another same angle, satisfying l1+l2>l3+l4; the angle is the gripping angle or an angle perpendicular to the gripping angle.

[0021] In step S2, the state s of the deep reinforcement learning model MA-TD3H for the robotic arm to grasp the moving object includes the three-dimensional position and velocity of the robotic arm gripper and the three-dimensional position and velocity of the moving object. Action a is the three-dimensional velocity of the robotic arm gripper. The reward is the total reward value r for a single step. The experience replay pool includes a normal experience replay pool and a high-quality experience replay pool.

[0022] The formula for the total reward value r in a single step is:

[0023] r = r dense +r sparse

[0024] r dense =-w dis *l dis

[0025] Where, r dense r is the single-step dense reward value.sparse The single-step sparse reward value; l dis Let w be the Euler distance between the robotic arm's end effector and the target being grasped in the world coordinate system. dis Distance is a factor influencing the reward value;

[0026] Single-step sparse reward value r sparse The calculation formula is as follows:

[0027] r sparse =r v +r g +r c +r t

[0028]

[0029]

[0030]

[0031] r t =-R t *t step

[0032] Where, r v The reward value is the L2 norm of the velocity of the robotic arm's end effector relative to the grasped target in the world coordinate system; r g r is the reward value when the robotic arm successfully grasps a moving object. c r is the penalty value when the robotic arm collides with the target. t R is a penalty term for time steps. v R is a fixed bonus value for speed. g R is the fixed reward value for a successful capture. c R is a fixed penalty value for collisions. t A fixed penalty value for each time step. and To capture the target's three-dimensional velocity values, and t represents the three-dimensional velocity value of the robotic arm's end effector. step This is the current training time step.

[0033] In step S3, during the training of the deep reinforcement learning model MA-TD3H for the robotic arm to grasp moving objects, when the single-step total reward value r of a certain experience is higher than the current reward threshold... If the experience is in the high-quality experience replay pool, it is placed in the ordinary experience replay pool; otherwise, it is placed in the ordinary experience replay pool. When updating parameters, the experience sampled in batches consists of the experience in the ordinary experience replay pool with a sampling ratio of t and the experience in the high-quality experience replay pool with a ratio of (1-t).

[0034] Preset multiple reward thresholds With a sampling ratio τ, during the training process of the deep reinforcement learning model MA-TD3H, after each preset number of training rounds, the deep reinforcement learning model MA-TD3H performs a preset number of grasping tests, and selects a corresponding set of numerical reward thresholds based on the current grasping success rate of the model. And the sampling ratio τ.

[0035] In step S3, after a preset number of training rounds, the deep reinforcement learning model MA-TD3H performs a grasping test on each agent in the model, and the agent network with the highest grasping success rate is recorded as the optimal agent network in the current test. Based on optimal actor network The parameters of other actor networks are updated using the following formula:

[0036]

[0037] in, For the updated actor network of other intelligent agents, For the actor network before updates by other agents, μ is the multi-agent update factor, and ε is the agent network before updates by other agents. i For the updated actor network of other agents L2 regularization.

[0038] In step S4, when the distance between the robotic arm gripper and the moving object to be grasped is less than a set distance threshold, the robotic arm gripper grasps the moving object to be grasped.

[0039] The beneficial effects of this invention are:

[0040] (1) The MA-TD3H model proposed in this invention can complete the grasping task for different targets under different motion trajectories. Through simulation tests and real experiments, it was found that the grasping success rate of the MA-TD3H model is better than that of other models.

[0041] (2) This invention incorporates a high-quality experience replay pool into the MA-TD3H model. This method can select the corresponding pool based on the current capture success rate of the MA-TD3H model. The value of τ can be used to improve the model's success rate in capturing data.

[0042] (3) This invention uses a multi-agent mechanism for reinforcement learning training, which enables the MA-TD3H model to converge faster, thereby improving training efficiency and capture success rate. Attached Figure Description

[0043] Figure 1 This invention provides a framework for a robotic arm method for grasping moving objects.

[0044] Figure 2 This is a flowchart of the MA-TD3H model executing a robotic arm grasping a moving object.

[0045] Figure 3 This is a flowchart for calculating the grasping posture.

[0046] Figure 4 This is a diagram illustrating coordinate system transformation during the grasping of moving objects.

[0047] Figure 5 This is a diagram of the experience replay pool update process.

[0048] Figure 6 It is a multi-agent mechanism update process.

[0049] Figure 7 This refers to the network update process of the MA-TD3H model.

[0050] Figure 8 It is a moving object grasping simulation environment based on Coppeliasim software.

[0051] Figure 9 This is a graph showing the changes in the capture success rate of the MA-TD3H, TD3, DDPG, and SAC models during training.

[0052] Figure 10 This is a graph showing the changes in total reward across rounds during the training process for the MA-TD3H, TD3, DDPG, and SAC models.

[0053] Figure 11 This describes the process of the UR robot grasping a moving object using the MA-TD3H model under Coppeliasim simulation.

[0054] Figure 12 This is a comparison chart of the capture success rates of the MA-TD3H, TD3, DDPG, and SAC models under straight-line trajectories.

[0055] Figure 13 This is a comparison chart of the capture success rates of the MA-TD3H, TD3, DDPG, and SAC models under oblique trajectory.

[0056] Figure 14 This is a comparison chart of the capture success rates of the MA-TD3H, TD3, DDPG, and SAC models under reciprocating trajectories.

[0057] Figure 15 This is a comparison chart of the average time for each capture operation of the MA-TD3H, TD3, DDPG, and SAC models under a straight trajectory.

[0058] Figure 16This is a comparison chart of the average time for each capture operation of the MA-TD3H, TD3, DDPG, and SAC models under the oblique trajectory.

[0059] Figure 17 This is a comparison chart of the average time for each capture operation of the MA-TD3H, TD3, DDPG, and SAC models under reciprocating trajectories.

[0060] Figure 18 It is a real-world experimental environment for capturing data.

[0061] Figure 19 This describes the process of the MA-TD3H model controlling the UR5e robot to grasp a moving object in a real experiment. Detailed Implementation

[0062] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0063] Figure 1 This is the framework of the robotic arm grasping method for moving objects based on multi-agent deep reinforcement learning in this invention. Figure 2 This invention provides a flowchart for using the MA-TD3H model proposed in this paper to perform a robotic arm grasping a moving object. The invention collects object information using an RGBD camera, locates the target in the image using visual detection algorithms and image processing methods, calculates the target's 3D position in the robotic arm's base coordinate system, and finally outputs the 3D movement velocity of the robotic arm's end effector using the deep reinforcement learning model MA-TD3H to complete the moving object grasping task. The invention includes the following steps:

[0064] Step S1: Construct a moving object grasping platform, use an RGBD camera to capture images of moving objects in the working area of ​​the moving object grasping platform in real time, and then extract the target state from the moving object images to obtain the real-time target pose.

[0065] like Figure 3 As shown, in step S1, after extracting the target state from the moving object image, the real-time target pose is obtained, specifically as follows:

[0066] S11: The moving object image includes an RGB object image and a depth image. The target region and contour are extracted from the RGB image to obtain the target contour map.

[0067] In step S11, the trained YOLOv3 algorithm is used to extract the target image region from the RGB image. Then, the Canny operator is used to extract the contour of the target image region, resulting in a target contour map. Specifically, 100 RGB images of the target are taken from different angles, and the target region in each image is labeled. All labeled images are then used as a dataset to train the YOLOv3 network, resulting in a trained YOLOv3 network.

[0068] S12: Calculate the two-dimensional coordinates of the grasping center of the moving object in the camera coordinate system and the grasping angle of the robotic arm gripper based on the target contour map.

[0069] S12 specifically refers to:

[0070] S121: Use Hough transform to extract straight lines from the target contour map, count the number of slopes corresponding to each straight line in each angle interval, and determine the gripping angle of the robotic arm gripper based on the angle interval with the most slopes corresponding to the straight lines. In specific implementation, the following angle intervals are set at 10° intervals: [0°, 10°), [10°, 20°), ..., [170°, 180°). The midpoint of the angle interval with the most slopes corresponding to the straight lines is recorded as the gripping angle of the robotic arm gripper.

[0071] S122: Calculate the lengths from each point within the target contour map along the grasping angle and the angle perpendicular to the grasping angle to the edge of the contour, and then calculate the grasping confidence T for each point. The point with the highest grasping confidence T is taken as the grasping center. The formula for calculating the grasping confidence T for each point is as follows:

[0072]

[0073] Where l1 and l2 are the lengths from each point to the two edges of the contour along the same angle, respectively, and l3 and l4 are the lengths from each point to the two edges of the contour along another same angle, satisfying l1+l2>l3+l4; the angle is the gripping angle or the angle perpendicular to the gripping angle.

[0074] S13: After performing coordinate transformation on the two-dimensional coordinates of the gripping center in the camera coordinate system based on the coordinate transformation matrix from the camera to the robotic arm base, and then combining it with the depth image, the three-dimensional coordinates of the gripping center in the base coordinate system are obtained and recorded as the three-dimensional position of the moving object. The real-time target pose is composed of the three-dimensional position of the moving object and the gripping angle of the robotic arm gripper. The three-dimensional position of the moving object is used as one of the target states and participates in the model training, while the gripping angle of the robotic arm gripper does not participate in the model training. The gripping angle of the robotic arm gripper is directly sent to the robotic arm and is only used to control the robotic arm gripper.

[0075] In practice, the coordinate transformation matrix from the camera to the robotic arm base is obtained through the following steps: First, the camera's intrinsic parameters are read. Then, the calibration plate is fixed to the robotic arm gripper. Multiple images are acquired by moving the gripper. Finally, the coordinate transformation matrix from the camera to the robotic arm base is calculated while maintaining the positional relationship between the gripper and the calibration plate. As shown in the depth map, at time t, the depth value of the target object relative to the camera coordinate system is... The pixel can be calculated from the camera's intrinsic parameters. and The coordinate system is transformed to the base coordinate system by applying the calculated coordinate transformation matrix from the camera to the robot arm base. Figure 4 This is a schematic diagram of coordinate system transformation.

[0076] Step S2: Establish a deep reinforcement learning model MA-TD3H for the robotic arm to grasp moving objects;

[0077] In step S2, the model includes a target actor network and an actor network, as well as two target critic networks and two critic networks. The actor network is used to input the current state and output actions during the interaction between the robotic arm and the environment. The two critic networks simultaneously evaluate the actions of the actor network to reduce overestimation. The target actor network and the two target critic networks obtain the results through soft updates and finally output the optimal policy. The MA-TD3H model simplifies the current situation of the robotic arm and moving objects in the environment into a state s as input, outputting action a to control the movement of the robotic arm gripper, and providing a reward value r under state s. After being influenced by the current environmental situation, the MA-TD3H model continuously tries to move the gripper in different directions to obtain higher rewards, ultimately learning the skill of grasping moving objects. The deep reinforcement learning model MA-TD3H for robotic arm grasping moving objects comprises the 3D position and velocity of the robotic arm gripper and the 3D position and velocity of the moving object, represented as a 1×12 dimensional vector. The action 'a' is the 3D velocity of the robotic arm gripper, represented as a 1×3 dimensional vector. Action 'a', along with the grasping angle, controls the robotic arm gripper's grasping motion. The reward is the total reward value 'r' for each step. The experience replay pool includes a normal experience replay pool and a high-quality experience replay pool.

[0078] This invention uses a combination of sparse and dense rewards to set the single-step reward value when performing moving object grasping in reinforcement learning. The formula for the total single-step reward value r is:

[0079] r = r dense +r sparse

[0080] r dense =-w dis *l dis

[0081] Where, r dense r is the single-step dense reward value. sparse The single-step sparse reward value; l dis Let w be the Euler distance between the robotic arm's end effector and the target being grasped in the world coordinate system. dis Distance is a factor influencing the reward value;

[0082] Regarding sparse rewards, a reward is given when the robotic arm reaches a predetermined goal, and a penalty is given when a certain adverse condition is triggered. When neither goal is achieved, the sparse reward value is set to 0 (R). v R g R c R t (All are fixed positive values). Single-step sparse reward value r sparse The calculation formula is as follows:

[0083] r sparse =r v +r g +r c +r t

[0084]

[0085]

[0086]

[0087] r t =-R t *t step

[0088] Where, r v The reward value is the L2 norm of the velocity of the robotic arm's end effector relative to the grasped target in the world coordinate system; r g r is the reward value when the robotic arm successfully grasps a moving object. c This is the penalty value when the robotic arm collides with the target. To ensure that the control model can complete the grasping task more quickly, this invention adds a penalty term for the time step. t R is a penalty term for time steps. v R is a fixed bonus value for speed. g R is the fixed reward value for a successful capture. c R is a fixed penalty value for collisions. t A fixed penalty value for each time step. and To capture the target's three-dimensional velocity values, and t represents the three-dimensional velocity value of the robotic arm's end effector. step This is the current training time step.

[0089] Step S3: The deep reinforcement learning model MA-TD3H continuously controls the robotic arm to try to grasp the target object based on the real-time target pose. The deep reinforcement learning model MA-TD3H updates the model parameters based on the grasping results, thereby improving the grasping success rate until the model training is completed.

[0090] During the training of the deep reinforcement learning model MA-TD3H for robotic arms to grasp moving objects, when the total reward value r of a single step of an experience exceeds the current reward threshold... If the experience is in the high-quality experience replay pool, it is placed in the ordinary experience replay pool; otherwise, it is placed in the ordinary experience replay pool. When updating parameters, the experience sampled in batches consists of the experience in the ordinary experience replay pool with a sampling ratio of τ and the experience in the high-quality experience replay pool with a ratio of (1-τ).

[0091] like Figure 5 As shown, multiple reward thresholds are preset. Given a sampling ratio τ, during the training process of the deep reinforcement learning model MA-TD3H, after a preset number of training rounds (e.g., 200 rounds), the deep reinforcement learning model MA-TD3H performs a preset number of grasping tests (e.g., 50 times). Based on the current grasping success rate of the model, a corresponding set of numerical reward thresholds is selected. And the sampling ratio τ, specifically: when the capture success rate is low, set a smaller reward threshold. With a sampling ratio τ, a larger reward threshold is set when the capture success rate is high. And the sampling ratio t, which is selected during training based on the results of each successful object capture test. and τ.

[0092] Setting up a multi-agent training mechanism

[0093] This invention applies a multi-agent structure to the TD3 model. This structure allows the model to learn using additional global information, while making decisions using local information during application. To better collect sample information, this invention introduces asynchronous noise into the agents under different environments, specifically Gaussian noise with different parameters.

[0094] Simultaneously run multiple simulation environments, each with a separate agent that interacts with the environment and is trained. For example... Figure 6 As shown, after a preset number of training rounds (e.g., 200 rounds), the deep reinforcement learning model MA-TD3H performs a grasping test on each agent in the model. The agent network with the highest grasping success rate is recorded as the optimal agent network in the current test. Based on optimal actor network The parameters of other actor networks are updated using the following formula:

[0095]

[0096] in, For the updated actor network of other intelligent agents, For the actor network before updates by other agents, μ is the multi-agent update factor, and ε is the agent network before updates by other agents.i For the updated actor network of other agents L2 regularization.

[0097] MA-TD3H Model Training Strategy

[0098] The goal of the MA-TD3H model is to find the optimal policy p(θ) with network parameters θ. * This strategy maximizes the payoff function. In the actor-commentator framework, the policy function is updated using a deterministic policy gradient algorithm.

[0099] Data (s) t ,a t ,r t ,s t+1 The values ​​of , is_done) are stored in the buffer. t Let a be the state at time t. t The action output at time t, r t Let s be the single-step reward at time t. t+1 The state at time t+1 is indicated by `is_done`, which marks whether the current crawling attempt has ended. When the amount of data stored in the buffer exceeds a certain limit, random batch sampling data is used for network updates. Based on the current model's crawling success rate, data corresponding to that success rate is sampled from both the standard experience replay pool and the high-quality experience replay pool.

[0100] When updating the critic network, using a deterministic strategy is easily affected by inaccuracies caused by function approximation errors. Therefore, target actor smoothing regularization is introduced. Assuming the model transitions from state s to state s', in state s', the target actor network is used. The loss function of the actor network is defined as the difference between the single-step reward value and the sum of the rewards of the two critic networks in state s', and the reward of the target critic network in state s. This loss function is then updated by backpropagation. The critic network is updated by backpropagation using the difference between the value functions obtained by the critic network and the target critic network in the same state. Since the error of the actor network affects the estimation of the critic network, to reduce the volatility of the actor network, the update frequency of the actor network is set to be lower than that of the critic network. After each training epoch, the target actor network and the target critic network are updated using a soft update method, such as... Figure 7 As shown. Figure 7 The network update process of the MA-TD3H model is demonstrated.

[0101] Step S4: Send the real-time target pose of the moving object to be grasped to the trained deep reinforcement learning model MA-TD3H. Specifically, integrate the position and velocity of the robotic arm gripper and the grasped target into the current state input to the MA-TD3H model. The model outputs the motion velocity to the robotic arm. Combined with the grasping angle received by the robotic arm, control the gripper to continuously approach and grasp the moving object to be grasped, thereby completing the grasping task. Specifically, when the distance between the robotic arm gripper and the moving object to be grasped is less than a set distance threshold, the robotic arm implements the grasping command, so that the robotic arm gripper grasps the moving object to be grasped.

[0102] To further illustrate the effectiveness of the robotic arm grasping method for moving objects based on multi-agent deep reinforcement learning, this embodiment verifies the effectiveness of the method through simulation experiments and real experiments.

[0103] A simulation environment for moving object grasping based on Coppeliasim software, such as... Figure 8 As shown, it includes the UR5 robot, the moving object to be grasped, and an RGBD camera for collecting information.

[0104] The MA-TD3H model proposed in this invention, along with the DDPG, TD3, and SAC models, were trained for 5000 rounds in a simulation environment. The changes in the total reward value of the four models during training were obtained. After every 200 training rounds, 50 grasping tests were performed on the current model, and the grasping success rate of the current model was collected. After each of the four models was trained repeatedly for 5 rounds, Figure 9 and Figure 10 The results are the success rate and reward value.

[0105] After training moving object grasping models using MA-TD3H, TD3, DDPG, and SAC models, straight-line, diagonal, and reciprocating motion trajectories (target variable-speed motion) are designed. The target grasping speed settings are as follows:

[0106] Straight line trajectory:

[0107] Diagonal trajectory:

[0108] Reciprocating trajectory:

[0109] Where i = 1, 2... 7, and step is the time step of the target object's motion.

[0110] For each trajectory, perform 200 capture tests using each of the four models mentioned above. Figure 11 The grasping process in Coppeliasim simulation. Figure 12 , Figure 13 and Figure 14To determine the success rate of crawling under the three paths, Figure 15 , Figure 16 and Figure 17 The figure represents the average time to complete each grasp under the three paths. When the target moves along the three different paths, the MA-TD3H model achieves a near 100% success rate in grasping the moving object, and takes the shortest time. In summary, the MA-TD3H model outperforms other models in grasping moving objects in the simulation environment.

[0111] Real-world grasping experimental environment, such as Figure 18 As shown in Table 1, the results of 50 moving object grasping experiments were conducted using the MA-TD3H, TD3, DDPG, and SAC methods on three different trajectories with three grasping targets. Figure 19 This describes the grasping process of a moving object in the real world. Tracking failure refers to the failure of the end effector to accurately track the moving target, while grasping failure refers to the failure of the robotic arm's end effector to accurately track the moving target but fail to grasp it.

[0112] Table 1 shows the results of 50 moving object grasping experiments conducted using various methods and three grasping targets under three different trajectories.

[0113]

[0114]

[0115] As shown in Table 1, for different capture targets and movement paths, the MA-TD3H model is more stable and has a higher capture success rate than other models.

[0116] In summary, the mobile object grasping method based on multi-agent deep reinforcement learning proposed in this invention is effective and feasible, and can successfully complete the grasping task when the robotic arm moves under different grasping targets and different trajectories.

[0117] The above examples are merely illustrative of the technical content of the present invention to facilitate easier understanding by the reader, but do not imply that the implementation of the present invention is limited to these examples. Any technical extensions or re-creations made based on the present invention are protected by the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A method for grasping moving objects using a robotic arm based on multi-agent deep reinforcement learning, characterized in that, Includes the following steps: Step S1: Construct a moving object grasping platform, use an RGBD camera to capture images of moving objects on the moving object grasping platform in real time, and then extract the target state from the moving object images to obtain the real-time target pose. Step S2: Establish a deep reinforcement learning model MA-TD3H for the robotic arm to grasp moving objects; Step S3: The deep reinforcement learning model MA-TD3H continuously controls the robotic arm to try to grasp the target object based on the real-time target pose. The deep reinforcement learning model MA-TD3H updates the model parameters based on the grasping results, thereby improving the grasping success rate until the model training is completed. Step S4: Send the real-time target pose of the moving object to be grasped to the trained deep reinforcement learning model MA-TD3H, and then control the robotic arm gripper to approach the moving object and complete the grasping task. In step S2, the state s of the deep reinforcement learning model MA-TD3H for the robotic arm to grasp the moving object includes the three-dimensional position and velocity of the robotic arm gripper and the three-dimensional position and velocity of the moving object. Action a is the three-dimensional velocity of the robotic arm gripper. The reward is the total reward value r for a single step. The experience replay pool includes a normal experience replay pool and a high-quality experience replay pool. The formula for the total reward value r in a single step is: r＝r dense +r sparse r dense ＝-w dis *l dis Where, r dense r is the single-step dense reward value. sparse The single-step sparse reward value; l dis Let w be the Euler distance between the robotic arm's end effector and the target being grasped in the world coordinate system. dis Distance is a factor influencing the reward value; Single-step sparse reward value r sparse The calculation formula is as follows: r sparse ＝r v +r g +r c +r t r t =-R t *t step Where, r v The reward value is the L2 norm of the velocity of the robotic arm's end effector relative to the grasped target in the world coordinate system; r g r is the reward value when the robotic arm successfully grasps a moving object. c r is the penalty value when the robotic arm collides with the target. t R is a penalty term for time steps. v R is a fixed bonus value for speed. g R is the fixed reward value for a successful capture. c R is a fixed penalty value for collisions. t A fixed penalty value for each time step. and To capture the target's three-dimensional velocity values, and t represents the three-dimensional velocity value of the robotic arm's end effector. step This is the current training time step.

2. The method for grasping moving objects by a robotic arm based on multi-agent deep reinforcement learning according to claim 1, characterized in that, In step S1, after extracting the target state from the moving object image, the real-time target pose is obtained, specifically as follows: S11: The moving object image includes an RGB object image and a depth image. The target region and contour are extracted from the RGB object image to obtain the target contour map. S12: Calculate the two-dimensional coordinates of the grasping center of the moving object in the camera coordinate system and the grasping angle of the robotic arm gripper based on the target contour map. S13: Based on the coordinate transformation matrix from camera to robotic arm base, the two-dimensional coordinates of the gripping center in the camera coordinate system are transformed, and then combined with the depth image, the three-dimensional coordinates of the gripping center in the base coordinate system are obtained and recorded as the three-dimensional position of the moving object. The real-time target pose is composed of the three-dimensional position of the moving object and the gripping angle of the robotic arm gripper.

3. The method for grasping moving objects by a robotic arm based on multi-agent deep reinforcement learning according to claim 2, characterized in that, In step S11, the target image region in the RGB object image is extracted using the YOLOv3 algorithm, and then the contour of the target image region is extracted using the Canny operator to obtain the target contour map.

4. The method for grasping moving objects by a robotic arm based on multi-agent deep reinforcement learning according to claim 2, characterized in that, Specifically, S12 is: S121: Use Hough transform to extract straight lines in the target contour map, count the number of slopes corresponding to each straight line in each angle interval, and determine the gripping angle of the robotic arm gripper based on the angle interval where the number of slopes corresponding to the straight lines is the largest. S122: Calculate the lengths from each point within the target contour map along the grasping angle and the angle perpendicular to the grasping angle to the edge of the contour, and then calculate the grasping confidence T for each point. The point with the highest grasping confidence T is taken as the grasping center. The formula for calculating the grasping confidence T for each point is as follows: Where l1 and l2 are the lengths from each point to the two edges of the contour along the same angle, respectively, and l3 and l4 are the lengths from each point to the two edges of the contour along another same angle, satisfying l1+l2>l3+l4; the angle is the gripping angle or an angle perpendicular to the gripping angle.

5. The method for grasping moving objects by a robotic arm based on multi-agent deep reinforcement learning according to claim 1, characterized in that, In step S3, during the training of the deep reinforcement learning model MA-TD3H for the robotic arm to grasp moving objects, when the single-step total reward value r of a certain experience is higher than the current reward threshold... If the experience is in a certain condition, it is placed in the high-quality experience replay pool; otherwise, it is placed in the ordinary experience replay pool. When updating parameters, the experience sampled in batches consists of the experience in the ordinary experience replay pool with a sampling ratio of τ and the experience in the high-quality experience replay pool with a ratio of 1-τ.

6. The method for grasping moving objects by a robotic arm based on multi-agent deep reinforcement learning according to claim 5, characterized in that, Preset multiple reward thresholds With a sampling ratio τ, during the training process of the deep reinforcement learning model MA-TD3H, after each preset number of training rounds, the deep reinforcement learning model MA-TD3H performs a preset number of grasping tests, and selects a corresponding set of numerical reward thresholds based on the current grasping success rate of the model. And the sampling ratio τ.

7. The method for grasping moving objects by a robotic arm based on multi-agent deep reinforcement learning according to claim 1, characterized in that, In step S3, after a preset number of training rounds, the deep reinforcement learning model MA-TD3H performs a grasping test on each agent in the model, and the agent network with the highest grasping success rate is recorded as the optimal agent network in the current test. Based on optimal actor network The parameters of other actor networks are updated using the following formula: in, For the updated actor network of other intelligent agents, For the actor network before the updates by other agents, μ is the multi-agent update factor, and ε is the agent network. i For the updated actor network of other agents L2 regularization.

8. The method for grasping moving objects by a robotic arm based on multi-agent deep reinforcement learning according to claim 1, characterized in that, In step S4, when the distance between the robotic arm gripper and the moving object to be grasped is less than a set distance threshold, the robotic arm gripper grasps the moving object to be grasped.

Images