Double-mechanical-arm cooperative carrying system based on deep reinforcement learning

Through the dual-robot collaborative handling system based on deep reinforcement learning, the problem of two-arm collaboration in complex closed-chain constraint tasks is solved, efficient grasping and handling in a dynamic environment is achieved, and the adaptability and intelligence level of the system are improved.

CN118990489AActive Publication Date: 2024-11-22DONGHUA UNIV

Patent Information

Application Number
CN202411240464.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-22
Estimated Expiration
2044-09-05

AI Technical Summary

Technical Problem

The existing dual robotic arm collaboration system is difficult to deal with the problem of cooperation and coupling between the two arms when facing complex closed-chain constraint tasks. The traditional method has heavy calculation burden, is difficult to respond quickly to environmental changes, and is difficult to achieve efficient in dynamic environments. Real-time control.

Method used

Using a dual robotic arm collaborative handling system based on deep reinforcement learning, the camera collects environmental information through state perception and environmental modeling, combined with the deep reinforcement learning algorithm SAC in the form of Actor-Critic and improved CSAC algorithm to control the dual robotic arm in complexity. Independent decision-making and collaboration in the environment to achieve capture and transport tasks.

Benefits of technology

It significantly improves the adaptability and intelligence of the system, can independently learn the optimal strategy in a dynamic environment, realize accurate object grabbing and handling, avoid collisions, improve operational efficiency and safety, and is suitable for intelligent manufacturing and logistics sorting and other fields.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118990489A_ABST
    Figure CN118990489A_ABST
Patent Text Reader

Abstract

The invention relates to a double-mechanical-arm cooperation system based on deep reinforcement learning, environment perception and target detection are carried out through a camera, a Faster R-CNN is used for recognizing a grabbing point, and a VGG-16 network is used for assisting in grabbing strategy decision making. According to the system, a single SAC algorithm based on an Actor-Critic form is creatively used for controlling the two arms to work cooperatively, and intelligence of grabbing and carrying tasks is achieved. In the carrying process, mechanical arms of a navigator and a collaborator dynamically adjust actions according to an improved CSAC algorithm, efficient path planning and posture synchronization are ensured, and a reward mechanism promotes target approaching while avoiding collision and maintaining cooperation stability under closed-loop constraints. The whole system is verified in a AppeliaSim simulation environment, and the whole-process automatic processing capacity from object recognition to stable carrying is shown.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robot motion control, and in particular to a dual-manipulator multi-step grasping and collaborative handling operation system based on deep reinforcement learning. Background Art

[0002] With the advent of the Industrial 4.0 era, the demand for automated and intelligent production is growing, placing higher demands on the flexibility, collaborative operation capabilities and intelligent decision-making levels of robot systems.

[0003] Existing dual-arm collaborative systems mainly rely on traditional control methods, which are usually based on solving inverse problems of robot kinematics and combining preset path planning algorithms to achieve coordinated movements of the robotic arms. For example, some studies use inverse kinematics solvers combined with predefined path planning algorithms to continuously adjust the paths of the robotic arms through optimization algorithms to achieve the desired collaborative effect. These systems have certain advantages in accuracy and stability, but their flexibility and intelligent decision-making capabilities are limited when faced with complex environments and task changes.

[0004] The existing dual-arm collaborative system is only applicable to a single customized application scenario. Its main disadvantage is that all motion tasks rely on inverse kinematics solutions. This method has a heavy computational burden when facing complex, multi-constrained collaborative tasks, and the accuracy of the solution depends on the accuracy of the model. In a dynamic environment, traditional methods are difficult to respond to changes quickly and cannot achieve efficient real-time control. In addition, inverse kinematics solutions are essentially deterministic calculations that are difficult to adapt to environmental uncertainties and the randomness of operational tasks. Therefore, in order to improve the adaptability and intelligence level of the system, research based on deep reinforcement learning (DRL) has gradually attracted attention in recent years. This type of method enables robots to make autonomous decisions and adapt to changes in the environment by allowing the agent to repeatedly trial and error and learn strategies in a simulated environment.

[0005] However, existing methods often combine deep reinforcement learning with inverse kinematics solutions, using DRL to determine the target position of the robot arm, and then calculating the specific joint angles through inverse kinematics. Although this type of method improves the intelligence of the system, it still retains the defects of inverse kinematics solutions, such as high solution complexity, accuracy still depends on the model, and difficulty in handling coordinated actions under multiple constraints. In addition, when faced with complex closed-chain constraint tasks, this type of method often cannot handle the collaboration and coupling problems between multiple arms well. Summary of the invention

[0006] Aiming at the difficulty of existing dual-arm collaborative systems to handle the collaboration and coupling problems between the two arms for complex closed-chain constraint tasks, a dual-arm collaborative handling system based on deep reinforcement learning was proposed.

[0007] The technical solution of the present invention is:

[0008] A dual-robot collaborative handling system based on deep reinforcement learning is built by the following steps:

[0009] Step 1: State perception and environment modeling: Use cameras to collect information about the surrounding environment and target objects, and construct a digital representation of the scene for deep reinforcement learning algorithms to make decisions.

[0010] Based on CoppeliaSim, a dual-arm collaborative handling system was built, which includes an operating table, a pilot arm, a collaborator arm, a robot gripper, a workpiece, and a material box;

[0011] The operating table consists of three parts: two ends and a middle part. A navigator robot arm is set on one end, a collaborator robot arm is set on the other end, and workpieces and material boxes are set at a certain interval in the middle part. There is a depth camera just above the operating table to collect visual information for the visual algorithm to identify the grasping point. The bases of the navigator robot arm and the collaborator robot arm are both located at both ends of the operating table. The initial positions of the end effectors of both are on both sides of the operating table. Robot claws are installed at the ends of both robots, waiting to grasp the workpiece. After the navigator robot arm and the collaborator robot arm grasp the workpiece, they move to the top of the material box in a coordinated and synchronous manner to place the workpiece in the material box.

[0012] Step 2: Based on the visual information collected by the camera, use the visual algorithm to identify the grasping point;

[0013] Step 3: Based on the grasping points identified in step 2, the Actor-Critic deep reinforcement learning algorithm SAC controls the dual robotic arms to grasp the object;

[0014] Step 4: Use the improved dual-robotic arm collaborative deep reinforcement learning algorithm to move the object to the target point.

[0015] Further, the specific construction steps are as follows:

[0016] Step 1: Use CoppeliaSim to build a simulation environment for deep reinforcement learning algorithm training of dual-arm collaborative handling. The proposed deep reinforcement learning algorithm is trained based on the dual-arm collaborative handling system built by CoppeliaSim.

[0017] The dual-robotic-arm collaborative handling system includes an operating table, a pilot robotic arm, a collaborator robotic arm, a robotic gripper, a workpiece, and a material box;

[0018] The operating table consists of three parts: two ends and a middle part. A navigator robot arm is set on one end, a collaborator robot arm is set on the other end, and workpieces and material boxes are set at a certain interval in the middle part. There is a depth camera just above the operating table to collect visual information for the visual algorithm to identify the grasping point; the bases of the navigator robot arm and the collaborator robot arm are both located at both ends of the operating table, and the initial postures of the end effectors of both are on both sides of the operating table. Robot claws are installed at the ends of both robots, waiting to grasp the workpiece; after the navigator robot arm and the collaborator robot arm grasp the workpiece, they move to the top of the material box in a collaborative and synchronous manner to place the workpiece in the material box; among them, the base of the robot arm is fixed on both sides of the operating table, and the position of the base will not change during the working process. The "end" refers to the end of the last connecting rod of the robot arm, and the end is usually equipped with a robot claw. During the working process, the posture of the end of the robot arm will change, so it is necessary to specify the initial posture of the end before the work starts;

[0019] Step 2: Based on the visual information collected by the camera, use the visual algorithm to identify the grasping point;

[0020] The classic Faster R-CNN architecture is used for target detection. The architecture consists of three main parts: feature extraction network, region proposal network, and classification and bounding box regression layer.

[0021] The image output by the camera is I, which is a three-dimensional tensor; the input image I of the feature extraction network passes through the convolution layer and the pooling layer, and the output feature map F is:

[0022] F=f conv (I) (1)

[0023] where f conv (·) represents the forward propagation process of the convolutional network;

[0024] Use a window of predefined size and aspect ratio to slide on the feature map F. If the window slides to a certain position in the feature map and contains the detected target, it is defined as r k And add it to the candidate box set R; slide on the feature map F to generate a series of candidate box sets R = {r1, r2, ..., r k ,},k is the index of the candidate box containing the detected target. In the coordinate system where the first pixel position in the upper left corner of the feature map is the origin, each candidate box r k From its upper left corner coordinate (x k ,y k ), width w k and height h k Definition: The region proposal network provides an object score s for each candidate box k, used to measure the probability of the candidate box containing the target; each candidate box containing the target is converted into a feature vector of fixed size, and then sent to the fully connected layer for classification and border fine-tuning; let C be the total number of categories of the target to be detected, then for the candidate box r k , the probability that it belongs to the cth class is p k,c , the corrected bounding box coordinates are Finally, through threshold screening, the target boxes with confidence higher than the set value are retained and mapped back to the original image to achieve accurate positioning of the target; the total number of categories of the target to be detected is C, and the candidate box r k There is a probability for each category, that is, there are C probabilities, and the lowercase c represents the category in the uppercase C;

[0025] Once the target object is accurately detected and located, the next step is to identify the appropriate grasping point; grasping point recognition relies on the analysis of the surface features of the object, including shape, texture, and color. First, the VGG-16 feature extraction network is applied to the target area to obtain the feature map F target ; Then, the features around the candidate grasping points are extracted through the region of interest alignment operation to form a feature vector set F g ={f g1 ,f g2 ,...,f gm ,...}, where m is the index of the candidate point, f gm is the coordinate of the mth candidate grasping point; s g is the grasp stability score, and the coordinates of the center of mass of the grasped object are (x z ,y z ,z z ), the candidate grasping point coordinates are (x, y, z), and the grasping stability score is s g The calculation formula is:

[0026]

[0027] In formula (2), η is a positive constant. Determines the rate at which the score is affected, l is one-third of the length of the grasped object; larger k values ​​indicate that as As the value of increases, the score decreases faster, reflecting the preference for grasping close to the center of mass; when the grasp point is close to (x z ,y z ±l,z z )(Right now ),s g Close to 1, the stability score is high; Increase, s g Rapid decrease, lower stability score;

[0028] Crawl stability scoresg It is used to evaluate the suitability of each candidate grasping point, and the input is the coordinates of the candidate grasping point f gm =(x m ,y m ,z m ), then according to formula (2):

[0029]

[0030] By traversing all candidate points, the point with the highest score is selected as the final grasping point g * :

[0031] g * = argmaxs g (f gm ) (4)

[0032] Step 3: According to the final grasping point g identified in step 2 * , using the deep reinforcement learning algorithm SAC in the Actor-Critic form to control the dual robotic arms to grasp the workpiece;

[0033] Different from the traditional solution of one Actor network controlling one robotic arm, one Actor network is used to control two robotic arms; for each time step t, the current environment state s of the robot agent t , select action a t = [a1, a2] and take out the joint angle increments a1 and a2 respectively and apply them to the leader robot arm and the collaborator robot arm, interact with the environment, and get the next environment state s t+1 Bonus with two robotic arms N 、r C , then the total reward r of the agent at time step t t for:

[0034]

[0035] r t =r N +r C (7)

[0036] In the above formula, k1 and k2 are regularization coefficients, p1 and p2 are the Cartesian coordinates of the two manipulators, is the final grasping point coordinate, distance(·) is the distance reward, which encourages approaching the target material box, and collision(·) is the collision reward, which prevents the robot arm from colliding;

[0037] Four-tuple (s t, a t ,s t+1 ,r t) is stored in the memory bank and used to update the SAC Actor network π after interacting with the environment at each time step t. φ and Critic Network

[0038] The Actor network aims to optimize the policy π φ , to maximize the expected long-term reward plus policy entropy to increase exploration; the update formula is as follows:

[0039]

[0040] In formula (8), J(φ) is the target formula for updating the Actor network, α is the entropy adjustment coefficient, φ is the parameter of the Actor network, π(a|s,φ) is the probability of selecting action a depending on state s and parameter φ, Q q (s,a|θ q ) represents the evaluation of the expected cumulative discounted return that can be obtained by the qth Critic network after taking action a in a given state s and making subsequent decisions according to the current strategy;

[0041] The update goal of the Critic network is:

[0042]

[0043] In formula (9), JQ is the target formula for updating the Critic network, r is the immediate reward, γ is the discount factor, δ is the indicator function of the terminal state, which is 0 when it is terminated and 1 otherwise, a′ is the sampled action of the next state s′, π(a′|s′) is the policy probability of the action, and α is the entropy adjustment coefficient;

[0044] After interacting with the environment n times, if the end effectors of the leader and collaborator reach their respective graspable points g * , then control the robot claws of each robot arm to grab the workpiece, and then cooperate to move the workpiece to the material box according to step 4; if within the specified time step, any end effector of the leader robot arm and the collaborator robot arm does not reach the graspable point g * , then repeat step 3 until the end effectors of the pilot robot arm and the collaborator robot arm reach their respective graspable points g * ;

[0045] Step 4: Use the dual-manipulator collaborative deep reinforcement learning algorithm to move the object to the target point;

[0046] In the dual-arm collaborative handling task, the navigator robot arm is involved Navigator With the collaborative robot arm Collaborator; The main responsibility of the navigator robot arm is to use the improved collaborative SAC algorithm CSAC strategy Learn an efficient path to reach a preset goal g, where a Nav represents the action of the pilot robot arm, s={p Nav ,p g ,E} is the state space, including the position of the leader p Nav 、Target position p g and the surrounding environment E; the reward function of the navigator robot arm r Nav Designed for:

[0047] r Nav (s,a Nav )=λ1·distance(p Nav ,p g )+λ2·Collision(s,a Nav ) (10)

[0048] In formula (10), λ1, λ2 are regularization coefficients, distance() is the distance reward, which encourages approaching the target material box, and Collision(·) is the collision reward, which prevents the robot arm from colliding.

[0049] Collaboration reward collaboration() is used to maintain the relative posture relationship between the leader robot arm and the collaborator robot arm. The dual-robot collaborative handling system satisfies the closed-chain constraint in the mechanical structure. The grasping point g * Homogeneous transformation matrix relative to the world coordinate system It can be obtained by the related homogeneous transformation matrix of the two robotic arms respectively, and the formula is as follows:

[0050]

[0051] In formula (11), i = 1, 2 represents the index of the two manipulators, W, base, and end represent the coordinate systems with the world, the manipulator base, and the manipulator end effector as the coordinate origin, respectively;

[0052] Through the homogeneous transformation matrix of the two manipulators The homogeneous transformation matrix of the end effector of the robot arm with index 2 relative to the end effector of the robot arm with index 1 can be derived: The following formula (12):

[0053]

[0054] When step 3 is completed, the two robotic arms reach the grasping point g respectively. *, at this time, the closed-loop constraint is satisfied. Through the robot kinematics and robot parameters, the homogeneous transformation matrix of the base of the navigator robot arm relative to the world coordinate system can be calculated respectively. Homogeneous transformation matrix of the end effector of the navigator robot arm relative to the base of the navigator robot arm Homogeneous transformation matrix of the base of the collaborator robot relative to the world coordinate system and the homogeneous transformation matrix of the end effector of the collaborator robot arm relative to the base of the collaborator robot arm According to formula (12), we can further obtain the homogeneous transformation matrix T that satisfies the closed-loop constraint: gt ; In the process of dual-manipulator collaborative handling, the homogeneous transformation matrix in the calculation process is calculated using formula (12): and the homogeneous transformation matrix T that satisfies the closed-loop constraint gt The difference between the two can get the cooperation reward; the second transformation matrix of the robot arm base relative to the world coordinate system and It is fixed and does not change with the collaborative handling process of the two robotic arms. Calculate the homogeneous transformation matrix of the end effector of the collaborator robotic arm relative to the end effector of the navigator robotic arm It is only necessary to calculate the homogeneous transformation matrix of the end effector of the robot arm relative to the base of the robot arm through the robot kinematics and robot parameters. and That's it;

[0055] The collaboration(·) reward is derived from the Euler rotation theorem, and its formula is:

[0056]

[0057] In formula (13), T jj represents the j-th row and j-th column element of matrix T;

[0058] The collaborator robot uses the CSAC strategy To achieve synchronization with the navigator robot arm, the state s colla ={p Nav ,a Nav ,p Colla} contains not only the position and motion of the leader robot arm, but also the position of the end effector of the collaborator robot arm Colla ; The learning goal of the collaborator arm is to maintain the relative posture and position relationship and ensure the following accuracy by predicting the behavior of the leader arm and adjusting its own actions; The strategy of the collaborator arm aims to maximize the long-term reward. Considering the following efficiency, synchronization and adaptability, the reward function of the collaborator arm is r Colla (s Colla ,a Colla)as follows:

[0059]

[0060] In formula (14), μ1, μ2 and μ3 are regularization coefficients, collaboration(·) is the collaboration reward, and T gt They are the homogeneous transformation matrix between the end-effector of the leader robot arm and the end-effector of the collaborator robot arm during the training process and the homogeneous transformation matrix that satisfies the closed-loop constraint;

[0061] The improved CSAC algorithm combines the standard SAC algorithm with the homogeneous transformation matrix of the end effector of the collaborator robot arm relative to the end effector of the leader robot arm in the process. and the homogeneous transformation matrix T that satisfies the closed-loop constraint gt The additional penalty term between is intended to improve the coordinated control performance of the dual-manipulator system. First, the homogeneous transformation matrix of the collaborator's end effector relative to the leader's end effector is calculated during the collaborative handling process. Then calculate and the homogeneous transformation matrix T that satisfies the closed-loop constraint gt The mean square error (MSE) loss between:

[0062]

[0063] In formula (15), B represents the number of elements in the matrix, and b represents the index of the element in the matrix;

[0064] The improved CASC algorithm takes into account the homogeneous transformation matrix of the end effector of the collaborator robot arm relative to the end effector of the leader robot arm in the Q value update formula. and the homogeneous transformation matrix T that satisfies the closed-loop constraint gt The mean square error (MSE) loss between the two, the updated target Q value calculation formula of the improved CSAC algorithm is as follows:

[0065]

[0066] In formula (16), r is the immediate reward, γ is the discount factor, α is the weight of controlling entropy, β is the importance of adjusting the position error of the end effector, and Q min (s′,a′)=min(Q1(s′,a′),Q2(s′,a′)), Q1 and Q2 are two action value networks in the Critic network respectively;

[0067] The update formula of the Critic network of the improved CASC algorithm is:

[0068]

[0069] The update formula of the Actor network of the improved CASC algorithm is:

[0070] L actor =-Ε s~D [Q min (s,a)-αlogπ(a|s)] (18)

[0071] In formula (18), D represents the set of environmental states, E s~D It means to find the expectation of the target formula when taking state s from the environment state set D;

[0072] After interacting with the environment n times, if the end effectors of the navigator robot arm and the collaborator robot arm both reach the material box, the robot claws of each robot arm are controlled to put down the workpiece; if within the specified time step, the end effector of either the navigator robot arm or the collaborator robot arm does not reach the material box, or the two robots are out of sync at any time step, repeat step three until the end effectors of both the navigator robot arm and the collaborator robot arm reach the material box.

[0073] Preferably, in step 1, the navigator robot arm is KUKA LBR iiwa 7R800, the collaborator robot arm is Universal Robots UR5, and the robot gripper is Robotiq 140.

[0074] Preferably, the feature extraction network in step 2 may be ResNet.

[0075] Preferably, in step 2, the grasping point recognition relies on the analysis of the surface features of the object, including shape, texture, and color.

[0076] The beneficial effects of the present invention are:

[0077] Deep reinforcement learning (DRL), as a bridge connecting artificial intelligence and robot control, enables robots to self-learn the optimal strategy through continuous trial and error, effectively solving the problem that traditional programming is difficult to deal with in complex and dynamic environments. Through the deep reinforcement learning framework, the dual-arm collaborative system can not only achieve accurate object grasping, handling and placement, but also autonomously learn how to optimally allocate tasks, avoid collisions, and adaptively adjust strategies in dynamic environments during task execution, thereby significantly improving work efficiency and safety. In addition, combined with advanced visual recognition and environmental perception technology, the system can understand and adapt to the working environment in real time, and achieve effective recognition and processing of a variety of objects. It is widely used in smart manufacturing, logistics sorting, disaster relief and other fields, and is driving robotics technology towards a smarter and more flexible direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is a diagram of a dual-manipulator collaborative handling system constructed using CoppeliaSim simulation software in the present invention;

[0079] Figure 2 This is a workflow diagram of the dual-manipulator collaborative handling system based on deep reinforcement learning of the present invention;

[0080] Figure 3 It is a flowchart of the algorithm update for the dual arms to reach the grasping point of the present invention;

[0081] Figure 4 This is a flowchart of the algorithm update for the dual-arm collaborative transport of the present invention.

[0082] Figure ID:

[0083] 1. Operating table; 2. Navigator robotic arm; 3. Collaborator robotic arm; 4. Robotic gripper; 5. Workpiece; 6. Material box. DETAILED DESCRIPTION

[0084] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0085] A dual-robot collaborative handling system based on deep reinforcement learning is built by the following steps:

[0086] Step 1: Use CoppeliaSim (a robot simulation platform) to build a simulation environment for deep reinforcement learning algorithm training. The dual robot arms collaborative handling is in view of the high cost, potential risk of mechanical damage and limited data collection faced by direct training in the real world. The simulation environment can not only safely and cost-effectively generate a variety of training scenarios and massive data, and support the dual robot arms to continuously learn and optimize strategies in complex handling tasks, but also enhance the generalization ability of the algorithm by finely controlling the experimental conditions, ensuring the best learning effect without damaging the physical equipment, and laying a solid foundation for actual deployment. In this embodiment, the use of Figure 1 The dual-robotic-arm collaborative handling system built based on CoppeliaSim shown trains the deep reinforcement learning algorithm proposed in the present invention.

[0087] like Figure 1 The dual-robotic-arm collaborative handling system shown is constructed based on CoppeliaSim and includes an operating platform 1, a pilot robotic arm 2, a collaborator robotic arm 3, a robotic gripper 4, a workpiece 5 and a material box 6.

[0088] The operating table 1 includes two ends and a middle part. A pilot robot arm 2 is arranged on one end, a collaborator robot arm 3 is arranged on the other end, and a workpiece 5 and a material box 6 are arranged at a certain interval in the middle part. A depth camera is arranged just above the operating table 1 to collect visual information for the visual algorithm to identify the grasping point; the bases of the pilot robot arm 2 and the collaborator robot arm 3 are both located at both ends of the operating table 1, and the initial positions of the end effectors of both are on both sides of the operating table. Robot claws 4 are installed at the ends of both robots, waiting to grasp the workpiece 5; after the pilot robot arm 2 and the collaborator robot arm 3 grasp the workpiece 5, they move to the top of the material box 6 in a coordinated and synchronous manner and place the workpiece 5 in the material box 6; the workflow diagram of the dual-robot collaborative handling system based on deep reinforcement learning is shown in FIG. Figure 2 As shown. In this embodiment, the navigator robot arm 2 uses KUKA LBR iiwa7R800, the collaborator robot arm 3 uses Universal Robots UR5, and the robot gripper 4 uses Robotiq 140. Among them, the base of the robot arm is fixed on both sides of the operating table, and the position of the base will not change during the working process. The "end" refers to the end of the last connecting rod of the robot arm, and the end is usually equipped with a robot gripper. During the working process, the posture of the end of the robot arm will change, so it is necessary to specify the initial posture of the end before the work starts.

[0089] Step 2: Based on the visual information collected by the camera, use the visual algorithm to identify the grasping point;

[0090] The classic Faster R-CNN architecture is used for target detection, which consists of three main parts: feature extraction network (such as ResNet), region proposal network (RPN) and classification and bounding box regression layer.

[0091] The image output by the camera is I, which is a three-dimensional tensor. The input image I of the feature extraction network passes through the convolution layer and the pooling layer, and the output feature map F is:

[0092] F=f conv (I) (1)

[0093] where f conv (·) represents the forward propagation process of the convolutional network.

[0094] Use a window of predefined size and aspect ratio to slide on the feature map F. If the window slides to a certain position in the feature map and contains the detected target, it is defined as r k And add it to the candidate box set R. Slide on the feature map F to generate a series of candidate box sets R = {r1, r2, ..., r k,...}, k is the index of the candidate box containing the detected target. In the coordinate system where the first pixel position in the upper left corner of the feature map is the origin, each candidate box r k From its upper left corner coordinate (x k ,y k ), width w k and height h k Definition. The region proposal network provides an object score s for each candidate box k , which is used to measure the probability that the candidate box contains the target. Each candidate box containing the target is converted into a feature vector of fixed size, and then sent to the fully connected layer for classification and border fine-tuning. Let C be the total number of categories of the target to be detected, then for the candidate box r k , which belongs to the cth category (the total number of categories of the target to be detected is C, the candidate box r k There is a probability for each category, that is, there are C probabilities, with lowercase c representing the category in uppercase C) pk,c , the corrected bounding box coordinates are Finally, through threshold screening, the target boxes with confidence levels higher than the set value are retained and mapped back to the original image to achieve accurate positioning of the target.

[0095] Once the target object is accurately detected and located, the next step is to identify the appropriate grasping points. Grasping point recognition relies on the analysis of the surface features of the object, including shape, texture, color, etc. First, the VGG-16 feature extraction network is applied to the target area to obtain the feature map F target Then, the features around the candidate grasping points are extracted through the region of interest alignment operation to form a feature vector set F g ={f g1 ,f g2 ,...,f gm ,...}, where m is the index of the candidate point, f gm is the coordinate of the mth candidate grasping point. g is the grasp stability score, and the coordinates of the center of mass of the grasped object are (x z ,y z ,z z ), the candidate grasping point coordinates are (x, y, z), and the grasping stability score is s g The calculation formula is:

[0096]

[0097] In formula (2), η is a positive constant. determines the rate at which the score is affected, where l is one third of the length of the grasped object. Larger values ​​of k indicate that as As the value of increases, the score decreases faster, reflecting the preference for grasping close to the center of mass.z ,y z ±l,z z )(Right now ),s g Close to 1, the stability score is high; Increase, s g The faster it decreases, the lower the stability score.

[0098] Grab stability score g It is used to evaluate the suitability of each candidate grasping point, and the input is the coordinates of the candidate grasping point f gm =(x m ,y m ,z m ), then according to formula (2):

[0099]

[0100] By traversing all candidate points, the point with the highest score is selected as the final grasping point g * :

[0101] g * = argmaxs g (f gm ) (4)

[0102] Step 3: According to the final grasping point g identified in step 2 * , using the deep reinforcement learning algorithm SAC in the Actor-Critic form to control the dual robotic arms to grasp the workpiece 5;

[0103] Different from the traditional solution of one Actor network controlling one robotic arm, one Actor network is used to control two robotic arms. For each time step t, the current environment state s of the robot agent t , select action a t = [a1, a2] and take out the joint angle increments a1 and a2 respectively and apply them to the navigator robot arm 2 and the collaborator robot arm 3, interact with the environment, and get the next environment state s t+1 Bonus with two robotic arms N 、r C (The English name of the navigator is Navigator, and the English name of the collaborator is Collarator), then the total reward rt of the agent at time step t is:

[0104]

[0105] r t =r N +r C (7)

[0106] In the above formula, k1 and k2 are regularization coefficients, p1 and p2 are the Cartesian coordinates of the two manipulators, is the final grasping point coordinate, distance(·) is the distance reward, which encourages approaching the target material box 6, and Collision(·) is the collision reward, which prevents the robot arm from colliding.

[0107] Four-tuple (s t, a t ,s t+1 ,r t ) is stored in the memory bank and used to update the SAC Actor network π after interacting with the environment at each time step t. φ and Critic Network q=1,2.

[0108] The Actor network aims to optimize the policy π φ , to maximize the expected long-term reward plus policy entropy to increase exploration. The update formula is as follows:

[0109]

[0110] In formula (8), J(φ) is the target formula for updating the Actor network, α is the entropy adjustment coefficient, φ is the parameter of the Actor network, π(a|s,φ) is the probability of selecting action a depending on state s and parameter φ, Q q (s,a|θ q ) represents the evaluation of the expected cumulative discounted return that can be obtained by the qth Critic network after taking action a in a given state s and making subsequent decisions according to the current strategy.

[0111] The update goal of the Critic network is:

[0112]

[0113] In formula (9), J Q is the target formula for updating the Critic network, r is the immediate reward, γ is the discount factor, δ is the indicator function of the terminal state (0 when terminated, otherwise 1), a′ is the sampled action of the next state s′, π(a′|s′) is the policy probability of the action, and α is the entropy adjustment coefficient.

[0114] After interacting with the environment n times, if the end effectors of the leader robot arm 2 and the collaborator robot arm 3 both reach their respective graspable points g * , then control the robot claw 4 of each robot arm to grab the workpiece 5, and then cooperate to move the workpiece 5 to the material box 6 according to step 4; if within the specified time step, any end effector of the navigator robot arm 2 and the collaborator robot arm 3 does not reach the graspable point g *, then repeat step 3 until the end effectors of the pilot robot arm 2 and the collaborator robot arm 3 reach their respective graspable points g * ,The algorithm update flow chart of step 3 where the two arms reach the grasping point is as follows Figure 3 shown.

[0115] Step 4: Use the dual-manipulator collaborative deep reinforcement learning algorithm to move the object to the target point;

[0116] In the dual-arm collaborative handling task, the Navigator Robot Arm 2 (Arm Navigator ) and the collaborator robot arm 3 (Arm Collaborator ). The main responsibility of the navigator 2 is to use the improved collaborative SAC algorithm (CSAC) strategy π θNav (a Nav |s) learn an efficient path to reach the preset goal g, where a Nav represents the action of the pilot robot arm 2, s = {p Nav ,p g ,E} is the state space, including the position of the leader p Nav 、Target position p g and the surrounding environment E. The reward function of the navigator robot arm r Nav Designed for:

[0117] r Nav (s,a Nav )=λ1·distance(p Nav ,p g )+λ2·Collision(s,a Nav ) (10)

[0118] In formula (10), λ1 and λ2 are regularization coefficients, distance(·) is the distance reward, which encourages approaching the target material box 6, and Collision(·) is the collision reward, which prevents the robot arm from colliding.

[0119] Collaboration reward collaboration() is used to maintain the relative posture relationship between the leader robot arm 2 and the collaborator robot arm 3. The dual-robot collaborative handling system satisfies the closed-chain constraint in the mechanical structure. The grasping point g * Homogeneous transformation matrix relative to the world coordinate system It can be obtained by the related homogeneous transformation matrix of the two robotic arms respectively, and the formula is as follows:

[0120]

[0121] In formula (11), i = 1, 2 represents the index of the two manipulators, and W, base, and end represent the coordinate systems with the world, the manipulator base, and the manipulator end effector as the coordinate origins, respectively.

[0122] Through the homogeneous transformation matrix of the two manipulators The homogeneous transformation matrix of the end effector of the robot arm with index 2 relative to the end effector of the robot arm with index 1 can be derived: The following formula (12):

[0123]

[0124] When step 3 is completed, the two robotic arms reach the grasping point g respectively. * , at this time, the closed-loop constraint is satisfied. Through robot kinematics and robot parameters (basic knowledge of robot kinematics), the homogeneous transformation matrix of the base of the navigator robot arm 2 relative to the world coordinate system can be calculated respectively. The homogeneous transformation matrix of the end effector of the pilot robot arm 2 relative to the base of the pilot robot arm 2 Homogeneous transformation matrix of the base of the collaborator robot 3 relative to the world coordinate system and the homogeneous transformation matrix of the end effector of the collaborator robot arm 3 relative to the base of the collaborator robot arm 3 According to formula (12), we can further obtain the homogeneous transformation matrix T that satisfies the closed-loop constraint: gt In the process of dual-manipulator collaborative handling, the homogeneous transformation matrix in the calculation process is calculated using formula (12): and the homogeneous transformation matrix T that satisfies the closed-loop constraint gt The difference between the two can get the cooperation reward. The second transformation matrix of the robot arm base relative to the world coordinate system is and It is fixed and does not change with the collaborative handling process of the two robotic arms. Calculate the homogeneous transformation matrix of the end effector of the collaborator robotic arm 3 relative to the end effector of the navigator robotic arm 2 It is only necessary to calculate the homogeneous transformation matrix of the end effector of the robot arm relative to the base of the robot arm through the robot kinematics and robot parameters. and That's it.

[0125] The collaboration(·) reward is derived from the Euler rotation theorem, and its formula is:

[0126]

[0127] In formula (13), T jj Represents the element in the j-th row and j-th column of matrix T.

[0128] Collaborator Arm 3 uses the CSAC strategy To achieve synchronization with the navigator robot arm, the state s colla ={p Nav ,a Nav ,p Colla} contains not only the position and motion of the leader robot arm 2, but also the position p of the end effector of the collaborator robot arm 3 Colla The learning goal of the collaborator arm 3 is to maintain the relative posture and position relationship and ensure the following accuracy by predicting the behavior of the leader arm and adjusting its own actions. The strategy of the collaborator arm 3 aims to maximize the long-term reward. Considering the following efficiency, synchronization and adaptability, the reward function of the collaborator arm 3 is r Colla (s Colla ,a Colla )as follows:

[0129]

[0130] In formula (14), μ1, μ2 and μ3 are regularization coefficients, collaboration(·) is the collaboration reward, and T gt They are the homogeneous transformation matrix between the end effector of the leader robot arm and the end effector of the collaborator robot arm during the training process and the homogeneous transformation matrix that satisfies the closed-loop constraints.

[0131] The improved CSAC algorithm combines the standard SAC algorithm with the homogeneous transformation matrix of the end effector of the collaborator robot arm 3 relative to the end effector of the leader robot arm 2 in the process and the homogeneous transformation matrix T that satisfies the closed-loop constraint gt The additional penalty term between is intended to improve the coordinated control performance of the dual-manipulator system. First, the homogeneous transformation matrix of the collaborator robot end effector relative to the leader robot end effector is calculated during the collaborative handling process. Then calculate and the homogeneous transformation matrix T that satisfies the closed-loop constraint gt The mean square error (MSE) loss between:

[0132]

[0133] In formula (15), B represents the number of elements in the matrix, and b represents the index of the element in the matrix.

[0134] The improved CASC algorithm considers the homogeneous transformation matrix of the end effector of the collaborator robot arm 3 relative to the end effector of the leader robot arm 2 in the Q value update formula. and the homogeneous transformation matrix T that satisfies the closed-loop constraint gtThe mean square error (MSE) loss between the two, the updated target Q value calculation formula of the improved CSAC algorithm is as follows:

[0135]

[0136] In formula (16), r is the immediate reward, γ is the discount factor, α is the weight of controlling entropy, β is the importance of adjusting the position error of the end effector, and Q min (s′,a′)=min(Q1(s′,a′),Q2(s′,a′)), where Q1 and Q2 are two action-value networks in the Critic network.

[0137] The update formula of the Critic network of the improved CASC algorithm is:

[0138]

[0139] The update formula of the Actor network of the improved CASC algorithm is:

[0140] L actor =-Ε s~D [Q min (s,a)-αlogπ(a|s)] (18)

[0141] In formula (18), D represents the set of environmental states, E s~D It means to find the expectation of the target formula when taking state s from the environment state set D.

[0142] After interacting with the environment n times, if the end effectors of the pilot robot arm 2 and the collaborator robot arm 3 both reach the material box 6, the robot grippers 4 of their respective robots are controlled to put down the workpiece 5; if within the specified time step, the end effectors of either the pilot robot arm 2 or the collaborator robot arm 3 do not reach the material box 6, or the two robots are out of sync at any time step, then repeat step three until the end effectors of the pilot robot arm 2 and the collaborator robot arm 3 both reach the material box 6. The algorithm update flow chart of step four for dual-arm collaborative handling is as follows: Figure 4 shown.

[0143] The above-mentioned embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A dual-manipulator collaborative handling system based on deep reinforcement learning, characterized in that: The following steps are used to build it: Step 1: State perception and environment modeling: Use cameras to collect information about the surrounding environment and target objects, and construct a digital representation of the scene for deep reinforcement learning algorithms to make decisions. Based on CoppeliaSim, a dual-arm collaborative handling system was built, which includes an operating table, a pilot arm, a collaborator arm, a robot gripper, a workpiece, and a material box; The operating table consists of three parts: two ends and a middle part. A navigator robot arm is set on one end, a collaborator robot arm is set on the other end, and workpieces and material boxes are set at a certain interval in the middle part. There is a depth camera just above the operating table to collect visual information for the visual algorithm to identify the grasping point. The bases of the navigator robot arm and the collaborator robot arm are both located at both ends of the operating table. The initial positions of the end effectors of both are on both sides of the operating table. Robot claws are installed at the ends of both robots, waiting to grasp the workpiece. After the navigator robot arm and the collaborator robot arm grasp the workpiece, they move to the top of the material box in a coordinated and synchronous manner to place the workpiece in the material box. Step 2: Based on the visual information collected by the camera, use the visual algorithm to identify the grasping point; Step 3: Based on the grasping points identified in step 2, the Actor-Critic deep reinforcement learning algorithm SAC controls the dual robotic arms to grasp the object; Step 4: Use the improved dual-robotic arm collaborative deep reinforcement learning algorithm to move the object to the target point.

2. According to the dual-manipulator collaborative handling system based on deep reinforcement learning in claim 1, it is characterized in that: The specific construction steps are as follows: Step 1: Use CoppeliaSim to build a simulation environment for deep reinforcement learning algorithm training of dual-arm collaborative handling. The proposed deep reinforcement learning algorithm is trained based on the dual-arm collaborative handling system built by CoppeliaSim. The dual-robotic-arm collaborative handling system includes an operating table, a pilot robotic arm, a collaborator robotic arm, a robotic gripper, a workpiece, and a material box; The operating table consists of three parts: two ends and a middle part. A navigator robot arm is set on one end, a collaborator robot arm is set on the other end, and workpieces and material boxes are set at a certain interval in the middle part. There is a depth camera just above the operating table to collect visual information for the visual algorithm to identify the grasping point; the bases of the navigator robot arm and the collaborator robot arm are both located at both ends of the operating table, and the initial postures of the end effectors of both are on both sides of the operating table. Robot claws are installed at the ends of both robots, waiting to grasp the workpiece; after the navigator robot arm and the collaborator robot arm grasp the workpiece, they move to the top of the material box in a collaborative and synchronous manner to place the workpiece in the material box; among them, the base of the robot arm is fixed on both sides of the operating table, and the position of the base will not change during the working process. The "end" refers to the end of the last connecting rod of the robot arm, and the end is usually equipped with a robot claw. During the working process, the posture of the end of the robot arm will change, so it is necessary to specify the initial posture of the end before the work starts; Step 2: Based on the visual information collected by the camera, use the visual algorithm to identify the grasping point; The classic Faster R-CNN architecture is used for target detection. The architecture consists of three main parts: feature extraction network, region proposal network, and classification and bounding box regression layer. The image output by the camera is I, which is a three-dimensional tensor; the input image I of the feature extraction network passes through the convolution layer and the pooling layer, and the output feature map F is: F=f conv (I) (1) where f conv (·) represents the forward propagation process of the convolutional network; Use a window of predefined size and aspect ratio to slide on the feature map F. If the window slides to a certain position in the feature map and contains the detected target, it is defined as r k And add it to the candidate box set R; slide on the feature map F to generate a series of candidate box sets R = {r1, r2, ..., r k ,},k is the index of the candidate box containing the detected target. In the coordinate system where the first pixel position in the upper left corner of the feature map is the origin, each candidate box r k From its upper left corner coordinate (x k ,y k ), width w k and height h k Definition: The region proposal network provides an object score s for each candidate box k , used to measure the probability of the candidate box containing the target; each candidate box containing the target is converted into a feature vector of fixed size, and then sent to the fully connected layer for classification and border fine-tuning; let C be the total number of categories of the target to be detected, then for the candidate box r k , the probability that it belongs to the cth class is p k,c , the corrected bounding box coordinates are Finally, through threshold screening, the target boxes with confidence higher than the set value are retained and mapped back to the original image to achieve accurate positioning of the target; the total number of categories of the target to be detected is C, and the candidate box r k There is a probability for each category, that is, there are C probabilities, and the lowercase c represents the category in the uppercase C; Once the target object is accurately detected and located, the next step is to identify the appropriate grasping point; grasping point identification relies on the analysis of the surface features of the object. First, the VGG-16 feature extraction network is applied to the target area to obtain the feature map F target ; Then, the features around the candidate grasping points are extracted through the region of interest alignment operation to form a feature vector set F g ={f g1 ,f g2 ,...,f gm ,...}, where m is the index of the candidate point, f gm is the coordinate of the mth candidate grasping point; s g is the grasp stability score, and the coordinates of the center of mass of the grasped object are (x z ,y z ,z z ), the candidate grasping point coordinates are (x, y, z), and the grasping stability score is s g The calculation formula is: In formula (2), η is a positive constant. Determines the rate at which the score is affected, l is one-third of the length of the grasped object; larger k values ​​indicate that as As the value of increases, the score decreases faster, reflecting the preference for grasping close to the center of mass; when the grasp point is close to (x z ,y z ±l,z z )Right now When g Close to 1, the stability score is high; Increase, s g Rapid decrease, lower stability score; Crawl stability score sg It is used to evaluate the suitability of each candidate grasping point, and the input is the coordinates of the candidate grasping point f gm =(x m ,y m ,z m ), then according to formula (2): By traversing all candidate points, the point with the highest score is selected as the final grasping point g * : g * =argmaxs g (f gm ) (4) Step 3: According to the final grasping point g identified in step 2 * , using the deep reinforcement learning algorithm SAC in the Actor-Critic form to control the dual robotic arms to grasp the workpiece; Different from the traditional solution of one Actor network controlling one robotic arm, one Actor network is used to control two robotic arms; for each time step t, the current environment state s of the robot agent t , select action a t = [a1, a2] and take out the joint angle increments a1 and a2 respectively and apply them to the leader robot arm and the collaborator robot arm, interact with the environment, and get the next environment state s t+1 Bonus with two robotic arms N 、r C , then the total reward r of the agent at time step t t for: r t =r N +r C (7) In the above formula, k1 and k2 are regularization coefficients, p1 and p2 are the Cartesian coordinates of the two manipulators, is the final grasping point coordinate, distance(·) is the distance reward, which encourages approaching the target material box, and collision(·) is the collision reward, which prevents the robot arm from colliding; Four-tuple (s t ,a t ,s t+1 ,r t ) is stored in the memory bank and used to update the SAC Actor network π after interacting with the environment at each time step t. φ and Critic Network q=1,2; The Actor network aims to optimize the policy π φ , to maximize the expected long-term reward plus policy entropy to increase exploration; the update formula is as follows: In formula (8), J(φ) is the target formula for updating the Actor network, α is the entropy adjustment coefficient, φ is the parameter of the Actor network, π(a|s,φ) is the probability of selecting action a depending on state s and parameter φ, Q q (s,a|θ q ) represents the evaluation of the expected cumulative discounted return that can be obtained by the qth Critic network after taking action a in a given state s and making subsequent decisions according to the current strategy; The update goal of the Critic network is: In formula (9), JQ is the target formula for updating the Critic network, r is the immediate reward, γ is the discount factor, δ is the indicator function of the terminal state, which is 0 when it is terminated and 1 otherwise, a′ is the sampled action of the next state s′, π(a′|s′) is the policy probability of the action, and α is the entropy adjustment coefficient; Interact with the environment n After that, if the end effectors of the pilot robot and the collaborator robot reach their respective graspable points g * , then control the robot claws of each robot arm to grab the workpiece, and then cooperate to move the workpiece to the material box according to step 4; if within the specified time step, any end effector of the leader robot arm and the collaborator robot arm does not reach the graspable point g * , then repeat step 3 until the end effectors of the pilot robot arm and the collaborator robot arm reach their respective graspable points g * ; Step 4: Use the dual-manipulator collaborative deep reinforcement learning algorithm to move the object to the target point; In the dual-arm collaborative handling task, the navigator robot arm is involved Navigator With the collaborative robot arm Collaborator ; The main responsibility of the navigator robot arm is to use the improved collaborative SAC algorithm CSAC strategy Learn an efficient path to reach a preset goal g, where a Nav represents the action of the pilot robot arm, s={p Nav ,p g ,E} is the state space, including the position of the leader p Nav 、Target position p g and the surrounding environment E; the reward function of the navigator robot arm r Nav Designed for: r Nav (s,a Nav )=λ1·distance(p Nav ,p g )+λ2·Collision(s,a Nav ) (10) In formula (10), λ1, λ2 are regularization coefficients, distance() is the distance reward, which encourages approaching the target material box, and Collision(·) is the collision reward, which prevents the robot arm from colliding. Collaboration reward collaboration() is used to maintain the relative posture relationship between the leader robot arm and the collaborator robot arm. The dual-robot collaborative handling system satisfies the closed-chain constraint in the mechanical structure. The grasping point g * Homogeneous transformation matrix relative to the world coordinate system It can be obtained by the related homogeneous transformation matrix of the two robotic arms respectively, and the formula is as follows: In formula (11), i = 1, 2 represents the index of the two manipulators, W, base, and end represent the coordinate systems with the world, the manipulator base, and the manipulator end effector as the coordinate origin, respectively; Through the homogeneous transformation matrix of the two manipulators The homogeneous transformation matrix of the end effector of the robot arm with index 2 relative to the end effector of the robot arm with index 1 can be derived: The following formula (12): When step 3 is completed, the two robotic arms reach the grasping point g respectively. * , at this time, the closed-loop constraint is satisfied. Through the robot kinematics and robot parameters, the homogeneous transformation matrix of the base of the navigator robot arm relative to the world coordinate system can be calculated respectively. Homogeneous transformation matrix of the end effector of the navigator robot arm relative to the base of the navigator robot arm Homogeneous transformation matrix of the base of the collaborator robot relative to the world coordinate system and the homogeneous transformation matrix of the end effector of the collaborator robot arm relative to the base of the collaborator robot arm According to formula (12), we can further obtain the homogeneous transformation matrix T that satisfies the closed-loop constraint: gt ; In the process of dual-manipulator collaborative handling, the homogeneous transformation matrix in the calculation process is calculated using formula (12): and the homogeneous transformation matrix T that satisfies the closed-loop constraint gt The difference between the two can get the cooperation reward; the second transformation matrix of the robot arm base relative to the world coordinate system and It is fixed and does not change with the collaborative handling process of the two robotic arms. Calculate the homogeneous transformation matrix of the end effector of the collaborator robotic arm relative to the end effector of the navigator robotic arm It is only necessary to calculate the homogeneous transformation matrix of the end effector of the robot arm relative to the base of the robot arm through the robot kinematics and robot parameters. and That's it; The collaboration(·) reward is derived from the Euler rotation theorem, and its formula is: In formula (13), T jj represents the j-th row and j-th column element of matrix T; The collaborator robot uses the CSAC strategy To achieve synchronization with the navigator robot arm, the state s colla ={p Nav ,a Nav ,p Colla } contains not only the position and motion of the leader robot arm, but also the position of the end effector of the collaborator robot arm Colla ; The learning goal of the collaborator arm is to predict the behavior of the leader arm and adjust its own actions to maintain the relative posture and position relationship and ensure the following accuracy; The collaborator arm strategy aims to maximize the long-term reward. Considering the following efficiency, synchronization and adaptability, the collaborator arm reward function r Colla (s Colla ,a Colla )as follows: In formula (14), μ1, μ2 and μ3 are regularization coefficients, collaboration(·) is the collaboration reward, and T gt They are the homogeneous transformation matrix between the end-effector of the leader robot arm and the end-effector of the collaborator robot arm during the training process and the homogeneous transformation matrix that satisfies the closed-loop constraint; The improved CSAC algorithm combines the standard SAC algorithm with the homogeneous transformation matrix of the end effector of the collaborator robot arm relative to the end effector of the leader robot arm in the process. and the homogeneous transformation matrix T that satisfies the closed-loop constraint gt The additional penalty term between is intended to improve the coordinated control performance of the dual-manipulator system. First, the homogeneous transformation matrix of the collaborator's end effector relative to the leader's end effector is calculated during the collaborative handling process. Then calculate and the homogeneous transformation matrix T that satisfies the closed-loop constraint gt The mean square error MSE loss between: In formula (15), B represents the number of elements in the matrix, and b represents the index of the element in the matrix; The improved CASC algorithm takes into account the homogeneous transformation matrix of the end effector of the collaborator robot arm relative to the end effector of the leader robot arm in the Q value update formula. and the homogeneous transformation matrix T that satisfies the closed-loop constraint gt The mean square error MSE loss between them, the updated target Q value calculation formula of the improved CSAC algorithm is as follows: In formula (16), r is the immediate reward, γ is the discount factor, α is the weight of controlling entropy, β is the importance of adjusting the position error of the end effector, and Q min (s′,a′)=min(Q1(s′,a′),Q2(s′,a′)), Q1 and Q2 are two action value networks in the Critic network respectively; The update formula of the Critic network of the improved CASC algorithm is: The update formula of the Actor network of the improved CASC algorithm is: L actor =-E s~D [Q min (s,a)-αlogπ(a|s)] (18) In formula (18), D represents the set of environmental states, E s~D It means to find the expectation of the target formula when taking state s from the environment state set D; After interacting with the environment n times, if the end effectors of the navigator robot arm and the collaborator robot arm both reach the material box, the robot claws of each robot arm are controlled to put down the workpiece; if within the specified time step, the end effector of either the navigator robot arm or the collaborator robot arm does not reach the material box, or the two robots are out of sync at any time step, repeat step three until the end effectors of both the navigator robot arm and the collaborator robot arm reach the material box.

3. According to the dual-manipulator collaborative handling system based on deep reinforcement learning in claim 1, it is characterized in that: In step 1, the navigator robot arm uses KUKA LBR iiwa 7R800, the collaborator robot arm uses UniversalRobots UR5, and the robot gripper uses Robotiq 140.

4. According to the dual-manipulator collaborative handling system based on deep reinforcement learning in claim 1, it is characterized in that: In step 2, the feature extraction network can be ResNet.

5. The dual-manipulator collaborative handling system based on deep reinforcement learning according to claim 1 is characterized in that: In step 2, grasping point recognition relies on the analysis of the surface features of the object, including shape, texture, and color.

Citation Information

Patent Citations

  • Spatial double-arm system constraint motion planning method based on deep intensive learning

    CN112171660A

  • Human-machine cooperation pipeline system based on deep reinforcement learning

    CN113341706A

  • Two-arm robot grabbing pose selection method and system and storage medium

    CN115008478A

  • Assembly line double-mechanical-arm cooperative grabbing method based on reinforcement learning

    CN117207186A

  • Method and system for capturing objects using robotic device

    RU2745380C1

Cited By

  • Intelligent control system for gray fabric carrying and stacking manipulator

    CN119460760A

  • Mobile mechanical arm grabbing system based on reinforcement learning cooperative controller and implementation method thereof

    CN119704183A

  • Cooperative control method for double-arm robot

    CN121157045A

  • A dual-arm robot cooperative control method

    CN121157045B