A dual-robot arm cooperative carrying system based on deep reinforcement learning

Abstract

The application relates to a dual-robot arm cooperation system based on deep reinforcement learning, which performs environment perception and target detection through a camera, uses a Faster R-CNN to identify a grabbing point, and adopts a VGG-16 network to assist in grabbing strategy decision-making. The system innovatively uses a single SAC algorithm based on an Actor-Critic form to control the two arms to cooperatively work, and realizes the intelligentization of grabbing and carrying tasks. In the carrying process, the leader and the collaborator arm dynamically adjust the action according to the improved CSAC algorithm, ensure efficient path planning and posture synchronization, wherein the reward mechanism promotes target approach while avoiding collision, and maintains the cooperation stability under closed-loop constraints. The whole system is verified in a CoppeliaSim simulation environment, and exhibits the whole-process automatic processing capability from object identification to stable carrying.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot motion control, and particularly relates to a dual-arm multi-step grasping and collaborative handling system based on deep reinforcement learning. BACKGROUND

[0002] With the advent of the Industry 4.0 era, the demand for automation and intelligent production is growing, and higher requirements are put forward for the flexibility, collaborative work capability and intelligent decision-making level of robot systems.

[0003] Existing dual-arm collaborative systems mainly rely on traditional control methods, which are usually based on inverse kinematics to solve the inverse problem, combined with pre-defined path planning algorithms to achieve coordinated motion of the arms. For example, some studies use inverse kinematics solvers combined with pre-defined path planning algorithms, and continuously adjust the path of the arms through optimization algorithms to achieve the expected collaborative effect. These systems have certain advantages in precision and stability, but have limited flexibility and intelligent decision-making capability when faced with complex environments and task changes.

[0004] Existing dual-arm collaborative systems are only suitable for single customized application scenarios, and their main drawback is that all motion tasks rely on inverse kinematics solving. This method has heavy computational burden when faced with complex, multi-constrained collaborative tasks, and the accuracy of the solution depends on the accuracy of the model. In dynamic environments, traditional methods are difficult to respond quickly to changes, and cannot achieve efficient real-time control. In addition, inverse kinematics solving is essentially a deterministic calculation, which is difficult to adapt to environmental uncertainty and the randomness of operation tasks. Therefore, in recent years, research based on deep reinforcement learning (DRL) has gradually attracted attention. This method allows the agent to repeatedly try and error in a simulated environment, learn strategies, so that the robot can make autonomous decisions and adapt to changes in the environment.

[0005] However, existing methods often combine deep reinforcement learning with inverse kinematics solving, using DRL to determine the target position of the arm, and then calculating the specific joint angles through inverse kinematics. Although this method improves the intelligence level of the system, it still retains the shortcomings of inverse kinematics solving, such as high complexity, accuracy still dependent on the model, and difficulty in handling coordinated motion under multi-constrained conditions. In addition, this method often cannot handle the collaboration and coupling problem between multiple arms when faced with complex closed-chain constraint tasks. SUMMARY

[0006] To address the problem that existing dual-arm collaborative systems are difficult to handle the collaboration and coupling problem between dual arms when facing complex closed-chain constraint tasks, a dual-arm collaborative handling system based on deep reinforcement learning is proposed.

[0007] The technical solution of this invention is as follows:

[0008] A collaborative material handling system with two robotic arms based on deep reinforcement learning is constructed by the following steps:

[0009] Step 1: State perception and environment modeling. Use cameras to collect information about the surrounding environment and target objects to construct a digital representation of the scene, which will then be used as a basis for decision-making by deep reinforcement learning algorithms.

[0010] A collaborative material handling system with two robotic arms was built based on CoppeliaSim. The system includes an operating table, a navigator robotic arm, a collaborator robotic arm, a robotic gripper, a workpiece, and a material box.

[0011] The control panel consists of three parts: two ends and a middle section. One end is equipped with a navigator robotic arm, the other end with a collaborator robotic arm, and the middle section has workpieces and material boxes spaced at intervals. A depth camera is located directly above the control panel surface to collect visual information for visual algorithms to identify gripping points. The bases of the navigator and collaborator robotic arms are located at both ends of the control panel, and the initial poses of their end effectors are on both sides of the control panel. Both robotic arms are equipped with robotic grippers at their ends, ready to grasp workpieces. Once the navigator and collaborator robotic arms have grasped a workpiece, they move synchronously to the top of the material box and place the workpiece inside.

[0012] Step 2: Based on the visual information collected by the camera, use visual algorithms to identify the grab points;

[0013] Step 3: Based on the grasping points identified in Step 2, the SAC deep reinforcement learning algorithm, which belongs to the Actor-Critic type, controls the dual robotic arms to grasp the object.

[0014] Step 4: Use an improved dual-robotic-arm collaborative deep reinforcement learning algorithm to move the object to the target point.

[0015] Furthermore, the specific setup steps are as follows:

[0016] Step 1: Use CoppeliaSim to build a simulation environment to train the deep reinforcement learning algorithm for collaborative handling of dual robotic arms. The proposed deep reinforcement learning algorithm is trained on the collaborative handling system of dual robotic arms built based on CoppeliaSim.

[0017] The dual-arm collaborative handling system includes an operating table, a navigator robotic arm, a collaborator robotic arm, a robotic gripper, a workpiece, and a material box;

[0018] The operation table includes three parts of two ends and a middle part, one end of which is provided with a navigator mechanical arm, the other end is provided with a collaborator mechanical arm, and the middle part is provided with workpieces and material boxes at intervals, and a depth camera is arranged above the operation table top to collect visual information for visual algorithm to identify the grabbing point; the bases of the navigator mechanical arm and the collaborator mechanical arm are located at the two ends of the operation table, and the initial poses of the end effectors of the two are located on the two sides of the operation table, and the end effectors of the two are provided with mechanical claws for waiting for grabbing workpieces; after the navigator mechanical arm and the collaborator mechanical arm grab the workpieces, they are moved to above the material box in cooperation and synchronization to place the workpieces in the material box; wherein the base of the mechanical arm is fixed on the two sides of the operation table, and the position of the base will not change in the working process, the "end" refers to the end of the last connecting rod of the mechanical arm, and the end is usually provided with a mechanical claw, and in the working process, the pose of the end of the mechanical arm will change, so the initial pose of the end when the work has not started needs to be indicated;

[0019] Step two: according to the visual information collected by the camera, a visual algorithm is used to identify the grabbing point;

[0020] The classic Faster R-CNN architecture is used for target detection, and the architecture includes three main parts: a feature extraction network, a region proposal network and a classification and bounding box regression layer;

[0021] The image output by the camera is I, which is a three-dimensional tensor; the input image I of the feature extraction network is output as a feature map F after convolution and pooling layers:

[0022] F=f conv (I) (1)

[0023] Where f conv (·) represents the forward propagation process of the convolution network;

[0024] A window with a predefined size and aspect ratio is used to slide on the feature map F, and if the window slides to a position of the feature map that contains a detection target, it is defined as r k and added to the candidate box set R; sliding on the feature map F generates a series of candidate box set R={r1,r2,…,r k ,},k is the index of the candidate box containing the detection target, and each candidate box r k is defined by its upper left corner coordinates (x k ,y k ), width w k and height h k ; the region proposal network provides an object score s k, for measuring the probability of the target contained in the candidate box; converting each candidate box containing the target into a fixed-size feature vector, and then sending it into a fully connected layer for classification and bounding box fine-tuning; let C be the total number of classes of the target to be detected, then for each candidate box r k , the probability of belonging to the c-th class is p k,c , and the corrected bounding box coordinates are Finally, through threshold screening, the target box with a confidence higher than a set value is retained and mapped back to the original image to achieve accurate positioning of the target; wherein the total number of classes of the target to be detected is C, and the candidate box r k has a probability for each class, that is, there are C probabilities, and the lower-case c represents the class in the upper-case C.

[0025] Once the target object is accurately detected and positioned, the next step is to identify the appropriate grasping point; the grasping point identification relies on the analysis of the surface features of the object, including shape, texture, and color; first, the VGG-16 feature extraction network is applied to the target region to obtain the feature map F target ; then, the features around the candidate grasping point are extracted through the region of interest alignment operation to form a feature vector set F g ={f g1 ,f g2 ,...,f gm ,...}, where m is the index of the candidate point, f gm is the coordinates of the m-th candidate grasping point; s g is the grasping stability score, the coordinates of the centroid of the grasped object are (x z ,y z ,z z ), the coordinates of the candidate grasping point are (x,y,z), and the calculation formula of the grasping stability score s g is:

[0026]

[0027] In formula (2), η is a positive constant, which determines the rate of influence on the score, and l is one-third of the length of the grasped object; a larger k value indicates that the score decreases faster as increases, reflecting the preference for grasping near the centroid; when the grasping point is close to (x z ,y z ±l,z z ) (i.e. ), s g is close to 1, and the stability score is high; as increases, s g quickly decreases, and the stability score decreases;

[0028] Grasping stability scoresg The suitability of each candidate grasping point is evaluated, with input being the coordinates f of the candidate grasping point gm = (x m ,y m ,z m ), then according to formula (2) we have:

[0029]

[0030] By traversing all candidate points, the point with the highest score is selected as the final grasping point g * :

[0031] g * = argmaxs g (f gm ) (4)

[0032] Step three: using the deep reinforcement learning algorithm SAC belonging to the Actor-Critic form to control the double robot arms to grasp the workpiece according to the final grasping point g * identified in step two;

[0033] Different from the traditional scheme of using one Actor network to control one robot arm, one Actor network is used to control two robot arms; for each time step t, the current environment state s of the robot agent t , the action a t = [a1, a2] is selected and the joint angle increments a1 and a2 are taken out from it and applied to the leader robot arm and the collaborator robot arm respectively, and the next environment state s t+1 is obtained by interacting with the environment, and the rewards r N , r C of the two robot arms, then the total reward r t of the agent at time step t is:

[0034]

[0035] r t = r N + r C (7)

[0036] In the above formula, k1 and k2 are regularization coefficients, p1 and p2 are the Cartesian coordinates of the two robot arms respectively, is the final grasping point coordinates, distance(·) is the distance reward, which encourages approaching the target material box, and Collision(·) is the collision reward, which avoids collision of the robot arms;

[0037] The four-tuple (s t, a t , s t+1 , r t) stored in the memory bank, after each time step t interacting with the environment, for updating the Actor network π of SAC φ and the Critic network

[0038] The Actor network aims to optimize the policy π φ to maximize the expected long-term reward plus the policy entropy to increase exploratory; the update formula is as follows:

[0039]

[0040] In formula (8), J(φ) is the target formula for updating the Actor network, α is the entropy adjustment coefficient; φ is the parameter of the Actor network; π(a|s,φ) is the probability of selecting action a depending on state s and parameter φ; Q q (s,a|θ q ) represents the evaluation of the value of the expected cumulative discounted return that the qth Critic network can obtain after taking action a in the given state s, and making subsequent decisions according to the current policy;

[0041] The update target of the Critic network is:

[0042]

[0043] In formula (9), JQ is the target formula for updating the Critic network, r is the immediate reward, γ is the discount factor, δ is the indicator function of the terminal state, which is 0 when the terminal state, otherwise 1, a' is the sampled action of the next state s', π(a'|s') is the policy probability of the action, and α is the entropy adjustment coefficient;

[0044] After interacting with the environment n times, if the end effectors of the leader robot arm and the collaborator robot arm reach the respective graspable points g * , then control the gripper of each robot arm to grasp the workpiece, and then according to step four, cooperatively carry the workpiece to the material box; if within the specified time step, either end effector of the leader robot arm and the collaborator robot arm does not reach the graspable point g * , then repeat step three until the end effectors of the leader robot arm and the collaborator robot arm reach the respective graspable points g * ;

[0045] Step four: using a dual-robot arm cooperative deep reinforcement learning algorithm to carry the object to the target point;

[0046] In the dual-robot arm cooperative carrying task, the leader robot arm Arm Navigator and the collaborator robot arm Arm Collaborator; the main responsibility of the leader robot arm is to find an efficient path to reach the preset goal g by improved collaborative SAC algorithm CSAC strategy learning an efficient path to reach the preset goal g, where a Nav represents the action of the leader robot arm, s={p Nav ,p g ,E} is the state space, including the position of the leader p Nav , the target position p g and the surrounding environment E; the reward function r Nav of the leader robot arm is designed as:

[0047] r Nav (s,a Nav )=λ1·distance(p Nav ,p g )+λ2·Collision(s,a Nav ) (10)

[0048] In equation (10), λ1, λ2 are regularization coefficients, distance() is the distance reward, which encourages approaching the target bin, and Collision(·) is the collision reward, which avoids collision of the robot arm;

[0049] The collaborative reward collaboration() is used to maintain the relative pose relationship between the leader robot arm and the collaborator robot arm. The dual-robot arm collaborative handling system meets the closed-chain constraint in mechanical structure, and the homogeneous transformation matrix of the grasp point g * relative to the world coordinate system can be obtained from the relevant homogeneous transformation matrix of the two robot arms, and the formula is as follows:

[0050]

[0051] In equation (11), i=1,2 represents the index of the two robot arms, W, base, end represent the coordinate system with the world, robot arm base and robot arm end effector as the coordinate origin respectively;

[0052] Through the homogeneous transformation matrix of the two robot arms, the homogeneous transformation matrix of the end effector of the robot arm with index 2 relative to the end effector of the robot arm with index 1 can be derived as follows: equation (12):

[0053]

[0054] When step three is completed, the two robot arms reach the grasp point g *At this time, the closed-loop constraint is satisfied, and the homogeneous transformation matrix of the base of the leader robot arm relative to the world coordinate system can be calculated by robot kinematics and robot parameters The homogeneous transformation matrix of the end effector of the leader robot arm relative to the base of the leader robot arm The homogeneous transformation matrix of the base of the collaborator robot arm relative to the world coordinate system And the homogeneous transformation matrix of the end effector of the collaborator robot arm relative to the base of the collaborator robot arm According to formula (12), the homogeneous transformation matrix T satisfying the closed-loop constraint can be further obtained gt ; In the process of dual-robot collaborative carrying, the difference between the homogeneous transformation matrix T in the process of calculating formula (12) And the homogeneous transformation matrix T satisfying the closed-loop constraint gt , that is, the collaboration reward can be obtained; the second transformation matrix of the base of the robot relative to the world coordinate system And is fixed and does not change with the process of dual-robot collaborative carrying, and the homogeneous transformation matrix of the end effector of the collaborator robot arm relative to the end effector of the leader robot arm is calculated Only by calculating the homogeneous transformation matrix of the end effector of the robot relative to the base of the robot through robot kinematics and robot parameters And ;

[0055] The collaboration(·) collaboration reward is evolved according to the Euler rotation theorem, and its formula is:

[0056]

[0057] In formula (13), T jj represents the jth row and jth column element of matrix T;

[0058] The collaborator robot arm uses the CSAC strategy to realize synchronization with the leader robot arm, wherein the state s colla ={p Nav ,a Nav ,p Colla} not only contains the position and action of the leader robot arm, but also the position p Colla of the end effector of the collaborator robot arm; the learning goal of the collaborator robot arm is to maintain the relative attitude and position relationship by predicting the behavior of the leader robot arm and adjusting its own action, and to ensure the following accuracy; the strategy of the collaborator robot arm aims to maximize the long-term reward, considering the following efficiency, synchronization and adaptability, and the reward function r Colla (s Colla ,a Colla) as follows:

[0059]

[0060] In formula (14), μ1, μ2 and μ3 are regularization coefficients, collaboration(·) is a collaboration reward, and T gt are the homogeneous transformation matrix between the end effector pose of the leader robot arm and the end effector pose of the collaborator robot arm in the training process and the homogeneous transformation matrix satisfying the closed-loop constraint, respectively;

[0061] The improved CSAC algorithm combines the standard SAC algorithm with an additional penalty term between the homogeneous transformation matrix and the homogeneous transformation matrix T gt satisfying the closed-loop constraint, aiming to improve the coordinated control performance of the dual robot arm system; first, the homogeneous transformation matrix between the end effector of the collaborator robot arm and the end effector of the leader robot arm in the collaboration process is calculated, and then the mean square error (MSE) loss between and the homogeneous transformation matrix T gt satisfying the closed-loop constraint is calculated:

[0062]

[0063] In formula (15), B represents the number of elements in the matrix, and b represents the index of the element in the matrix;

[0064] The improved CASC algorithm considers the mean square error (MSE) loss between the homogeneous transformation matrix between the end effector of the collaborator robot arm and the end effector of the leader robot arm and the homogeneous transformation matrix T gt satisfying the closed-loop constraint in the update formula of the Q value, and the update target Q value calculation formula of the improved CSAC algorithm is as follows:

[0065]

[0066] In formula (16), r is the immediate reward, γ is the discount factor, α is the weight of controlling entropy, β is the importance of adjusting the end effector position error, Q min (s′,a′)=min(Q1(s′,a′),Q2(s′,a′)),Q1 and Q2 are two action value networks in the Critic network, respectively;

[0067] The update formula of the Critic network of the improved CASC algorithm is as follows:

[0068]

[0069] The update formula of the Actor network of the improved CASC algorithm is:

[0070] L actor = -E s～D [Q min (s,a)-αlogπ(a|s)] (18)

[0071] In formula (18), D represents a set of environment states, E s～D represents the expectation of the target formula when taking state s from the environment state set D.

[0072] After interacting with the environment for n times, if the end effectors of the leader robot arm and the collaborator robot arm both reach the material box, the mechanical gripper of each robot arm is controlled to put down the workpiece; if in the specified time step, the end effectors of the leader robot arm and the collaborator robot arm do not reach the material box, or the two robot arms are not synchronized at any time step, step three is repeated until the end effectors of the leader robot arm and the collaborator robot arm both reach the material box.

[0073] Preferably, in step one, the leader robot arm selects KUKA LBR iiwa 7R800, the collaborator robot arm selects Universal Robots UR5, and the mechanical gripper adopts Robotiq 140.

[0074] Preferably, the feature extraction network in step two can be ResNet.

[0075] Preferably, in step two, the grasping point identification depends on the analysis of the surface features of the object, including shape, texture and color.

[0076] The present application has the following advantages:

[0077] Deep reinforcement learning (DRL) as a bridge connecting artificial intelligence and robot control enables robots to learn optimal strategies through continuous trial and error, effectively solving the problem that traditional programming is difficult to cope with in complex and dynamic environments. Through the deep reinforcement learning framework, the dual-robot arm collaboration system not only can achieve precise object grasping, carrying and placing, but also can learn how to optimally allocate tasks, avoid collisions and adaptively adjust strategies in dynamic environments during task execution, thereby significantly improving work efficiency and safety. In addition, combined with advanced visual recognition and environmental perception technology, the system can understand and adapt to the working environment in real time, effectively identify and process various objects, and is widely applicable to intelligent manufacturing, logistics sorting, disaster rescue and other fields, promoting the development of robot technology towards more intelligent and flexible directions. BRIEF DESCRIPTION OF DRAWINGS

[0078] Figure 1 A double-arm collaborative carrying operation system diagram built by using CoppeliaSim simulation software for the present application;

[0079] Figure 2 A workflow diagram of the double-arm collaborative carrying system based on deep reinforcement learning for the present application;

[0080] Figure 3 An algorithm update flowchart for the double arms to reach the grabbing point for the present application;

[0081] Figure 4 An algorithm update flowchart for the double-arm collaborative carrying for the present application.

[0082] The accompanying drawings illustrate the present application:

[0083] 1, operation table; 2, leader arm; 3, collaborator arm; 4, mechanical gripper; 5, workpiece; 6, material box. DETAILED DESCRIPTION

[0084] The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The present embodiment is implemented on the premise of the technical solution of the present application, and detailed implementation and specific operation processes are given, but the protection scope of the present application is not limited to the following embodiments.

[0085] A double-arm collaborative carrying system based on deep reinforcement learning, built by the following steps:

[0086] Step one: use CoppeliaSim (a robot simulation platform) to build a simulation environment to train the double-arm collaborative carrying of the deep reinforcement learning algorithm, which is due to the high cost, potential mechanical damage risk and data collection limitation challenges faced in the real world. The simulation environment not only can safely and cost-effectively generate diversified training scenarios and massive data to support the double arms to continuously learn and optimize strategies in complex carrying tasks, but also can enhance the generalization ability of the algorithm by fine control of experimental conditions, and ensure the best learning effect without damaging the entity equipment, laying a solid foundation for actual deployment. In the present embodiment, the double-arm collaborative carrying system built based on CoppeliaSim as shown in Figure 1 is used to train the deep reinforcement learning algorithm proposed in the present application.

[0087] The double-arm collaborative carrying system built based on CoppeliaSim as shown in Figure 1 includes an operation table 1, a leader arm 2, a collaborator arm 3, a mechanical gripper 4, a workpiece 5, and a material box 6.

[0088] The operation table 1 includes three parts of two ends and a middle part, one end of which is provided with a leader mechanical arm 2, the other end of which is provided with a collaborator mechanical arm 3, and the middle part is provided with workpieces 5 and material boxes 6 at intervals, and a depth camera is arranged above the operation table 1, which collects visual information for a visual algorithm to identify a grabbing point; the bases of the leader mechanical arm 2 and the collaborator mechanical arm 3 are located at the two ends of the operation table 1, and the initial poses of the end effectors of the two are on the two sides of the operation table, and the tips of the two mechanical arms are provided with mechanical grippers 4, which are waiting to grab the workpieces 5; after the leader mechanical arm 2 and the collaborator mechanical arm 3 grab the workpieces 5, they are moved synchronously to above the material boxes 6 and place the workpieces 5 in the material boxes 6; the working flow chart of the double-robot arm collaborative carrying system based on deep reinforcement learning is shown in Figure 2 In this embodiment, the leader mechanical arm 2 is KUKA LBR iiwa7R800, the collaborator mechanical arm 3 is Universal Robots UR5, and the mechanical gripper 4 is Robotiq 140. Among them, the base of the mechanical arm is fixed on the two sides of the operation table, and the position of the base will not change in the working process, the "tip" refers to the tip of the last link of the mechanical arm, and the tip is usually provided with a mechanical gripper, and the pose of the tip of the mechanical arm will change in the working process, so the initial pose of the tip at the beginning of the work needs to be indicated.

[0089] Step two: according to the visual information collected by the camera, a visual algorithm is used to identify the grabbing point;

[0090] The classic Faster R-CNN architecture is used for target detection, which includes three main parts: feature extraction network (such as ResNet), region proposal network (Region Proposal Network, RPN) and classification and bounding box regression layer.

[0091] The image output by the camera is I, which is a three-dimensional tensor. The input image I of the feature extraction network is output as a feature map F after convolution and pooling:

[0092] F = f conv (I) (1)

[0093] Where f conv (·) represents the forward propagation process of the convolution network.

[0094] A window with a predefined size and aspect ratio is used to slide on the feature map F, and if the window slides to a position on the feature map that contains a detection target, it is defined as r k And added to the candidate box set R. Sliding on the feature map F generates a series of candidate box sets R = {r1, r2,..., r k}, k is the index of the candidate box containing the detection target, each candidate box r k is defined by its top-left corner coordinates (x k ,y k ), width w k and height h k . The region proposal network provides an object score s k for each candidate box, which measures the probability of the candidate box containing a target. Each candidate box containing a target is converted into a fixed-size feature vector, which is then fed into a fully connected layer for classification and bounding box refinement. Let C be the total number of classes of the target to be detected, then for a candidate box r k , the probability of it belonging to the c-th class (the total number of classes of the target to be detected is C, and the candidate box r k belongs to the c-th class) is pk,c , and the corrected bounding box coordinates are Finally, through threshold screening, the target box with a confidence higher than the set value is retained and mapped back to the original image to achieve accurate positioning of the target.

[0095] Once the target object is accurately detected and positioned, the next step is to identify the appropriate grasping point. Grasping point identification relies on the analysis of object surface features, including shape, texture, color, etc. First, apply the VGG-16 feature extraction network to the target region to obtain the feature map F target . Then, through the region of interest alignment operation, the features around the candidate grasping point are extracted to form a feature vector set F g = {f g1 ,f g2 ,...,f gm ,...}, where m is the index of the candidate point, f gm is the coordinates of the m-th candidate grasping point. s g is the grasping stability score, the coordinates of the center of mass of the grasped object are (x z ,y z ,z z ), the coordinates of the candidate grasping point are (x, y, z), and the calculation formula of the grasping stability score s g is:

[0096]

[0097] In equation (2), η is a positive constant, which determines the rate of influence on the score, and l is one-third of the length of the grasped object. A larger k value indicates that the score decreases faster as increases, reflecting a preference for grasping near the center of mass. When the grasping point is close to the center of mass (xz ,y z ±l,z z )(Right now When ), s g A value close to 1 indicates a high stability score; as Increase, s g Rapid decrease leads to a reduction in the stability score.

[0098] Capture stability score s g It is used to evaluate the suitability of each candidate grab point, with the input being the coordinates f of the candidate grab point. gm =(x m ,y m ,z m ), then according to formula (2) we have:

[0099]

[0100] By iterating through all candidate points, the point with the highest score is selected as the final grab point g. * :

[0101] g * =argmaxs g (f gm (4)

[0102] Step 3: Based on the final grab point g identified in Step 2 * The SAC deep reinforcement learning algorithm, which belongs to the Actor-Critic type, is used to control the dual robotic arms to grasp the workpiece.

[0103] Unlike the traditional approach where one Actor network controls one robotic arm, this method uses one Actor network to control two robotic arms. For each time step t, the current environmental state s of the robotic agent... t Select action a t = [a1, a2] and extract the joint angle increments a1 and a2 respectively, apply them to the navigator robotic arm 2 and the collaborator robotic arm 3, interact with the environment, and obtain the next environmental state s. t+1 The reward r of the two robotic arms N r C (The English words for navigator and collaborator are 'Navigator' and 'Collarator', respectively.) The total reward rt of the agent at time step t is:

[0104]

[0105] r t =r N +r C (7)

[0106] In the above formula, k1 and k2 are regularization coefficients, p1 and p2 are the Cartesian coordinates of the two mechanical arms respectively, is the final grasping point coordinate, distance(·) is the distance reward, which encourages approaching the target material box 6, and Collision(·) is the collision reward, which avoids collision of the mechanical arms.

[0107] The quadruple (s t, a t , r t+1 , a t ) is stored in the memory, and after each time step t of interaction with the environment, it is used to update the Actor network π φ and the Critic network q = 1, 2.

[0108] The Actor network aims to optimize the policy π φ to maximize the expected long-term reward plus the policy entropy to increase exploratory. The update formula is as follows:

[0109]

[0110] In formula (8), J(φ) is the target formula for updating the Actor network, α is the entropy adjustment coefficient; φ is the parameter of the Actor network; π(a|s, φ) is the probability of selecting action a depending on state s and parameter φ; Q q (s, a|θ q ) represents the evaluation of the value of the expected cumulative discounted reward that the qth Critic network can obtain after taking action a in the given state s, following the subsequent decision-making according to the current policy.

[0111] The update target of the Critic network is:

[0112]

[0113] In formula (9), J Q is the target formula for updating the Critic network, r is the immediate reward, γ is the discount factor, δ is the indicator function of the terminal state (0 when terminated, otherwise 1), a' is the sampled action of the next state s', π(a'|s') is the policy probability of the action, and α is the entropy adjustment coefficient.

[0114] After interacting with the environment n times, if the end effectors of the leader arm 2 and the collaborator arm 3 both reach the respective graspable points g * , the mechanical gripper 4 of each arm is controlled to grasp the workpiece 5, and then the workpiece 5 is cooperatively carried to the material box 6 according to step four; if either end effector of the leader arm 2 and the collaborator arm 3 does not reach the graspable point g *If not, repeat Step Three until both the leader arm 2 and the collaborator arm 3 end effectors reach their respective graspable points g * The algorithm flowchart for Step Three, the dual-arm reaching graspable point, is shown in FIG. 4. Figure 3

[0115] Step Four: Use the dual-arm collaborative deep reinforcement learning algorithm to transport the object to the target point.

[0116] In the dual-arm collaborative transportation task, the leader arm 2 (Arm Navigator ) and the collaborator arm 3 (Arm Collaborator ) are involved. The main responsibility of the leader arm 2 is to learn an efficient path to reach the pre-set target g through the improved collaborative SAC algorithm (CSAC) policy π θNav (a Nav |s), where a Nav represents the action of the leader arm 2, s = {p Nav , p g , E} is the state space, which includes the leader position p Nav , the target position p g and the surrounding environment E. The leader arm reward function r Nav is designed as:

[0117] r Nav (s, a Nav ) = λ1 · distance(p Nav , p g ) + λ2 · Collision(s, a Nav ) (10)

[0118] In equation (10), λ1, λ2 are regularization coefficients, distance(·) is the distance reward, which encourages approaching the target bin 6, and Collision(·) is the collision reward, which avoids collision between the arms.

[0119] The collaboration reward collaboration() is used to maintain the relative pose relationship between the leader arm 2 and the collaborator arm 3. The dual-arm collaborative transportation system satisfies the closed-chain constraint in mechanical structure, and the graspable point g * relative to the world coordinate system can be obtained from the relevant homogeneous transformation matrix of the two arms, and its formula is as follows:

[0120]

[0121] ​​In equation (11), i = 1, 2 represent the indices of the two robotic arms, and W, base, and end represent the coordinate systems with the world, the robotic arm base, and the robotic arm end effector as the coordinate origins, respectively.

[0122] Through the homogeneous transformation matrix of the two robotic arms The homogeneous transformation matrix of the end effector of the robotic arm with index 2 relative to the end effector of the robotic arm with index 1 can be derived. The following formula (12):

[0123]

[0124] When step three is completed, both robotic arms reach the gripping point g. * At this point, the closed-loop constraint is satisfied. Using robot kinematics and robot parameters (basic knowledge of robot kinematics), the homogeneous transformation matrix of the navigator robot arm 2's base relative to the world coordinate system can be calculated. Homogeneous transformation matrix of the end effector of Navigator Robotic Arm 2 relative to the base of Navigator Robotic Arm 2 Homogeneous transformation matrix of the base of the Collaborator robotic arm 3 relative to the world coordinate system Homogeneous transformation matrix of the end effector of collaborator robotic arm 3 relative to the base of collaborator robotic arm 3 According to equation (12), the homogeneous transformation matrix T that satisfies the closed-loop constraint can be further obtained. gt In the collaborative handling process of the two robotic arms, the homogeneous transformation matrix in the process is calculated using equation (12). With the homogeneous transformation matrix T satisfying the closed-loop constraint gt The difference will be rewarded for collaboration. The secondary transformation matrix of the robotic arm base relative to the world coordinate system. and It is fixed and does not change during the collaborative handling process of the two robotic arms. Calculate the homogeneous transformation matrix of the end effector of the collaborating robotic arm 3 relative to the end effector of the navigating robotic arm 2. Simply calculate the homogeneous transformation matrix of the robot arm's end effector relative to the robot arm's base using the robot's kinematics and parameters. and That's all.

[0125] The collaboration reward is derived from Euler's rotation theorem, and its formula is as follows:

[0126]

[0127] In equation (13), T jj This represents the element in the j-th row and j-th column of matrix T.

[0128] The follower robot arm 2 uses the CSAC strategy to achieve synchronization with the leader robot arm, where the state s colla = {p Nav , a Nav , p Colla} contains not only the position and action of the leader robot arm 2, but also the position p Colla of the end effector of the collaborator robot arm 3. The learning goal of the collaborator robot arm 3 is to maintain the relative pose and position relationship by predicting the behavior of the leader robot arm and adjusting its own action to ensure the following accuracy. The strategy of the collaborator robot arm 3 aims to maximize the long-term reward, taking into account the following efficiency, synchronization and adaptability. The reward function r Colla (s Colla , a Colla ) of the collaborator robot arm 3 is as follows:

[0129]

[0130] In equation (14), μ1, μ2 and μ3 are regularization coefficients, collaboration(·) is the collaboration reward, and T gt are the homogeneous transformation matrix between the end effector of the leader robot arm and the end effector of the collaborator robot arm in the training process and the homogeneous transformation matrix satisfying the closed-loop constraint, respectively.

[0131] The improved CSAC algorithm combines the standard SAC algorithm with an additional penalty term between the homogeneous transformation matrix of the end effector of the collaborator robot arm 3 relative to the end effector of the leader robot arm 2 in the process and the homogeneous transformation matrix T gt satisfying the closed-loop constraint, aiming to improve the coordinated control performance of the dual robot arm system. First, the homogeneous transformation matrix of the end effector of the collaborator robot arm relative to the end effector of the leader robot arm in the collaborative handling process is calculated, and then the mean square error (MSE) loss between and the homogeneous transformation matrix T gt satisfying the closed-loop constraint is calculated:

[0132]

[0133] In equation (15), B represents the number of elements in the matrix, and b represents the index of the element in the matrix.

[0134] The improved CASC algorithm considers the homogeneous transformation matrix of the end effector of the collaborator robot arm 3 relative to the end effector of the leader robot arm 2 in the update formula of the Q value and the homogeneous transformation matrix T gtThe average mean square error (MSE) loss between them, the update target Q value calculation formula of the improved CSAC algorithm is as follows:

[0135]

[0136] In formula (16), r is the instant reward, gamma is the discount factor, alpha is the weight of entropy control, beta is the importance of adjusting the end effector position error, Q min (s',a')=min(Q1(s',a'),Q2(s',a')), Q1 and Q2 are two action value networks in the Critic network respectively.

[0137] The update formula of the Critic network of the improved CASC algorithm is as follows:

[0138]

[0139] The update formula of the Actor network of the improved CASC algorithm is as follows:

[0140] L actor = -E s～D [Q min (s,a)-alpha log pi(a|s)] (18)

[0141] In formula (18), D represents a set of environment states, E s～D represents the expectation of the target formula when the state s is taken from the environment state set D.

[0142] After interacting with the environment for n times, if the end effectors of the leader robot arm 2 and the collaborator robot arm 3 reach the material box 6, the mechanical gripper 4 of each robot arm is controlled to put down the workpiece 5; if any end effector of the leader robot arm 2 and the collaborator robot arm 3 does not reach the material box 6 within the specified time step, or any time step is out of sync, step three is repeated until the end effectors of the leader robot arm 2 and the collaborator robot arm 3 reach the material box 6, and the algorithm update flowchart of the double-arm cooperative carrying is as shown in Figure 4 .

[0143] The above-described embodiment only expresses one embodiment of the present application, which is described in detail, but it cannot be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A dual robot arm collaborative handling system based on deep reinforcement learning, characterized by, The following steps are built: Step one: state awareness and environment modeling, using a camera to collect information about the surrounding environment and target objects, and to construct a digital representation of the scene for the deep reinforcement learning algorithm to make decisions; A dual-arm cooperation carrying system is built based on CoppeliaSim, which includes an operating table, a leader robot arm, a follower robot arm, a mechanical gripper, a workpiece and a material box; The operating table includes three parts: two ends and a middle part. The leader robot arm is arranged on one end, the follower robot arm is arranged on the other end, and the workpiece and the material box are arranged at intervals on the middle part. There is a depth camera above the operating table surface to collect visual information for the visual algorithm to identify the grabbing point. The bases of the leader robot arm and the follower robot arm are located at both ends of the operating table, and the initial poses of the end effectors of the two are on both sides of the operating table. The mechanical gripper is installed at the end of each robot arm, waiting to grab the workpiece. After the leader robot arm and the follower robot arm grab the workpiece, they move cooperatively and synchronously above the material box and place the workpiece in the material box. Step two: use the visual algorithm to identify the grabbing point according to the visual information collected by the camera; The classic Faster R-CNN architecture is used for target detection, which includes three parts: feature extraction network, region proposal network and classification and bounding box regression layer. The image output by the camera is , which is a three-dimensional tensor; the input image of the feature extraction network After the convolutional layer and the pooling layer, the output feature map is : （1） wherein denotes the forward propagation process of the convolutional network; Using a pre-defined size and aspect ratio window to slide over the feature map If the window contains the detection target when it slides to a certain position on the feature map, it is defined as and added to the candidate box set Slide over the feature map , generate a series of candidate box sets that may contain the target , The index of the candidate box containing the detection target, in the coordinate system with the first pixel position in the upper left corner of the feature map as the origin, each candidate box is defined by its upper left corner coordinates , width and height ; the region proposal network provides an object score for each candidate box , which measures the probability of containing the target in the candidate box; convert each candidate box containing the target into a fixed-size feature vector, and then send it to the fully connected layer for classification and bounding box fine-tuning; let be the total number of classes of the target to be detected, then for the candidate box , the probability of belonging to the th class is , and the corrected bounding box coordinates are ; finally, through threshold screening, retain the target box with confidence higher than the set value, and map it back to the original image to achieve accurate positioning of the target; where the total number of classes of the target to be detected is , and the candidate box has a probability for each class, that is, there are probabilities, represented by lowercase c for the th class in uppercase C. Once the target object is accurately detected and located, the next step is to identify a suitable grasp point; grasp point identification relies on the analysis of the object surface features; first, the VGG-16 feature extraction network is applied to the target region to obtain a feature map ; then, the features around the candidate grasp point are extracted through the region of interest alignment operation to form a feature vector set , where is the index of the candidate point, is the coordinate of the th candidate grasp point; is the grasp stability score, the coordinate of the center of mass of the grasped object is , the coordinate of the candidate grasp point is , and the calculation formula of the grasp stability score is: （2） In equation (2) is a positive constant, determines the rate of influence of the score, is one third of the length of the object being grasped; larger values indicate that the score decreases more rapidly as increases, reflecting a preference for grasping near the center of mass; when the grasp point is close to i.e. , the score approaches 1 and the stability score is high; as increases, the score decreases rapidly and the stability score decreases. grabbing stability score is used to evaluate the suitability of each candidate grabbing point, with the input being the coordinates of the candidate grabbing point then according to equation (2) we have: （3） by traversing all the candidate points, selecting the highest scoring point as the final grasp point : （4） Step three: the dual-arm grabbing object is controlled by the deep reinforcement learning algorithm SAC in the form of Actor-Critic according to the grabbing point identified in step two. Step four: use the improved dual-arm cooperation deep reinforcement learning algorithm to carry the object to the target point.

2. The dual robot arm collaborative handling system based on deep reinforcement learning according to claim 1, characterized in that, The specific construction steps are as follows: Step one: use CoppeliaSim to build a simulation environment to train the dual-arm cooperation carrying of the deep reinforcement learning algorithm. The dual-arm cooperation carrying system built based on CoppeliaSim is used to train the deep reinforcement learning algorithm proposed; The dual-arm cooperation carrying system includes an operating table, a leader robot arm, a follower robot arm, a mechanical gripper, a workpiece and a material box. The operating table includes three parts: two ends and a middle part. The leader robot arm is arranged on one end, the follower robot arm is arranged on the other end, and the workpiece and the material box are arranged at intervals on the middle part. There is a depth camera above the operating table surface to collect visual information for the visual algorithm to identify the grabbing point. The bases of the leader robot arm and the follower robot arm are located at both ends of the operating table, and the initial poses of the end effectors of the two are on both sides of the operating table. The mechanical gripper is installed at the end of each robot arm, waiting to grab the workpiece. After the leader robot arm and the follower robot arm grab the workpiece, they move cooperatively and synchronously above the material box and place the workpiece in the material box. Step three: final grasping point identified according to step two a dual-arm workpiece grasping system is controlled using a deep reinforcement learning algorithm SAC belonging to the Actor-Critic form; Unlike the traditional scheme where one actor network controls one robot arm, we adopt one actor network to control two robot arms; for each time step , the current environment state of the robot agent , the agent selects an action and from which the joint angle increments , are extracted , are applied to the leader arm and the co-agent arm, which interact with the environment and obtain the next environment state , , the total reward of the agent at time step is: （5） （6） （7） In the above formula, and are regularization coefficients, and are the Cartesian coordinates of the two mechanical arms, is the final grasp point coordinate, is the distance reward, encouraging the approach to the target material box, is the collision reward, avoiding the collision of the mechanical arm; Quadruplet Stored in the memory bank, at each time step After interacting with the environment, for updating the Actor network of SAC and the Critic network , ; The actor network aims to optimize the policy to maximize the expected long-term reward plus the policy entropy to increase exploratory; the update formula is as follows: （8） In formula (8), is the target formula for updating the Actor network, is an entropy adjustment coefficient; is a parameter of the Actor network; is a probability of selecting an action depending on the state and the parameter ; represents the value of the expected cumulative discounted return that the th Critic network evaluates, given that the action is taken in the state and the subsequent decisions are made according to the current policy. The base of the robot arm is fixed on both sides of the operating table, and the position of the base does not change during work. The "end" refers to the end of the last link of the robot arm, which is usually equipped with a mechanical gripper. The pose of the robot arm end changes during work, so the initial pose of the end is specified when the work has not started. The update target of the Critic network is: （9） In equation (9), This is the objective formula for updating the Critic network. It's an instant reward. It is a discount factor. This is an indicator function for the termination status; it is 0 when the process terminates and 1 otherwise. The next state The sampling action, It is the policy probability of that action. It is the entropy adjustment coefficient; Interacting with the environment Subsequently, if the end effectors of the leader robot arm and the co-worker robot arm reach the respective graspable points , the gripper of the respective robot arm is controlled to grasp the workpiece, and then the workpiece is cooperatively transported to the bin according to step four; if the end effectors of the leader robot arm and the co-worker robot arm do not reach the graspable points within a specified time step, step three is repeated until the end effectors of the leader robot arm and the co-worker robot arm reach the respective graspable points . Step four: using double mechanical arm cooperation deep reinforcement learning algorithm to carry the object to the target point; In collaborative material handling tasks involving two robotic arms, the Navigator robotic arm is involved. With the collaborator robotic arm The navigator robotic arm's role is to employ an improved cooperative SAC algorithm and CSAC strategy. Learning to achieve the preset goals The efficient path, in which This indicates the movement of the navigator robotic arm. The state space contains the navigator's position. Target location and surrounding environment ; Navigator robotic arm reward function Designed as follows: （10） In formula (10), , is a regularization coefficient, is a distance reward, encouraging approaching the target bin, is a collision reward, avoiding collision of the robotic arm; Collaborative reward To maintain the relative pose relationship between the leader robot arm and the collaborator robot arm, the dual robot arm collaborative handling system meets the closed chain constraint in mechanical structure, the grabbing point Homogeneous transformation matrix relative to the world coordinate system Respectively by two robot arms related homogeneous transformation matrix, the formula is as follows: （11） In formula (11), denotes the index of two robot arms, , , respectively denote coordinate systems with the world, the robot base and the robot end effector as coordinate origins; A homogeneous transformation matrix of the two robot arms A homogeneous transformation matrix of the end effector of the robot arm with index 2 with respect to the end effector of the robot arm with index 1 can be derived As follows equation (12): （12） When step three is completed, the two robotic arms reach the grasp points respectively At this time, the closed-loop constraint is satisfied, and the homogeneous transformation matrix of the base of the leader robotic arm relative to the world coordinate system can be calculated respectively through the robot kinematics and the robot parameters , the homogeneous transformation matrix of the end effector of the leader robotic arm relative to the base of the leader robotic arm , the homogeneous transformation matrix of the base of the follower robotic arm relative to the world coordinate system and the homogeneous transformation matrix of the end effector of the follower robotic arm relative to the base of the follower robotic arm According to formula (12), the homogeneous transformation matrix satisfying the closed-loop constraint can be further obtained; during the dual-robotic-arm collaborative carrying process, the difference between the homogeneous transformation matrix in the calculation process of formula (12) and the homogeneous transformation matrix satisfying the closed-loop constraint , that is, the collaborative reward can be obtained; the second transformation matrix of the robotic arm base relative to the world coordinate system and is fixed and does not change with the dual-robotic-arm collaborative carrying process, and the homogeneous transformation matrix of the end effector of the follower robotic arm relative to the end effector of the leader robotic arm can be calculated only by calculating the homogeneous transformation matrix of the robotic arm end effector relative to the robotic arm base and through the robot kinematics and the robot parameters; The cooperative reward is evolved according to Euler's rotation theorem, and its formula is: （13） In formula (13), represents the element in the i-th row and the j-th column of the matrix ; and ; and ; and The collaborator robotic arm utilizes the CSAC strategy. To achieve synchronization with the navigator robotic arm, where the state... It includes not only the position and movement of the navigator robotic arm, but also the position of the collaborator robotic arm's end effector. The learning objective of the collaborator robotic arm is to predict the behavior of the navigator robotic arm and adjust its own movements to maintain relative posture and positional relationships, ensuring tracking accuracy. The collaborator robotic arm strategy aims to maximize long-term rewards, taking into account tracking efficiency, synchronization, and adaptability. The collaborator robotic arm reward function... as follows: （14） In formula (14), , and is a regularization coefficient, is a cooperation reward, and are the homogeneous transformation matrix between the end effector of the leader robot arm and the end effector of the collaborator robot arm in the training process and the homogeneous transformation matrix satisfying the closed-loop constraint, respectively. The improved CSAC algorithm combines the standard SAC algorithm with the homogeneous transformation matrix of the collaborator robot's end effector relative to the navigator robot's end effector during the process. Homogeneous transformation matrix satisfying closed-loop constraints Additional penalty terms are introduced to improve the coordinated control performance of the dual-arm system. First, the homogeneous transformation matrix of the collaborator's end effector relative to the leader's end effector during collaborative handling is calculated. , and then calculate Homogeneous transformation matrix satisfying closed-loop constraints Mean Squared Error (MSE) loss between: （15） In formula (15), denotes the number of elements in a matrix, denotes the index of an element in a matrix; The improved CASC algorithm considers a homogeneous transformation matrix of an end effector of a cooperator robot arm relative to an end effector of a leader robot arm in an updating formula of a Q value The improved CASC algorithm considers a homogeneous transformation matrix of an end effector of a cooperator robot arm relative to an end effector of a leader robot arm in an updating formula of a Q value The improved CASC algorithm considers a homogeneous transformation matrix of an end effector of a cooperator robot arm relative to an end effector of a leader robot arm in an updating formula of a Q value （16） In Equation (16), is the immediate reward, is the discount factor, is the weight controlling the entropy, is the importance of adjusting the end effector position error, , and are two action value networks in the Critic network, respectively. The update formula of the Critic network of the improved CASC algorithm is: （17） The update formula of the Actor network of the improved CASC algorithm is: （18） In formula (18), a set of environmental states, a state from the set of environmental states is taken the expectation of the target formula; Interacting with the environment Subsequently, if the end effectors of the leader robot arm and the co-worker robot arm both reach the tote, the mechanical grippers of the respective robot arms are controlled to lower the workpiece; if either end effector of the leader robot arm and the co-worker robot arm does not reach the tote within a prescribed number of time steps, or if the robot arms are out of sync for any time step, then step three is repeated until the end effectors of the leader robot arm and the co-worker robot arm both reach the tote.

3. The dual robot arm collaborative handling system based on deep reinforcement learning according to claim 1, characterized in that, In step one, the leader arm selects KUKA LBR iiwa 7 R800, the cooperator arm selects UniversalRobots UR5, and the mechanical gripper adopts Robotiq 140.

4. The dual robot arm collaborative handling system based on deep reinforcement learning according to claim 1, characterized in that, In step two, the feature extraction network can be ResNet.

5. The dual robot arm collaborative handling system based on deep reinforcement learning according to claim 1, characterized in that, In step two, the grasp point recognition depends on the analysis of the surface features of the object, including shape, texture, and color.

Images