Cooperative control method for double-arm robot

Abstract

The two-arm robot cooperative control method comprises the steps that an environment model is initialized; according to the double-arm robot cooperative control strategy, a Markov decision model is constructed, the Markov decision model comprises state space and action space definition, an environment interaction mechanism and a mixed constraint hierarchical near-end strategy optimization algorithm, and the execution process of the model is formally described as state observation, strategy reasoning, action execution, reward calculation and strategy updating. The grabbing task of the double-arm robot is converted into a learnable control strategy optimization problem; the dynamic mixing constraint comprises the following steps: carrying out weighted mixing on a near-end cutting target and a divergence KL penalty target; according to the layered time scale optimization, strategy optimization is divided into bottom-layer optimization and high-layer optimization based on a dynamic mixed reduction target; and training a double-arm robot cooperative control strategy, deploying the strategy to a mechanical arm, and carrying out a mechanical arm grabbing task. The method has the characteristics of double-arm collaborative distribution, difficulty in collision and good self-adaptability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a dual-arm robot cooperative control method. BACKGROUND

[0002] As one of the core technologies of robot control, dual-arm cooperation plays a crucial role in electronic component assembly, part manufacturing and other discrete manufacturing scenarios. The low efficiency of material sorting caused by tool dispersion has become a key factor restricting the improvement of production capacity. As a new solution, dual-arm cooperative control has shown great application potential due to its high coordination and flexibility. However, the dual-arm cooperative control still faces great challenges in the aspects of cooperative distribution, collision avoidance and high-dimensional continuous control complexity.

[0003] In the prior art, collision avoidance and appropriate control strategies are the focus of cooperative tasks in the face of complex dual-arm cooperative control tasks. Li et al. proposed a DRL algorithm with automatic entropy adjustment for the problem of redundant robot obstacle avoidance, combined deep reinforcement learning with traditional path planners, and integrated a general optimal motion planning framework. Jiang et al. used the proximal policy optimization algorithm (PPO) to train the model in a simulated environment, and then specified the generative adversarial imitation learning (GAIL) to transfer the model to the real world, so as to better complete complex assembly tasks on real robot arms. While there has been progress in complex assembly tasks and dynamic obstacle avoidance research, fine operation control of robot arms has also gradually emerged. Francisco et al. integrated vision, operation and learning technologies to control TIAGo++ robots through training of a deep reinforcement learning algorithm to achieve a dual-arm cooperative robot bottle opening task, facilitating the accessibility of daily needs of the injured or disabled. In addition, there are dynamic motion primitives (DMP) methods, dynamic neural obstacle avoidance (DNLOA) schemes, image-based visual servoing, and some progress in cooperative assembly research. However, the dual-arm cooperative control method based on deep reinforcement learning has an exponential increase in coordination difficulty as the dimension increases. Due to the diversity of complex high-dimensional space, deep reinforcement learning has difficulty in accurately identifying and understanding the mutual relationship and priority between tasks, high-dimensional motion paths are difficult to predict and plan, and dual-arms are prone to collision, which need to be solved. SUMMARY

[0004] The purpose of the present application is to overcome the above-mentioned shortcomings and provide a dual-arm cooperative distribution, collision-avoiding and self-adapting dual-arm robot cooperative control method.

[0005] A dual-arm robot cooperative control method, wherein the method comprises the following steps:

[0006] Step 1: initialize the environment model, including dual-arm robot parameters;

[0007] Step 2: dual-arm robot cooperative control strategy: build a Markov decision model, including state space and action space definition, environment interaction mechanism, mixed constraint hierarchical near-optimal policy optimization algorithm, the model execution process is formalized as state observation, policy inference, action execution, reward calculation, policy update, which converts the dual-arm robot grasping task into a learnable control policy optimization problem, specifically including:

[0008] Step 2.1 state space and action space definition, wherein the state space S t is defined as:

[0009]

[0010] wherein S position represents the pose state of the end effector of the robot arm, S task represents the task state, i.e. the current task execution sequence, S gripper represents the gripper state of the end effector of the robot arm, S env represents the environment image, L represents the left arm of the dual-arm robot, and R represents the right arm of the dual-arm robot;

[0011] The action space A t is defined as:

[0012]

[0013] wherein A position represents the execution action sequence of the end effector of the robot arm, A task is the task action sequence, and A gripper is the gripper action sequence of the end effector of the robot arm;

[0014] Step 2.2 environment interaction mechanism: provide current state information, including object position, dual-arm pose, obstacle information, task completion progress, return new state s t and reward r t+1 according to the execution of action a t , guiding the dual-arm robot to optimize behavior;

[0015] Step 2.3: The mixed constraint layered near-end policy optimization algorithm: based on the improvement of deep reinforcement learning near-end policy optimization, through dynamic mixed constraint, layered time scale optimization, adaptive parameter adjustment mechanism, while maintaining the efficiency of near-end policy optimization PPO, introduce trust region policy optimization TRPO, increase the stability and policy exploration ability of the optimization process, form a hierarchical optimization system, realize the stability and efficiency balance of policy optimization in complex tasks;

[0016] The dynamic mixed constraint: the near-end clipping target is weighted and mixed with the divergence KL penalty target;

[0017] The layered time scale optimization: based on the dynamic mixed constraint target, the policy optimization is divided into bottom layer optimization and high layer optimization, the bottom layer optimization is a fast update layer, which uses gradient descent to minimize the total loss, and is used to improve the policy quickly in each update cycle; the high layer optimization is a low frequency update layer with trust region constraint, which performs constraint update in the form of TRPO, to keep the overall policy update within the trust region boundary, so as to maintain the training stability;

[0018] Step 2.4: design reward function: design reward from safety, task completion degree, time efficiency elements, total reward is equal to the sum of collision reward, distance reward and time reward;

[0019] Step 3: train the dual-arm robot cooperative control strategy, deploy it to the robot arm, and perform the robot arm grasping task.

[0020] The above-mentioned dual-arm robot cooperative control method, wherein: in step 2, the Markov decision model is represented as a five-tuple M=(S,A,p,r,γ), wherein S represents a state set in the environment; A represents a set of actions in the environment; p represents the probability of an event: in the current state s, the robot arm performs action a, and the state of the environment becomes s', represented as r(s,a,s'), r is a reward function; γ is the discount rate.

[0021] The above-mentioned dual-arm robot cooperative control method, wherein: in step 2.2, the environment interaction mechanism provides current state information, including object position, dual-arm posture, obstacle information, task completion progress, and returns new state s t and reward r t+1 according to the execution of action a t to guide the dual-arm robot to optimize behavior, the specific steps are:

[0022] a) initialize the environment: including setting the initial state of the dual-arm, the position of the grasping task workpiece, and the target position;

[0023] b) get the current state s tThis includes the angles of the two arm joints and the condition of the workpiece.

[0024] c) Decision-making optimal action a t The dual-arm robot determines the optimal action based on the input state information using a hybrid constraint hierarchical proximal strategy optimization algorithm.

[0025] d) Perform actions and update the environment: The dual-arm robot performs actions in the environment, and the environmental state is updated accordingly;

[0026] e) Feedback on new status and rewards: The environment returns status s t+1 and reward r t ;

[0027] f) Update strategy: Optimize actions based on new status and reward values ​​to improve the decision-making ability of the dual-arm robot;

[0028] g) Looping interaction: Repeat steps b)-f) above until the grabbing task is completed or the training steps reach the maximum.

[0029] The above-mentioned collaborative control method for a dual-arm robot includes the following step: In step 2.3, the hybrid constraint hierarchical proximal policy optimization algorithm is an improvement on deep reinforcement learning-based proximal policy optimization. Through dynamic hybrid constraints, hierarchical time-scale optimization, and adaptive parameter adjustment mechanisms, it introduces trust domain policy optimization (TRPO) while maintaining the high efficiency of proximal policy optimization (PPO), increasing the stability and policy exploration capability of the optimization process, forming a hierarchical optimization system, and achieving a balance between stability and efficiency in policy optimization for complex tasks. The specific steps are as follows:

[0030] Step 2.3.1 Define the policy: The policy is defined by a probability distribution controlled by parameter θ. The long-term expected return of the policy in the current state is obtained through the state value function and the action value function. The state value function V... π (s t )for:

[0031]

[0032] The action value function Q π (s t ,a t )for:

[0033]

[0034] Among them, E π To achieve the desired result, r t+l γ is the reward obtained at step l after starting from time step t, and s is the discount rate. t express The state at time a t Indicates the action at time t;

[0035] Step 2.3.2 Advantage Function Estimation: Based on the aforementioned action value function and state value function, calculate the advantage function, which measures the relative merits of actions and guides the policy gradient direction. The advantage function A t for:

[0036]

[0037] Step 2.3.3 Policy Update Ratio: To compare the old and new policies during optimization, a probability ratio r is introduced. t (θ):

[0038] Where, π θ (s t， a t ) indicates that the new policy is in state s t Choose action a t The probability, π θold (s t， a t ) indicates that the old policy is in state s t Choose action a t The probability of this ratio is limited to prevent excessive policy update bias from causing instability;

[0039] Step 2.3.4 Dynamic Hybrid Constraints: The near-end pruning objective and the divergence KL penalty objective are weighted and mixed to construct a hybrid optimization objective function L. mix (θ) is:

[0040]

[0041] Where α is the mixing coefficient, used to dynamically balance the weights of the two; L clip (θ) represents the PPO pruning objective of the near-end strategy optimization, L KL (θ) represents the KL penalty target, expressed as:

[0042]

[0043]

[0044] Where E is the expectation, clipr t (*) represents the clipping range of probability ratios. To control the clipping range, β is the divergence KL penalty factor, and D... KL For divergence, For relative operators;

[0045] Step 2.3.5 Hierarchical time scale optimization: based on the dynamic mixing objective, the policy optimization is divided into bottom layer optimization and high layer optimization, the bottom layer optimization is a fast update layer, which uses gradient descent to minimize the total loss, and is used to quickly improve the policy in each update cycle; the high layer optimization is a low frequency update layer with a trust region constraint, which performs constraint update in the form of TRPO to keep the overall policy update within the trust region boundary, so as to maintain the stability of training;

[0046] The bottom layer optimization is represented as:

[0047]

[0048] Wherein, L total (θ) is the total loss function of the bottom layer optimization, which quickly improves the policy in each update cycle in the form of gradient descent; L(θ) is the total loss function; c1 is the value function survival coefficient; V(s t ) is the state value; R t is the discounted reward; c2 is the entropy regularization coefficient; H(π θ ) is the entropy of the policy distribution under the probability π θ .

[0049] The high layer optimization is:

[0050]

[0051] Wherein, Maximize the parameter θ, and δ is the constraint threshold;

[0052] Step 2.3.6 Adaptive parameter adjustment mechanism: adjust the mixing coefficient α and the divergence KL penalty factor β in step 2.3.4 by monitoring the divergence; if the divergence D KL exceeds 1.5 times the target threshold D target , then α is reduced, and △α is the change of α in each adjustment, which is biased towards TRPO stable exploration; if the divergence D KL is too small, increase α, which is biased towards PPO efficient exploration; so that the strategy can realize adaptive trade-off between update speed and stability; the mixing coefficient α adjustment mechanism is:

[0053] .

[0054] The above-mentioned dual-arm robot cooperative control method, wherein the mixing coefficient α adjustment mechanism is:

[0055]

[0056] In order to further improve the constraint adaptability and reduce the demand for manual parameter adjustment, after each round of policy update, the divergence D KLThe degree of deviation from the target threshold value automatically adjusts β, ensuring the stability of the strategy update; first, calculate the real-time divergence The ratio of the target threshold D target :

[0057]

[0058] Then, according to the ratio, adjust the penalty coefficient at the tth iteration β t , combined with the adjustment factor η, and then perform truncation processing on the adjusted β t+1 , so that it does not exceed the limit range; the β adjustment mechanism is:

[0059]

[0060]

[0061] The above-mentioned dual-arm robot cooperative control method, wherein: in step 2.4, the reward function is designed: from safety, task completion degree, time efficiency factor, the total reward is equal to the sum of collision reward, distance reward and time reward; the designed reward function is the single-step total reward R of reinforcement learning in the process of robot arm grasping, which is expressed as:

[0062]

[0063] Where, R col represents the collision reward, R dis represents the distance reward, and R t represents the time reward.

[0064] The above-mentioned dual-arm robot cooperative control method, wherein: the designed reward function is the single-step total reward R of reinforcement learning in the process of robot arm grasping, which is expressed as:

[0065]

[0066] Where, R col represents the collision reward, R dis represents the distance reward, and R t represents the time reward.

[0067] The collision reward R col is expressed as:

[0068]

[0069] Where, r col is the collision reward setting value, and α is the collision factor, α <0.

[0070] The distance reward ​is represented as:

[0071]

[0072] wherein r dis is a distance reward setting value;

[0073] The time reward R t is represented as:

[0074]

[0075] wherein t is a training time.

[0076] Compared with the prior art, the present application has obvious beneficial effects. As can be seen from the above scheme, the present application is based on deep reinforcement learning, defines a dual-arm state space and an action space, constructs a closed-loop interaction mechanism of perception-decision-update, proposes a Hybrid Constraint Hierarchical Proximal Policy Optimization (HCH-PPO) control algorithm, and designs a reward function. The HCH-PPO introduces a dual-time-scale updating mechanism and a hierarchical reinforcement learning (HRL) idea to improve the traditional Proximal Policy Optimization (PPO), designs a hybrid objective function, and realizes multi-policy adaptive constraint balance by dynamic adjustment planning, effectively solving the single constraint limitation and the multi-stage collaborative task problem of complex high-dimensional discrete action space. In summary, the present application has the following advantages:

[0077] 1. Design a dual-time-scale hierarchical strategy. In view of the high dimensionality and multi-stage decision complexity of the dual-arm collaborative task, the hierarchical reinforcement learning (HRL) idea is introduced, a high-low layer strategy is designed, the complex task is decomposed into multiple levels of subtasks, and the problems of difficult convergence and sparse reward of direct learning of complex tasks are solved. Among them, the high-level decision adopts Trust Region Policy Optimization (TRPO) for policy update to ensure the stability of the long-term goal; the bottom strategy is aimed at specific action execution, adopts Proximal Policy Optimization (PPO) for policy update to realize fast local optimization, and separates the time scale, adopts a dual-time-scale updating mechanism to balance the contribution of high-low layer strategies, reduces the strategy conflict, and improves the strategy stability.

[0078] 2. Establish an adaptive hybrid constraint mechanism. In view of the limitations of the traditional PPO single constraint mechanism and the slow convergence speed of the pure KL constraint, the KL divergence penalty is dynamically combined with the PPO Clip constraint, a dynamic weight coefficient a is introduced, a hybrid objective function is designed, the constraint dominant mode is dynamically adjusted according to the policy change amplitude, and the constraint efficiency is maintained. At the same time, the KL penalty coefficient β is introduced, and the KL penalty is adaptively adjusted to solve the problem of mismatching of the punishment strength. The dual coefficients ensure the flexibility of the constraint mode selection and the adaptability of the punishment strength.

[0079] 3. A hybrid constraint hierarchical proximal policy optimization (HCH-PPO) algorithm is proposed. In view of the policy deviation of the traditional PPO single-layer optimization architecture and the limitations of the constraint mechanism, the above dual-time-scale hierarchical strategy and adaptive hybrid constraint mechanism are fused, and a hybrid constraint hierarchical proximal policy optimization (HCH-PPO) algorithm is proposed. The task model is built, the state space and action space are defined, and the reward mechanism is set. Through multi-mechanism collaborative optimization, the efficiency and stability of the proposed algorithm are verified.

[0080] The beneficial effects of the present application are further illustrated by the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0081] Figure 1 The flow structure diagram of the present application. DETAILED DESCRIPTION

[0082] The specific embodiments, features and effects of the dual-arm robot cooperative control method according to the present application are described in detail as follows in combination with the preferred embodiments and the accompanying drawings.

[0083] Referring to Figure 1 , the dual-arm robot cooperative control method comprises the following steps:

[0084] Step 1: initialize the environment model, including the parameters of the dual-arm robot;

[0085] Step 2: dual-arm robot cooperative control strategy: build a Markov decision model, including state space and action space definition, environment interaction mechanism, hybrid constraint hierarchical proximal policy optimization algorithm, the model execution process is formalized as state observation, policy reasoning, action execution, reward calculation, policy update, and the dual-arm robot grasping task is converted into a learnable control strategy optimization problem, which specifically includes:

[0086] Step 2.1, the state space and action space definition, wherein the state space S t is defined as:

[0087]

[0088] wherein, Sposition represents the pose state of the end effector of the robot arm, S task represents the task state, i.e., the current task execution sequence, S gripper represents the gripper state of the end effector of the robot arm, including the open, closed or holding state, S env represents the environment image, L represents the left arm of the dual-arm robot, and R represents the right arm of the dual-arm robot.

[0089] The action space A t is defined as:

[0090]

[0091] wherein A position represents the execution action sequence of the end effector of the robot arm, A task is the task action sequence, A gripper is the gripper action sequence of the end effector of the robot arm, performing the open, close or hold action.

[0092] The Markov decision model includes state space, action space, reward function, state transition function, discount rate, represented as a five-tuple M=(S,A,p,r,γ), wherein S represents the state set in the environment; A represents the action set in the environment; p: p(s'|s,a), p represents the probability of an event: in the current state s, the robot arm performs the action a, and the state of the environment becomes s', represented as r(s,a,s'), r is the reward function; and γ is the discount rate.

[0093] Step 2.2 The environment interaction mechanism provides current state information, including object position, dual-arm pose, obstacle information, task completion progress, and returns new state s t and reward r t+1 according to the execution of action a t , guiding the dual-arm robot to optimize behavior, and the specific steps are as follows:

[0094] a) Initialize the environment: including setting the initial state of the dual-arm, the position of the workpiece for grasping, and the target position;

[0095] b) Obtain the current state s t : including the joint angle of the dual-arm, the state of the workpiece, and whether there is a collision;

[0096] c) Decide the optimal action a t : the dual-arm robot decides the optimal action according to the input state information through the mixed constraint hierarchical proximal policy optimization algorithm;

[0097] d) Execute the action and update the environment: the dual-arm robot executes the action in the environment, and the environment state is updated accordingly;

[0098] e) Feedback of new state and reward: the environment returns state s t+1 and reward r t ;

[0099] f) Update policy: optimize action according to new state and reward value, improve dual-arm robot decision-making ability;

[0100] g) Loop interaction: repeat steps b) -f) above until the grasping task is completed or the maximum number of training steps is reached.

[0101] Step 2.3: The mixed constraint hierarchical proximal policy optimization algorithm: based on the improvement of deep reinforcement learning proximal policy optimization, through dynamic mixing constraints, hierarchical time scale optimization, and adaptive parameter adjustment mechanism, while maintaining the efficiency of proximal policy optimization PPO, it introduces trust region policy optimization TRPO to increase the stability and policy exploration ability of the optimization process, forms a hierarchical optimization system, and realizes the stability and efficiency balance of policy optimization in complex tasks. The specific implementation steps are as follows:

[0102] Step 2.3.1 Define policy: the policy is defined by the distribution probability controlled by parameter θ, and the long-term expected return of the policy in the current state is obtained through the state value function and the action value function. The state value function V π (s t ) is:

[0103]

[0104] The action value function Q π (s t ,a t ) is:

[0105]

[0106] Where E π is the expectation, r t+l is the reward obtained from the lth step after time step t, γ is the discount rate, s t represents the state at time t, and a t represents the action at time t.

[0107] Step 2.3.2 Advantage function estimation: calculate the advantage function based on the action value function and the state value function, which is used to measure the relative merits of actions and guide the policy gradient direction. The advantage function A t is:

[0108]

[0109] ​Step 2.3.3 Policy update ratio: To compare the new and old policies when optimizing, introduce the probability ratio r t (θ) is:

[0110] where π θ (s t， a t ) represents the probability of the new policy selecting action a t in state s t , π θold (s t， a t ) represents the probability of the old policy selecting action a t in state s t , and the ratio is limited to prevent policy update from being too large and causing instability;

[0111] Step 2.3.4 Dynamic mixing constraint: Mix the proximal clipping target and the KL divergence penalty target by weighting, construct a mixed optimization target function L mix (θ) is:

[0112]

[0113] where α is the mixing coefficient, used to dynamically balance the weights of the two; L clip (θ) represents the proximal policy optimization PPO clipping target, L KL (θ) represents the KL penalty target, which is expressed as:

[0114]

[0115]

[0116] where E is the expectation, clipr t (*) is the clipping range of the probability ratio, is the hyperparameter that controls the clipping range, β is the KL divergence penalty factor, D KL is the divergence, is the relative operator;

[0117] Step 2.3.5 Hierarchical time scale optimization: Based on the dynamic mixing target, the policy optimization is divided into bottom layer optimization and high layer optimization. The bottom layer optimization is a fast update layer, which uses gradient descent to minimize the total loss, and is used to quickly improve the policy within each update cycle. The high layer optimization is a low-frequency update layer with trust region constraint, which performs constrained update in the form of TRPO to keep the overall policy update within the trust region boundary, thereby maintaining training stability;

[0118] The bottom layer optimization is represented as:

[0119]

[0120] wherein, L total (θ) is the total loss function of the bottom layer optimization, which quickly improves the strategy in each update cycle in the way of gradient descent; L(θ) is …; c1 is the survival coefficient of the value function; V(s t ) is the state value; R t is the discounted reward; c2 is the entropy regularization coefficient; H(π θ ) is the entropy of the policy distribution under the probability π θ .

[0121] The high-level optimization is:

[0122]

[0123] wherein, maximization is performed on the parameter θ, and δ is a constraint threshold;

[0124] Step 2.3.6 Adaptive parameter adjustment mechanism: the mixing coefficient α and the divergence KL penalty factor β in step 2.3.4 are adjusted by monitoring the divergence; if the divergence D KL exceeds 1.5 times the target threshold D target , then α is reduced, and △α is the change amount of α each time, at this time, the strategy is biased towards TRPO stable exploration; if the divergence D KL is too small, then α is increased, and the strategy is biased towards PPO efficient exploration; thus, the strategy can achieve adaptive trade-off between update speed and stability; the mixing coefficient α adjustment mechanism is:

[0125]

[0126] In order to further improve the constraint adaptability and reduce the demand for manual parameter adjustment, β is automatically adjusted according to the deviation of the divergence D KL from the target threshold after each round of strategy update, so as to ensure the stability of the strategy update; first, the ratio of the real-time divergence D target to the target threshold D is calculated:

[0127]

[0128] Then, the penalty coefficient β t of the tthiteration is adjusted according to the ratio combined with the adjustment factor η, and then the adjusted β t+1 is truncated to avoid gradient explosion or disappearance, so as to avoid the threshold-based exponential adjustment mechanism only needs to compare the KL ratio with the threshold, without complex operation, and has parameter robustness, and the algorithm can balance the strategy exploration and stability for complex grasping tasks; the β adjustment mechanism is:

[0129]

[0130]

[0131] Step 2.4: Designing the reward function: from safety, task completion, time efficiency factors, design the reward, the total reward is equal to the sum of collision reward, distance reward and time reward, considering the safety of the agent, the speed and quality of task completion, rich feedback information, and then guide the agent to the expected target; the design reward function is the total reward R of single step reinforcement learning in the process of robot arm grasping, which is represented as:

[0132]

[0133] Wherein, R col represents the collision reward, R dis represents the distance reward, and R t represents the time reward.

[0134] The collision reward R col is represented as:

[0135]

[0136] Wherein, r col is the collision reward setting value, and a is the collision factor, a <0; if a is too large, the punishment of small agent will collide frequently; if a is too small, the punishment will be too large, which will lead to excessive conservation and reduce efficiency;

[0137] The distance reward R dis is represented as:

[0138]

[0139] Wherein, r dis is the distance reward setting value;

[0140] The time reward R t is represented as:

[0141]

[0142] Wherein, t is the training time;

[0143] Step 3: training the dual-arm robot cooperative control strategy, deploying to the robot arm, and performing the robot arm grasping task.

[0144] A dual-arm robot cooperative system, which comprises hardware facilities and software parts, the hardware facilities include left and right robot arms and a depth camera, and the software parts include a visual detection module and a robot arm control algorithm module.

[0145] The visual detection module: first captures the image RGB and depth information of the random object on the workbench by using the depth camera, identifies and determines the position of the object, places the target position, and assigns the grabbing task to the left and right mechanical arms for the grabbing task; then, the position and speed data of the object are transmitted to the left and right mechanical arm control algorithm module, the left and right mechanical arm control algorithm module outputs the accurate moving speed of each joint of the left and right mechanical arm, guides the end effector of the mechanical arm to move towards the target object, when the end effector approaches to the preset distance of the moving object, the gripper of the end effector closes, and the grabbing action is completed; after the grabbing is successful, the end effector carrying the grabbed object moves to the placement position gradually, and performs the alignment operation, after the alignment, the mechanical arm carries the object to place downward, when the distance between the object position and the placement target position is less than a threshold, the gripper is opened, and the object placement is completed;

[0146] The mechanical arm control algorithm module: controls the joint movement of the left and right mechanical arms and the grabbing operation of the gripper; first, the state of the moving object and the gripper is obtained, the speed of the gripper is calculated, and the left and right mechanical arms are controlled to approach the target; this module controls the left and right mechanical arms to grab the target object and move to the placement position to release the gripper to place the object; in addition, the control module continuously interacts with the environment to obtain the action with the highest reward in the current state, so as to realize the tracking, grabbing and placing of the target object by the left and right mechanical arms.

[0147] Performance analysis:

[0148] (1) Experimental setup

[0149] The specific task of the dual-arm cooperation experiment is to transfer the scattered workpieces randomly distributed on the workbench to the designated storage box within a limited time without collision. The dual-arm cooperation task involves visual perception, dynamic cooperation control strategy and three-dimensional modeling. For the requirement of high-dimensional discrete action multi-task, a simulation task scene is built by using three-dimensional modeling software Solidworks combined with CoppeliaSim simulation software. The visual sensing camera is placed above the mechanical arm for visual perception. The dual UR3 mechanical arms are equipped with two-finger grippers fixed on the base. Eight workpiece and tool box models are imported to establish the simulation experiment scene. The agent assigns the left arm task or the right arm task according to the object category and coordinates detected by the visual sensor and gives the task sequence decision, then guides the end effector to move above the to-be-picked object to perform the grabbing operation, moves to the picking position above after the grabbing is successful, aligns and then moves down to the specified position and releases the gripper, completes the classified placement of the object, and then the mechanical arm returns to the preparation position to wait for the next task until all the workpieces are picked up. The dual mechanical arms coordinate and distribute the workpieces according to the learned task strategy to place them at the target position, ensuring that no collision and misplacement occur during the whole process until all the sub-tasks are successfully placed.

[0150] (2) Result analysis

[0151] Under the condition of 5000 total training steps, A2C and RePPO did not learn the correct strategy effectively, so the time consumption and final success rate of the other three algorithms were analyzed, and the data in Table 1 was obtained. It can be seen that the first three algorithms can effectively complete 50 simulation experiments, and the time consumption of HCH-PPO is the shortest, which is 1 hour 28 minutes and 59 seconds, and the task success rate is 0.98, which is higher than that of PPO and TRPO. The success rate of A2C and RePPO is the lowest in 50 experiments, and the time consumption is the longest, which is due to the fact that the training of the first 5000 steps does not learn an effective strategy. In general, HCH-PPO performs better than other algorithms in all aspects, and has the best performance.

[0152] Table 1 Comparison of task success rate and time consumption

[0153] Algorithm name Task number Success number Success rate Total time consumption (h) HCH-PPO 50 49 0.98 1:28:59.17 PPO 50 43 0.86 1:32:59.89 TRPOA2C 5050 452 0.900.04 1:37:49.77 1:50:23.08 RePPO 50 3 0.06 1:45:30.51

[0154] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification, equivalent change and modification of the above embodiment without departing from the technical solution of the present application, according to the technical essence of the present application, are still within the scope of the technical solution of the present application.

Claims

1. A dual-arm robot cooperative control method, characterized by: The method comprises the following steps: Step 1: initialize the environment model, including dual-arm robot parameters; Step 2: dual-arm robot collaborative control strategy: build a Markov decision model, including state space and action space definition, environment interaction mechanism, hybrid constraint hierarchical proximal policy optimization algorithm, the model execution process is formalized as state observation, policy inference, action execution, reward calculation, policy update, which converts the dual-arm robot grasping task into a learnable control policy optimization problem, specifically including: Step 2.1 the state space and action space definitions, wherein the state space S t is defined as: ； wherein S position represents the pose state of the end effector of the robot arm, S task represents the task state, i.e. the current task execution sequence, S gripper represents the gripper state of the end effector of the robot arm, S env represents the environment image, L represents the left arm of the dual-arm robot, and R represents the right arm of the dual-arm robot; The action space A t is defined as: ； wherein A position represents a sequence of execution actions of the end effector of the robot arm, A task is a sequence of task actions, A gripper is a sequence of gripper actions of the end effector of the robot arm; Step 2.2 The environmental interaction mechanism: provide current state information, including object position, dual-arm posture, obstacle information, task completion progress, return new state s t and reward r t+1 according to the execution of action a t , guide the dual-arm robot to optimize behavior; Step 2.3 said hybrid constraint hierarchical proximal policy optimization algorithm: based on the improvement of proximal policy optimization of deep reinforcement learning, through dynamic hybrid constraint, hierarchical time scale optimization, adaptive parameter adjustment mechanism, while maintaining the efficiency of proximal policy optimization PPO, introduce trust region policy optimization TRPO, increase the stability and policy exploration ability of the optimization process, form a hierarchical optimization system, realize the stability and efficiency balance of policy optimization in complex tasks; Said dynamic hybrid constraint: the proximal clipping target and the divergence KL penalty target are weighted and mixed; Said hierarchical time scale optimization: based on dynamic hybrid optimization target, the policy optimization is divided into bottom layer optimization and high layer optimization, the bottom layer optimization is a fast update layer, which uses gradient descent to minimize the total loss, for improving the policy in each update cycle; the high layer optimization is a low frequency update layer with trust region constraint, which performs constraint update in the form of TRPO, to keep the overall policy update within the trust region boundary, so as to maintain the training stability; Step 2.4: design the reward function: design the reward from safety, task completion degree, time efficiency elements, the total reward is equal to the sum of collision reward, distance reward and time reward; Step 3: train the dual-arm robot collaborative control strategy, deploy it to the robot arm, and perform the robot arm grasping task.

2. The dual-arm robot cooperative control method of claim 1, wherein: In step 2, the Markov decision model is represented as a five-tuple M=(S,A,p,r,γ), where S represents the state set in the environment; A represents the action set in the environment; p:p(s'|s,a), p represents the probability of the event: in the current state s, the robot arm performs action a, and the state of the environment becomes s', which is represented as r(s,a,s'), r is the reward function; γ is the discount rate.

3. The dual-arm robot collaborative control method of claim 1, wherein: In step 2.2, the environment interaction mechanism provides the current state information, including object position, dual-arm pose, obstacle information, task completion progress, returns new state s t and reward r t+1 according to the execution of action a t , guiding the dual-arm robot to optimize behavior, the specific steps are: a) initialize the environment: including setting the initial state of the dual-arm, the position of the workpiece in the grasping task, and the target position; b) obtaining a current state s t : including double-arm joint angle, workpiece state; c) decision optimal action a t : the dual-arm robot decides the optimal action according to the input state information through a hybrid constraint hierarchical proximal policy optimization algorithm; d) execute the action and update the environment: the dual-arm robot executes the action in the environment, and the environment state is updated accordingly; e) Feedback of new state and reward: the environment returns state s t+1 and reward r t ; f) update the policy: according to the new state and reward value, optimize the action and improve the decision-making ability of the dual-arm robot; g) cycle interaction: repeat the above steps b) - f) until the grasping task is completed or the maximum number of training steps is reached.

4. The dual-arm robot collaborative control method of claim 1, wherein: In step 2.3, the mixed constraint layered near-end policy optimization algorithm: based on deep reinforcement learning near-end policy optimization is improved, by dynamically mixing constraints, layered time scale optimization, adaptive parameter adjustment mechanism, while maintaining the efficiency of near-end policy optimization PPO, introduces trust region policy optimization TRPO, increases the stability and policy exploration ability of the optimization process, forms a hierarchical optimization system, realizes the stability and efficiency balance of policy optimization in complex tasks, the specific steps are: Step 2.3.1 Define the policy: The policy is defined by a probability distribution controlled by parameter θ. The long-term expected return of the policy in the current state is obtained through the state value function and the action value function. The state value function V... π (s t )for: ； The action value function Q π (s t ,a t ) is: ； where E π is the expected reward, r t+l is the reward obtained at step / from time step t, and γ is the discount rate. t s denotes the state at time t, and a t denotes the action at time t. Step 2.3.2 Advantage function estimation: Compute an advantage function A(s, a) based on the described action-value function and state-value function, which measures the relative merits of actions and guides the direction of policy gradient, the advantage function A(s, a) is defined as: t A(s, a) = Q(s, a) - V(s) ； Step 2.3.3 Policy update ratio: To compare the new and old policies at the time of optimization, introduce the probability ratio r t (θ): ； where π θ (s t， a t ) denotes the probability of the new policy selecting action a t in state s t , and π θold (s t， a t ) denotes the probability of the old policy selecting action a t in state s t , and the ratio is limited to prevent instability caused by too large policy update bias. Step 2.3.4 Dynamic mixing constraint: The proximal clipping objective is mixed with the divergence KL penalty objective by weighting, and a mixed optimization objective function L is constructed mix (θ) is: ； wherein a is a mixing coefficient for dynamically balancing the weights of the two; L clip (θ) denotes a proximal policy optimization (PPO) clipped target, L KL (θ) denotes a KL penalty target, denoted as: ； ； where E is the expectation, clipr t (*) is the clipping range for the probability ratio, is a hyperparameter that controls the clipping range, β is the divergence KL penalty factor, D KL is the divergence, is the relative operator; Step 2.3.5 Layered time scale optimization: based on dynamic mixed goal, the policy optimization is divided into bottom layer optimization and high layer optimization, the bottom layer optimization is a fast update layer, which uses gradient descent to minimize the total loss, used to improve the policy in each update cycle; The high layer optimization is a low frequency update layer with trust region constraint, which is executed in the form of TRPO to update the constraint, so as to keep the overall policy update within the boundary of trust region, thus keeping the training stability; The bottom layer optimization is represented as: ； wherein L total (θ) is the total loss function of the base layer optimization, which quickly improves the policy in each update cycle in the way of gradient descent; L(θ) is the total loss function; c1 is the survival coefficient of the value function; V(s t ) is the state value; R t is the discounted reward; c2 is the entropy regularization coefficient; H(π θ ) is the entropy of the policy distribution under the probability π θ . The high layer optimization is: ； wherein denotes maximizing the parameter θ, and δ is a constraint threshold. Step 2.3.6 Adaptive parameter adjustment mechanism: adjust the mixing coefficient a and the divergence KL penalty factor b in step 2.3.4 by monitoring the divergence D KL If D exceeds the target threshold D by 1.5 times, a is reduced, and a is the change of a each time the adjustment is made. At this time, the strategy is biased towards TRPO stable exploration. If the divergence D is too small, a is increased, and the strategy is biased towards PPO efficient exploration. Thus, the strategy can achieve adaptive trade-off between update speed and stability. The mixing coefficient a adjustment mechanism is: target KL ​​ 。 5. The dual-arm robot collaborative control method of claim 4, wherein: In step 2.3.6, the mixed coefficient α adjustment mechanism is: ； To further enhance the constraint adaptability and reduce the demand for manual parameter adjustment, the divergence D KL is calculated after each round of strategy update, and the deviation of the divergence from the target threshold is used to automatically adjust β, ensuring the stability of the strategy update. First, the real-time divergence D is calculated, and then the ratio of the divergence to the target threshold D target is calculated : ； Then, based on this ratio and the adjustment factor η, the penalty coefficient β at the t-th iteration is adjusted. t Then adjust β t+1 The truncation process is performed to ensure it does not exceed the limit; the β adjustment mechanism is as follows: 。 6. The dual-arm robot collaborative control method of claim 1, wherein: In step 2.4, the design of reward function: from safety, task completion degree, time efficiency elements to design reward, the total reward is equal to the sum of collision reward, distance reward and time reward; The design of reward function is to set the single step total reward R of reinforcement learning in the process of robot arm grasping, which is represented as: ； where R col represents a collision reward, R dis represents a distance reward, R t represents a time reward.

7. The dual-arm robot collaborative control method of claim 6, wherein: The design of reward function is to set the single step total reward R of reinforcement learning in the process of robot arm grasping, which is represented as: ； where R col represents a collision reward, R dis represents a distance reward, R t represents a time reward; The collision reward R col is represented as: ； wherein, r col a collision reward setting value, a is a collision factor, a <0; The distance reward is represented as: ； wherein r dis is a distance reward set value; The temporal reward R t is represented as: ； Where t is the training time.

Images