A dual-arm robot cooperative control method

Abstract

The application discloses a dual-arm robot cooperative control method, which comprises the following steps: initializing an environment model; a dual-arm robot cooperative control strategy: constructing a Markov decision model, including state space and action space definition, environment interaction mechanism, mixed constraint hierarchical proximal policy optimization algorithm, the model execution process is formalized as state observation, policy reasoning, action execution, reward calculation, policy update, and the dual-arm robot grasping task is converted into a learnable control policy optimization problem; the dynamic mixed constraint: the proximal cutting target is mixed with the divergence KL penalty target by weighting; the hierarchical time scale optimization: based on the dynamic mixed optimization target, the policy optimization is divided into bottom optimization and high-level optimization; training the dual-arm robot cooperative control strategy, deploying to the mechanical arm, and performing the mechanical arm grasping task. The application has the characteristics of dual-arm cooperative distribution, no collision and good adaptability.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more specifically to a collaborative control method for a dual-arm robot. Background Technology

[0002] Dual-arm collaboration, as one of the core technologies of robot control, plays a crucial role in discrete manufacturing scenarios such as electronic component assembly and parts manufacturing. The low material sorting efficiency caused by tool dispersion has become a key factor restricting capacity improvement. Dual-arm collaborative control, as an emerging solution, shows great application potential due to its high degree of coordination and flexibility. However, facing the complex tasks of collaborative allocation, collision avoidance, and high-dimensional continuous control of dual robotic arms, dual-arm collaborative control still faces significant challenges. In recent years, the rapid development of deep reinforcement learning (DRL) has provided a new approach to solving this technical problem.

[0003] In existing technologies, avoiding collisions and employing appropriate control strategies are key to complex dual-arm collaborative control tasks. Li et al. proposed a DRL algorithm with automatic entropy adjustment and pruning to address the obstacle avoidance difficulties of redundant robotic arms, integrating deep reinforcement learning with traditional path planners to create a general optimal motion planning framework. Jiang et al. addressed the difficulty of acquiring large amounts of real robot motion data by training a model in a simulated environment using the Proximal Policy Optimization (PPO) algorithm, then transferring the model to the real world using Generative Adversarial Imitation Learning (GAIL), enabling better completion of complex assembly tasks on real robotic arms. While progress has been made in complex assembly tasks and dynamic obstacle avoidance, refined manipulation control of robotic arms is also emerging. Francisco et al. integrated vision, manipulation, and learning technologies, using a deep reinforcement learning algorithm to control the TIAGo++ robot to perform a dual-arm collaborative robot bottle-opening task, improving accessibility for people with injuries or disabilities. Furthermore, collaborative control of dual-arm robots also includes Dynamic Motion Primitives (DMP) methods, Dynamic Neural Learning for Obstacle Avoidance (DNLOA) schemes, and image-based visual servoing, with some progress also made in collaborative assembly research. However, the difficulty of collaborative control of two arms based on deep reinforcement learning increases exponentially with the increase of dimensionality. Due to the diversity of complex high-dimensional space, deep reinforcement learning has difficulty in accurately identifying and understanding the interrelationships and priorities between tasks, the high-dimensional motion path is difficult to predict and plan, and the two arms are prone to collisions. These problems need to be solved. Summary of the Invention

[0004] The purpose of this invention is to overcome the above-mentioned shortcomings and propose a collaborative control method for dual-arm robots that is less prone to collisions and has good adaptability.

[0005] The present invention provides a cooperative control method for a dual-arm robot, wherein the method includes the following steps:

[0006] Step 1: Initialize the environment model, including the parameters of the dual-arm robot;

[0007] Step 2: Cooperative Control Strategy for Dual-Arm Robots: Construct a Markov decision model, including the definition of state and action spaces, environmental interaction mechanisms, and a hybrid constraint hierarchical proximal policy optimization algorithm. The execution process of this model is formally described as state observation, policy reasoning, action execution, reward calculation, and policy update. This transforms the dual-arm robot grasping task into a learnable control policy optimization problem, specifically including:

[0008] Step 2.1 defines the state space and action space, wherein the state space S t Defined as:

[0009]

[0010] Among them, S position S represents the pose state of the robotic arm's end effector. task S represents the task status, i.e., the current task execution sequence. gripper S indicates the gripper state of the robotic arm's end effector. env The image represents the environment; L represents the left arm of the dual-arm robot, and R represents the right arm of the dual-arm robot.

[0011] The action space A t Defined as:

[0012]

[0013] Among them, A position A represents the sequence of actions performed by the end effector of the robotic arm. task For the task action sequence, A gripper This refers to the gripper motion sequence of the robotic arm's end effector.

[0014] Step 2.2 The environmental interaction mechanism provides current state information, including object position, arm posture, obstacle information, and task completion progress, based on action a. t The execution status returns a new status s t+1 and reward r t To guide the dual-arm robot in optimizing its behavior;

[0015] Step 2.3 Hybrid Constraint Hierarchical Proximal Policy Optimization Algorithm: Based on deep reinforcement learning, the algorithm improves upon proximal policy optimization by using dynamic hybrid constraints, hierarchical time-scale optimization, and adaptive parameter adjustment mechanisms. While maintaining the high efficiency of proximal policy optimization (PPO), it introduces trust domain policy optimization (TRPO) to increase the stability and policy exploration capability of the optimization process, forming a hierarchical optimization system and achieving a balance between stability and efficiency in policy optimization for complex tasks.

[0016] The dynamic mixing constraint is to perform a weighted mixing of the near-end clipping target and the divergence KL penalty target;

[0017] The hierarchical timescale optimization is based on a dynamic hybrid reduction objective, dividing policy optimization into bottom-level optimization and high-level optimization. The bottom-level optimization is a fast update layer that uses gradient descent to minimize the total loss, which is used to quickly improve the policy in each update cycle. The high-level optimization is a low-frequency update layer with trust region constraints, which performs constraint updates in the form of TRPO to keep the overall policy update from not exceeding the trust region boundary, thereby maintaining training stability.

[0018] Step 2.4: Design the reward function: Design rewards based on safety, task completion, and time efficiency. The total reward is equal to the sum of the collision reward, distance reward, and time reward.

[0019] Step 3: Train the collaborative control strategy of the dual-arm robot, deploy it to the robotic arm, and perform the robotic arm grasping task.

[0020] The above-mentioned collaborative control method for a dual-arm robot includes the following: In step 2, the Markov decision model is represented as a quintuple M=(S,A,p,r,γ), where S represents the set of states in the environment; A represents the set of actions in the environment; p:p(s'|s,a), where p represents the probability of an event: in the current state s, the robotic arm performs action a, and the state of the environment changes to s', represented as r(s,a,s'), where r is the reward function; and γ is the discount rate.

[0021] The above-mentioned collaborative control method for a dual-arm robot includes, in step 2.2, the environmental interaction mechanism providing current state information, including object position, dual-arm posture, obstacle information, and task completion progress, based on action a. t The execution status returns a new status s t+1 and reward r t The specific steps for guiding the behavior optimization of the dual-arm robot are as follows:

[0022] a) Initialize the environment: This includes setting the initial state of the dual arms, the position of the workpiece to be grasped, and the target position;

[0023] b) Get the current state s tThis includes the angles of the two arm joints and the condition of the workpiece.

[0024] c) Decision-making optimal action a t The dual-arm robot determines the optimal action based on the input state information using a hybrid constraint hierarchical proximal strategy optimization algorithm.

[0025] d) Perform actions and update the environment: The dual-arm robot performs actions in the environment, and the environmental state is updated accordingly;

[0026] e) Feedback on new status and rewards: The environment returns status s t+1 and reward r t ;

[0027] f) Update strategy: Optimize actions based on new status and reward values ​​to improve the decision-making ability of the dual-arm robot;

[0028] g) Looping interaction: Repeat steps b)-f) above until the grabbing task is completed or the training steps reach the maximum.

[0029] The above-mentioned collaborative control method for a dual-arm robot includes the following step: In step 2.3, the hybrid constraint hierarchical proximal policy optimization algorithm is an improvement on deep reinforcement learning-based proximal policy optimization. Through dynamic hybrid constraints, hierarchical time-scale optimization, and adaptive parameter adjustment mechanisms, it introduces trust domain policy optimization (TRPO) while maintaining the high efficiency of proximal policy optimization (PPO), increasing the stability and policy exploration capability of the optimization process, forming a hierarchical optimization system, and achieving a balance between stability and efficiency in policy optimization for complex tasks. The specific steps are as follows:

[0030] Step 2.3.1 Define the policy: The policy is defined by a probability distribution controlled by parameter θ. The long-term expected return of the policy in the current state is obtained through the state value function and the action value function. The state value function V... π (s t )for:

[0031]

[0032] The action value function Q π (s t ,a t )for:

[0033]

[0034] Among them, E π To achieve the desired result, r t+l γ is the reward obtained at step l after starting from time step t, and s is the discount rate. t express The state at time a t Indicates the action at time t;

[0035] Step 2.3.2 Advantage Function Estimation: Based on the aforementioned action value function and state value function, calculate the advantage function, which measures the relative merits of actions and guides the policy gradient direction. The advantage function A t for:

[0036]

[0037] Step 2.3.3 Policy Update Ratio: To compare the old and new policies during optimization, a probability ratio r is introduced. t (θ):

[0038] Where, π θ (s t， a t ) indicates that the new policy is in state s t Choose action a t The probability, π θold (s t， a t ) indicates that the old policy is in state s t Choose action a t The probability of this ratio is limited to prevent excessive policy update bias from causing instability;

[0039] Step 2.3.4 Dynamic Hybrid Constraints: The near-end pruning objective and the divergence KL penalty objective are weighted and mixed to construct a hybrid optimization objective function L. mix (θ) is:

[0040]

[0041] Where α is the mixing coefficient, used to dynamically balance the weights of the two; L clip (θ) represents the PPO pruning objective of the near-end strategy optimization, L KL (θ) represents the KL penalty target, expressed as:

[0042]

[0043]

[0044] Where E is the expectation, clipr t (*) represents the clipping range of probability ratios. To control the clipping range, β is the divergence KL penalty factor, and D... KL For divergence, For relative operators;

[0045] Step 2.3.5 Layered Time Scale Optimization: Based on the dynamic hybrid reduction objective, policy optimization is divided into bottom-level optimization and high-level optimization. The bottom-level optimization is a fast update layer that uses gradient descent to minimize the total loss, which is used to quickly improve the policy in each update cycle. The high-level optimization is a low-frequency update layer with trust region constraints, which performs constraint updates in the form of TRPO to keep the overall policy update from not exceeding the trust region boundary, thereby maintaining training stability.

[0046] The underlying optimization is represented as follows:

[0047]

[0048] Among them, L total (θ) is the total loss function of the underlying optimization, which rapidly improves the strategy in each update cycle using gradient descent; L(θ) is the total loss function; c1 is the survival coefficient of the value function; V(s) t R represents the state value; t For discount rewards; c2 is the entropy regularization coefficient; H(π) θ ) represents the probability π θ The entropy of the policy distribution under the given conditions;

[0049] The high-level optimization is as follows:

[0050]

[0051] in, This represents maximizing the parameter θ, where δ is the constraint threshold.

[0052] Step 2.3.6 Adaptive parameter adjustment mechanism: Adjust the mixing coefficient α and divergence KL penalty factor β in step 2.3.4 by monitoring the divergence system; if the divergence D KL Exceeding the target threshold D target If the divergence is 1.5 times, then α decreases, and Δα is the change in α each time it is adjusted. At this point, the bias is towards TRPO stable exploration; if the divergence D KL If the value is too small, increase α to favor efficient exploration of PPO; thus, the strategy can achieve an adaptive trade-off between update speed and stability; the adjustment mechanism of the mixing coefficient α is as follows:

[0053] .

[0054] In the above-mentioned collaborative control method for a dual-arm robot, the adjustment mechanism for the mixing coefficient α is as follows:

[0055]

[0056] To further improve constraint adaptability and reduce the need for manual parameter tuning, after each policy update, the divergence D is used as the basis for the optimization. KLThe deviation from the target threshold is automatically adjusted by β to ensure the stability of policy updates; first, the real-time divergence is calculated. With target threshold D target ratio :

[0057]

[0058] Then, based on this ratio and the adjustment factor η, the penalty coefficient β at the t-th iteration is adjusted. t Then adjust β t+1 The truncation process is performed to ensure it does not exceed the limit; the β adjustment mechanism is as follows:

[0059]

[0060]

[0061] The above-mentioned collaborative control method for a dual-arm robot includes the following step: In step 2.4, the reward function is designed based on safety, task completion, and time efficiency factors, with the total reward equal to the sum of collision reward, distance reward, and time reward. The reward function is defined as the single-step total reward R set during the robotic arm's grasping process using reinforcement learning, expressed as:

[0062]

[0063] Among them, R col R represents the collision reward. dis R represents the distance reward. t This indicates a time-based reward.

[0064] In the aforementioned collaborative control method for a dual-arm robot, the designed reward function is the total single-step reward R obtained through reinforcement learning during the robotic arm's grasping process, expressed as:

[0065]

[0066] Among them, R col R represents the collision reward. dis R represents the distance reward. t Indicates a time-based reward;

[0067] The collision reward R col Represented as:

[0068]

[0069] Where, r col Collision reward setting, where α is the collision factor, and α < 0;

[0070] The distance reward Represented as:

[0071]

[0072] Where, r dis Set a value for the distance reward;

[0073] The time reward R t Represented as:

[0074]

[0075] Where t is the training time.

[0076] Compared with existing technologies, this invention has significant advantages. As can be seen from the above scheme, this invention is based on deep reinforcement learning, defines the state space and action space of two arms, constructs a closed-loop interaction mechanism of perception-decision-update, proposes a Hybrid Constraint Hierarchical Proximal Policy Optimization (HCH-PPO) control algorithm, and designs a reward function. HCH-PPO introduces a dual-timescale update mechanism and the idea of ​​Hierarchical Reinforcement Learning (HRL) to improve the traditional Proximal Policy Optimization (PPO), designs a hybrid objective function, and uses dynamic adjustment planning to achieve multi-policy adaptive constraint balance, effectively solving the limitations of single constraints and the problem of multi-stage collaborative tasks in complex high-dimensional discrete action spaces. In summary, this invention has the following advantages:

[0077] 1. Design a dual-timescale hierarchical strategy. Addressing the high dimensionality and multi-stage decision-making complexity of dual-arm collaborative tasks, a hierarchical reinforcement learning (HRL) approach is introduced. High- and low-level strategies are designed to decompose the complex task into multi-level sub-tasks, solving the convergence difficulties and reward sparsity problems associated with direct learning for complex tasks. Specifically, high-level decisions employ TrustRegion Policy Optimization (TRPO) for policy updates to ensure the stability of long-term goals; low-level policies, targeting specific action execution, utilize Proximal Policy Optimization (PPO) for policy updates to achieve rapid local optimization. Furthermore, timescale separation is implemented, employing a dual-timescale update mechanism to balance the contributions of high- and low-level strategies, reducing policy conflicts and improving policy stability.

[0078] 2. Establish an adaptive hybrid constraint mechanism. Addressing the limitations of the traditional PPO single constraint mechanism and the slow convergence speed of pure KL constraints, this mechanism dynamically combines KL divergence penalty with PPO Clip constraints. A dynamic weight coefficient α is introduced, and a hybrid objective function is designed to dynamically adjust the dominant constraint mode based on policy changes, maintaining constraint efficiency. Simultaneously, a KL penalty coefficient β is introduced to adaptively adjust the KL penalty, resolving the penalty strength mismatch problem. These dual coefficients ensure the flexibility of constraint mode selection and the adaptability of penalty strength.

[0079] 3. A Hybrid Constraint Hierarchical Proximal Policy Optimization (HCH-PPO) algorithm is proposed. Addressing the limitations of policy bias and constraint mechanisms in traditional single-layer PPO optimization architectures, this paper proposes a Hybrid Constraint Hierarchical Proximal Policy Optimization (HCH-PPO) algorithm by integrating the aforementioned dual-timescale hierarchical policy and adaptive hybrid constraint mechanism. A task model is constructed, the state space and action space are defined, and a reward mechanism is set. Through multi-mechanism collaborative optimization, the efficiency and stability of the proposed algorithm are verified.

[0080] The following specific embodiments further illustrate the beneficial effects of the present invention. Attached Figure Description

[0081] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0082] The following detailed description, in conjunction with the accompanying drawings and preferred embodiments, outlines the specific implementation methods, features, and effects of a collaborative control method for a dual-arm robot proposed in accordance with the present invention.

[0083] See Figure 1 The present invention discloses a collaborative control method for a dual-arm robot, wherein the method includes the following steps:

[0084] Step 1: Initialize the environment model, including the parameters of the dual-arm robot;

[0085] Step 2: Cooperative Control Strategy for Dual-Arm Robots: Construct a Markov decision model, including the definition of state and action spaces, environmental interaction mechanisms, and a hybrid constraint hierarchical proximal policy optimization algorithm. The execution process of this model is formally described as state observation, policy reasoning, action execution, reward calculation, and policy update. This transforms the dual-arm robot grasping task into a learnable control policy optimization problem, specifically including:

[0086] Step 2.1 defines the state space and action space, wherein the state space S t Defined as:

[0087]

[0088] Among them, Sposition S represents the pose state of the robotic arm's end effector. task S represents the task status, i.e., the current task execution sequence. gripper This indicates the gripper state of the robotic arm's end effector, including open, closed, or held states. env The image represents the environment; L represents the left arm of the dual-arm robot, and R represents the right arm of the dual-arm robot.

[0089] The action space A t Defined as:

[0090]

[0091] Among them, A position A represents the sequence of actions performed by the end effector of the robotic arm. task For the task action sequence, A gripper For the gripper action sequence of the robotic arm end effector, perform opening, closing or holding actions;

[0092] The Markov decision model includes a state space, action space, reward function, state transition function, and discount rate, represented as a quintuple M=(S,A,p,r,γ), where S represents the set of states in the environment; A represents the set of actions in the environment; p:p(s'|s,a), where p represents the probability of an event: in the current state s, the robotic arm performs action a, and the state of the environment changes to s', represented as r(s,a,s'), where r is the reward function; and γ is the discount rate.

[0093] Step 2.2 describes an environmental interaction mechanism that provides current state information, including object position, arm posture, obstacle information, and task completion progress, based on action a. t The execution status returns a new status s t+1 and reward r t The specific steps for guiding the behavior optimization of the dual-arm robot are as follows:

[0094] a) Initialize the environment: This includes setting the initial state of the dual arms, the position of the workpiece to be grasped, and the target position;

[0095] b) Get the current state s t This includes the angles of the two arm joints, the condition of the workpiece, and whether there is a collision.

[0096] c) Decision-making optimal action a t The dual-arm robot determines the optimal action based on the input state information using a hybrid constraint hierarchical proximal strategy optimization algorithm.

[0097] d) Perform actions and update the environment: The dual-arm robot performs actions in the environment, and the environmental state is updated accordingly;

[0098] e) Feedback on new status and rewards: The environment returns status s t+1 and reward r t ;

[0099] f) Update strategy: Optimize actions based on new status and reward values ​​to improve the decision-making ability of the dual-arm robot;

[0100] g) Looping interaction: Repeat steps b)-f) above until the grabbing task is completed or the training steps reach the maximum.

[0101] Step 2.3 The hybrid constraint hierarchical proximal policy optimization algorithm: Based on deep reinforcement learning, the algorithm improves upon proximal policy optimization by using dynamic hybrid constraints, hierarchical time-scale optimization, and adaptive parameter adjustment mechanisms. While maintaining the efficiency of proximal policy optimization (PPO), it introduces trust domain policy optimization (TRPO) to increase the stability and policy exploration capability of the optimization process, forming a hierarchical optimization system. This achieves a balance between stability and efficiency in policy optimization for complex tasks. The specific implementation steps are as follows:

[0102] Step 2.3.1 Define the policy: The policy is defined by a probability distribution controlled by parameter θ. The long-term expected return of the policy in the current state is obtained through the state value function and the action value function. The state value function V... π (s t )for:

[0103]

[0104] The action value function Q π (s t ,a t )for:

[0105]

[0106] Among them, E π To achieve the desired result, r t+l γ is the reward obtained at step l after starting from time step t, and s is the discount rate. t express The state at time a t Indicates the action at time t;

[0107] Step 2.3.2 Advantage Function Estimation: Based on the aforementioned action value function and state value function, calculate the advantage function, which measures the relative merits of actions and guides the policy gradient direction. The advantage function A t for:

[0108]

[0109] Step 2.3.3 Policy Update Ratio: To compare the old and new policies during optimization, a probability ratio r is introduced. t (θ):

[0110] Where, π θ (s) t， a t ) indicates that the new policy is in state s t Choose action a t The probability, π θold (s) t， a t ) indicates that the old policy is in state s t Choose action a t The probability of this ratio is limited to prevent excessive policy update bias from causing instability;

[0111] Step 2.3.4 Dynamic Hybrid Constraints: The near-end pruning objective and the divergence KL penalty objective are weighted and mixed to construct a hybrid optimization objective function L. mix (θ) is:

[0112]

[0113] Where α is the mixing coefficient, used to dynamically balance the weights of the two; L clip (θ) represents the PPO pruning objective of the near-end strategy optimization, L KL (θ) represents the KL penalty target, expressed as:

[0114]

[0115]

[0116] Where E is the expectation, clipr t (*) represents the clipping range of probability ratios. To control the clipping range, β is the divergence KL penalty factor, and D... KL For divergence, For relative operators;

[0117] Step 2.3.5 Layered Time Scale Optimization: Based on the dynamic hybrid reduction objective, policy optimization is divided into bottom-level optimization and high-level optimization. The bottom-level optimization is a fast update layer that uses gradient descent to minimize the total loss, which is used to quickly improve the policy in each update cycle. The high-level optimization is a low-frequency update layer with trust region constraints, which performs constraint updates in the form of TRPO to keep the overall policy update from not exceeding the trust region boundary, thereby maintaining training stability.

[0118] The underlying optimization is represented as follows:

[0119]

[0120] Among them, L total (θ) is the total loss function of the underlying optimization, which rapidly improves the strategy in each update cycle using gradient descent; L(θ) is...; c1 is the survival coefficient of the value function; V(s) t R represents the state value; t For discount rewards; c2 is the entropy regularization coefficient; H(π) θ ) represents the probability π θ The entropy of the policy distribution under the given conditions.

[0121] The high-level optimization is as follows:

[0122]

[0123] in, This represents maximizing the parameter θ, where δ is the constraint threshold.

[0124] Step 2.3.6 Adaptive parameter adjustment mechanism: Adjust the mixing coefficient α and divergence KL penalty factor β in step 2.3.4 by monitoring the divergence system; if the divergence D KL Exceeding the target threshold D target If the divergence is 1.5 times, then α decreases, and Δα is the change in α each time it is adjusted. At this point, the bias is towards TRPO stable exploration; if the divergence D KL If the value is too small, increase α to favor efficient exploration of PPO; thus, the strategy can achieve an adaptive trade-off between update speed and stability; the adjustment mechanism of the mixing coefficient α is as follows:

[0125]

[0126] To further improve constraint adaptability and reduce the need for manual parameter tuning, after each policy update, the divergence D is used as the basis for the optimization. KL The deviation from the target threshold is automatically adjusted by β to ensure the stability of policy updates; first, the real-time divergence is calculated. With target threshold D target ratio :

[0127]

[0128] Then, based on this ratio and the adjustment factor η, the penalty coefficient β at the t-th iteration is adjusted. t Then adjust β t+1 A threshold-based exponential adjustment mechanism is used to truncate gradients to prevent them from exceeding a certain limit, thus avoiding gradient explosion or vanishing. This mechanism only requires comparing the KL ratio with a threshold, without complex calculations, and is parameter robust. The algorithm can autonomously balance the exploratory nature and stability of the strategy for complex crawling tasks. The β adjustment mechanism is as follows:

[0129]

[0130]

[0131] Step 2.4: Design the reward function: Design the reward based on safety, task completion, and time efficiency. The total reward equals the sum of the collision reward, distance reward, and time reward. This comprehensively considers the agent's safety, task completion speed, and task completion quality, enriching feedback information and guiding the agent towards the expected goal. The designed reward function is the single-step total reward R set during the robotic arm's grasping process using reinforcement learning, expressed as:

[0132]

[0133] Among them, R col R represents the collision reward. dis R represents the distance reward. t This indicates a time-based reward.

[0134] The collision reward R col Represented as:

[0135]

[0136] Where, r col The collision reward setting is α, which is the collision factor. α < 0. If α is too large, the penalized AI will collide frequently. If α is too small, the excessive penalty will lead to over-conservatism and reduced efficiency.

[0137] The distance reward R dis Represented as:

[0138]

[0139] Where, r dis Set a value for the distance reward;

[0140] The time reward R t Represented as:

[0141]

[0142] Where t is the training time;

[0143] Step 3: Train the collaborative control strategy of the dual-arm robot, deploy it to the robotic arm, and perform the robotic arm grasping task.

[0144] A dual-arm robot collaborative system includes hardware facilities and software components. The hardware facilities include left and right robotic arms and a depth camera. The software components include a vision detection module and a robotic arm control algorithm module.

[0145] The vision detection module first uses a depth camera to capture the RGB and depth information of random objects on the worktable, identifies and determines the object's position and target placement position, and assigns the grasping task to the left and right robotic arms. Subsequently, the object's position and velocity data are transmitted to the left and right robotic arm control algorithm module, which outputs the precise movement speed of each joint of the left and right robotic arms to guide the end effector of the robotic arm to move towards the target object. When the end effector approaches the preset distance of the moving object, the gripper of the end effector closes, completing the grasping action. After successful grasping, the end effector carries the grasped object to the placement position and performs an alignment operation. After alignment, the robotic arm carries the object downwards. When the distance between the object's position and the target placement position is less than a threshold, the gripper opens, completing the placement of the object.

[0146] The robotic arm control algorithm module controls the joint movements of the left and right robotic arms and the gripper's grasping operation. First, it acquires the state of the moving object and the gripper, calculates the gripper's speed, and controls the left and right robotic arms to approach the target. This module outputs the motion space, controlling the left and right robotic arms to grasp the target object and move it to the placement position to release the gripper and place the object. Furthermore, the control module continuously interacts with the environment to obtain the action with the highest reward in the current state, thus enabling the left and right robotic arms to track, grasp, and place the target object.

[0147] Performance Analysis:

[0148] (1) Experimental setup

[0149] The specific task of the dual-arm collaborative operation in the experiment was for two UR3 robotic arms to efficiently and without collision transfer randomly distributed workpieces on a worktable to a designated storage box within a limited time. The dual-arm collaborative operation involved visual perception, dynamic collaborative control strategies, and 3D modeling. To address the requirements of high-dimensional discrete motion and multiple tasks, a simulation task scenario was built using Solidworks 3D modeling software combined with CoppeliaSim simulation software. A visual sensor camera was placed above the robotic arms for visual perception. The two UR3 robotic arms, equipped with two-finger grippers, were fixed to the base. Eight workpiece and toolbox models were imported to establish the simulation experimental scenario. The agent, based on the item category and coordinates detected by the visual sensors, assigned the task to either the left or right arm and provided a task sequence decision. It then guided the end effector to move above the item to be picked up to perform the gripping operation. After successful gripping, the end effector moved above the pickup position, aligned, and then moved down to the designated position before releasing the gripper, completing the item's classification and placement. The robotic arm then returned to the ready position to await the next task until all workpieces were picked up. The two robotic arms coordinate and allocate tasks according to the learned task strategy, placing the workpieces in the target positions in sequence to ensure that no collisions or misplacements occur throughout the process, until all sub-tasks are successfully placed.

[0150] (2) Results Analysis

[0151] The trained model was compared with the baseline model in 50 simulation experiments. With a total training step count of 5000, A2C and RePPO failed to effectively learn the correct policy. Therefore, the task completion time and final success rate of the other three algorithms were analyzed, resulting in the data in Table 1. It can be seen that the first three algorithms effectively completed the 50 simulation experiments. HCH-PPO had the shortest time (1 hour, 28 minutes, and 59 seconds) and a higher success rate of 0.98 than PPO and TRPO. A2C and RePPO had the lowest success rate and the longest time in the 50 experiments, which stems from the fact that they did not learn an effective policy in the first 5000 training steps. Overall, HCH-PPO outperformed the other algorithms in all aspects and had the best performance.

[0152] Table 1 Comparison of Task Success Rate and Duration

[0153] HCH-PPO 50 49 0.98 1:28:59.17 PPO 50 43 0.86 1:32:59.89 TRPOA2C 5050 452 0.900.04 1:37:49.77 1:50:23.08 RePPO 50 3 0.06 1:45:30.51

[0154] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments without departing from the technical essence of the present invention shall still fall within the scope of the present invention.

Claims

1. A collaborative control method for a dual-arm robot, characterized in that: The method includes the following steps: Step 1: Initialize the environment model, including the parameters of the dual-arm robot; Step 2: Cooperative Control Strategy for Dual-Arm Robots: Construct a Markov decision model, including the definition of state and action spaces, environmental interaction mechanisms, and a hybrid constraint hierarchical proximal policy optimization algorithm. The execution process of this Markov decision model is formally described as state observation, policy reasoning, action execution, reward calculation, and policy update, transforming the dual-arm robot grasping task into a learnable control policy optimization problem, specifically including: Step 2.1 defines the state space and action space, wherein the state space S t Defined as: Among them, S position S represents the pose state of the robotic arm's end effector. task S represents the task status, i.e., the current task execution sequence. gripper S indicates the gripper state of the robotic arm's end effector. env The image represents the environment; L represents the left arm of the dual-arm robot, and R represents the right arm of the dual-arm robot. The action space A t Defined as: Among them, A position A represents the sequence of actions performed by the end effector of the robotic arm. task For the task action sequence, A gripper This refers to the gripper motion sequence of the robotic arm's end effector. Step 2.2 The environmental interaction mechanism provides current state information, including object position, arm posture, obstacle information, and task completion progress, based on action a. t The execution status returns a new status s t+1 and reward r t To guide the dual-arm robot in optimizing its behavior; Step 2.3 describes a hybrid constraint hierarchical proximal policy optimization algorithm: This algorithm improves upon deep reinforcement learning-based proximal policy optimization by using dynamic hybrid constraints, hierarchical time-scale optimization, and adaptive parameter adjustment mechanisms. While maintaining the efficiency of proximal policy optimization (PPO), it introduces trust domain policy optimization (TRPO) to increase the stability and policy exploration capability of the optimization process, forming a hierarchical optimization system. This achieves a balance between stability and efficiency in policy optimization for complex tasks. The specific steps are as follows: Step 2.3.1 Define the policy: The policy is defined by a probability distribution controlled by parameter θ. The long-term expected return of the policy in the current state is obtained through the state value function and the action value function. The state value function V... π (s t )for: The action value function Q π (s t ,a t )for: Among them, E π In order to achieve expectations, Starting from time t, the th The reward obtained per step, γ is the discount rate, s t express The state at time a t Indicates the action at time t; Step 2.3.2 Advantage Function Estimation: Based on the aforementioned action value function and state value function, calculate the advantage function, which measures the relative merits of actions and guides the policy gradient direction. The advantage function A t for: Step 2.3.3 Policy Update Ratio: To compare the old and new policies during optimization, a probability ratio r is introduced. t (θ): Where, π θ (s t， a t ) indicates that the new policy is in state s t Choose action a t The probability, π θold (s t， a t ) indicates that the old policy is in state s t Choose action a t The probability of that probability is greater than r. t (θ) Limitations should be imposed to prevent excessive policy update deviations from causing instability; Step 2.3.4 Dynamic Hybrid Constraints: The near-end pruning objective and the divergence KL penalty objective are weighted and mixed to construct a hybrid optimization objective function L. mix (θ) is: Where α is the mixing coefficient, used to dynamically balance the weights of the two; L clip (θ) represents the PPO pruning objective of the near-end strategy optimization, L KL (θ) represents the KL penalty target, expressed as: Where E is the expectation, clipr t (*) represents the clipping range of probability ratios. To control the clipping range, β is the divergence KL penalty factor, and D... KL For divergence, For relative operators; Step 2.3.5 Layered Time Scale Optimization: Based on the dynamic hybrid constraint objective, policy optimization is divided into bottom-level optimization and high-level optimization. The bottom-level optimization is a fast update layer that uses gradient descent to minimize the total loss, which is used to quickly improve the policy in each update cycle. The high-level optimization is a low-frequency update layer with trust region constraints, which performs constraint updates in the form of TRPO to keep the overall policy update from not exceeding the trust region boundary, thereby maintaining training stability. The underlying optimization is represented as follows: Among them, L total (θ) is the total loss function of the underlying optimization, which rapidly improves the policy in each update cycle using gradient descent; L(θ) is the total loss function; V(s) t R represents the state value; t For discount rewards; c2 is the entropy regularization coefficient; H(π) θ ) represents the probability π θ The entropy of the policy distribution under the given conditions; The high-level optimization is as follows: in, This represents maximizing the parameter θ, where δ is the constraint threshold. Step 2.3.6 Adaptive parameter adjustment mechanism: Adjust the mixing coefficient α and divergence KL penalty factor β in step 2.3.4 by monitoring the divergence system; if the divergence D KL Exceeding the target threshold D target If the divergence is 1.5 times, then α decreases, and Δα is the change in α each time it is adjusted. At this point, the bias is towards TRPO stable exploration; if the divergence D KL If the value is too small, increase α to favor efficient exploration of PPO; thus, the strategy can achieve an adaptive trade-off between update speed and stability; the adjustment mechanism of the mixing coefficient α is as follows: Step 2.4: Design the reward function: Design rewards based on safety, task completion, and time efficiency. The total reward is equal to the sum of the collision reward, distance reward, and time reward. Step 3: Train the collaborative control strategy of the dual-arm robot, deploy it to the robotic arm, and perform the robotic arm grasping task.

2. The collaborative control method for a dual-arm robot as described in claim 1, characterized in that: In step 2.2, the environmental interaction mechanism provides current state information, including object position, arm posture, obstacle information, and task completion progress, based on action a. t The execution status returns a new status s t+1 and reward r t The specific steps for guiding the behavior optimization of the dual-arm robot are as follows: a) Initialize the environment: This includes setting the initial state of the dual arms, the position of the workpiece to be grasped, and the target position; b) Get the current state s t This includes the angles of the two arm joints and the condition of the workpiece. c) Decision-making optimal action a t The dual-arm robot determines the optimal action based on the input state information using a hybrid constraint hierarchical proximal strategy optimization algorithm. d) Perform actions and update the environment: The dual-arm robot performs actions in the environment, and the environmental state is updated accordingly; e) Feedback on new status and rewards: The environment returns status s t+1 and reward r t ; f) Update strategy: Optimize actions based on new status and reward values ​​to improve the decision-making ability of the dual-arm robot; g) Looping interaction: Repeat steps b)-f) above until the grabbing task is completed or the training steps reach the maximum.

3. The collaborative control method for a dual-arm robot as described in claim 1, characterized in that: In step 2.3.6, the adjustment mechanism for the mixing coefficient α is as follows: To further improve constraint adaptability and reduce the need for manual parameter tuning, after each policy update, the divergence D is used as the basis for the optimization. KL The deviation from the target threshold is automatically adjusted by β to ensure the stability of policy updates; first, the real-time divergence is calculated. With target threshold D target ratio : Then based on that ratio The penalty coefficient at time t is adjusted by combining the adjustment factor η with β. t Then adjust β t+1 The truncation process is performed to ensure it does not exceed the limit; the β adjustment mechanism is as follows: 。 4. The collaborative control method for a dual-arm robot as described in claim 1, characterized in that: In step 2.4, the reward function is designed based on factors such as safety, task completion, and time efficiency. The total reward equals the sum of the collision reward, distance reward, and time reward. The reward function is defined as the single-step total reward R set during the robotic arm's grasping process using reinforcement learning, expressed as: Among them, R col R represents the collision reward. dis R represents the distance reward. t This indicates a time-based reward.

5. The collaborative control method for a dual-arm robot as described in claim 4, characterized in that: The designed reward function, which is the total single-step reward R set during the robotic arm's grasping process using reinforcement learning, is expressed as: Among them, R col R represents the collision reward. dis R represents the distance reward. t Indicates a time-based reward; The collision reward R col Represented as: Where, r col Collision reward settings, For collision factor, <0; The distance reward Represented as: Where, r dis Set a value for the distance reward; The time reward R t Represented as: 。

Images