Double-arm cooperative operation control method based on intelligent coordination control

Abstract

The invention relates to the technical field of robot intelligent control, and discloses a double-arm cooperative operation control method based on intelligent coordination control, which adopts a layered architecture of high-level task strategy learning and bottom-level multi-target cooperative dynamics control. The high layer generates a task space target through hybrid reinforcement learning of an embedded double-arm multi-head collaborative attention mechanism; the bottom layer adopts a multi-target dynamics control law including a main task control item, a cooperative control item and a null space control item to calculate the joint torque according to the target; and closed-loop self-adaptive updating is realized through state feedback and a multi-target reward function. According to the method, a hierarchical control architecture of high-level task strategy learning and bottom-layer multi-target cooperative dynamics control is adopted, so that a high-level reinforcement learning strategy is not influenced by a bottom-layer complex dynamics equation, and the learning efficiency and migration ability of the strategy are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot intelligent control, and in particular to a dual-arm collaborative operation control method based on intelligent coordination control. BACKGROUND

[0002] With the rapid development of industrial automation, precision manufacturing and human-machine collaboration, robot intelligent control technology has ushered in the demand for large-scale application. Dual-arm robots, which can simulate human hand collaborative operation, can complete complex tasks such as precision assembly, collaborative transportation and flexible grasping that single robots cannot achieve, and have become a research hotspot and core application direction in the field of robots.

[0003] Currently, the control technology for dual-arm collaborative operation mainly focuses on a "single-layer integrated control architecture". Specifically, two types of mainstream technical solutions are formed: one is a traditional control scheme based on a model, including impedance control, hybrid force / position control and inverse dynamics compensation, etc. The core logic is to integrate high-level task goals and low-level physical execution into a single control link by establishing an accurate dual-arm dynamics model, and directly generate control instructions by solving the dynamics equation. The other is an intelligent control scheme based on reinforcement learning, which allows the robot to learn collaborative strategies during interaction with the environment by constructing a reward function, without relying on accurate dynamics modeling. However, in order to ensure the physical executability of the strategy, the high-level strategy network still needs to directly receive the low-level dynamics-related states and adapt to the low-level physical characteristics simultaneously during the training process.

[0004] However, the present inventors have found that the single-layer integrated control architecture used in existing dual-arm collaborative control technology does not clearly separate task planning and physical execution into high-level and low-level, but directly integrates them into a single control link, resulting in deep coupling between task planning and physical execution, and making it impossible to effectively separate them. SUMMARY

[0005] To address the above problems, the present application provides a dual-arm collaborative operation control method based on intelligent coordination control, which aims to improve the single-layer integrated control architecture used in existing dual-arm collaborative control technology, which does not clearly separate task planning and physical execution into high-level and low-level, but directly integrates them into a single control link, resulting in deep coupling between task planning and physical execution, and making it impossible to effectively separate them.

[0006] The present application provides the following technical solution: a dual-arm collaborative operation control method based on intelligent coordination control, which adopts a layered control architecture of high-level task strategy learning and low-level multi-objective collaborative dynamics control, including the following steps: S1, system state acquisition and initialization, acquiring joint angle, end pose, joint speed, contact force, object attitude and external disturbance data of the dual-arm robot, constructing a system state vector and initializing control parameters and a high-level reinforcement learning network; S2, high-level task policy learning, generating a task space target through a hybrid reinforcement learning policy network embedded with a dual-arm multi-head collaborative attention mechanism according to the system state vector, wherein the hybrid reinforcement learning policy network is trained in a dynamic weighted fusion manner of PPO algorithm and SAC algorithm; S3, bottom-level multi-objective collaborative dynamics control, calculating joint control torque by using a multi-objective collaborative dynamics control law containing a main task control item, a collaborative control item and a null space control item according to the task space target output by the high-level task policy learning step; S4, feedback and adaptive update, collecting reward signals according to the actual state after execution, and updating the hybrid reinforcement learning policy network in the high-level task policy learning step online or offline through a multi-objective reward function to form a closed-loop adaptive control.

[0007] Preferably, the specific process of the dual-arm multi-head collaborative attention mechanism embedded in the high-level task policy learning is as follows: First, feature extraction is performed on the left arm state feature vector and the right arm state feature vector respectively; Then, the cross-dependent relationship between the left arm features and the right arm features is calculated through multiple attention heads respectively; Subsequently, the outputs of all attention heads are fused and spliced; Finally, a global feature vector containing dual-arm collaborative information is generated through linear mapping.

[0008] Preferably, the specific process of the hybrid reinforcement learning policy network trained in a dynamic weighted fusion manner of PPO algorithm and SAC algorithm is as follows: In the early stage of training, the loss weight of the SAC algorithm branch is increased to enhance the exploration ability, and in the later stage of training, the loss weight of the PPO algorithm branch is increased to ensure the smoothness of policy update, and the loss functions of the two algorithm branches are weighted and summed through a dynamic weight coefficient that changes adaptively with the training round number to optimize the same policy network parameters.

[0009] Preferably, the multi-objective collaborative dynamics control law used in the bottom-level multi-objective collaborative dynamics control contains the following three specific control items: The main task control item is used to generate the corresponding expected control torque according to the task space position and force error; The collaborative control item is used to adjust the force balance between the left arm and the right arm according to the deviation of the actual contact force difference and the expected force difference between the left arm and the right arm and the adaptive gain coefficient; Null space control terms are used to achieve joint pose optimization, obstacle avoidance and joint limit protection without affecting the main task execution.

[0010] Preferably, the multi-objective reward function used in the feedback and adaptive update is composed of the following multiple sub-reward terms: Task completion degree reward sub-term, two-arm cooperation reward sub-term, force control accuracy reward sub-term, safety reward sub-term and execution efficiency reward sub-term, the weight coefficients of each sub-reward term remain fixed or are dynamically adjusted according to the task stage in the training process.

[0011] Preferably, the task space target output by the high-level task strategy learning includes left arm expected end pose, right arm expected end pose, left arm expected contact force, right arm expected contact force and object overall expected motion trajectory, and the task space target is converted into an input reference signal of the bottom layer control through an adjustable gain coefficient.

[0012] Preferably, the system state vector specifically includes: left arm end pose, right arm end pose, left arm contact force, right arm contact force, operated object pose, operated object velocity and external disturbance estimation value, and all state quantities are normalized before inputting the high-level task strategy learning step.

[0013] Preferably, when the method is applied to an assembly task, the high-level task strategy learning preferentially learns a two-arm relative pose coordination strategy, and the collaborative control term in the bottom layer multi-objective collaborative dynamics control preferentially ensures force balance in the insertion force direction.

[0014] Preferably, when the method is applied to a collaborative carrying task, the high-level task strategy learning preferentially learns a common holding strategy of the two arms on the object pose, and the collaborative control term in the bottom layer multi-objective collaborative dynamics control preferentially eliminates the internal force circulation between the two arms.

[0015] Preferably, the hierarchical control architecture achieves decoupling of task planning and physical execution through an explicit high-level to bottom layer one-way task target interface and a bottom layer to high-level state and reward feedback interface.

[0016] The present application has the following beneficial effects: 1、The present application adopts a hierarchical control architecture of high-level task strategy learning and bottom layer multi-objective collaborative dynamics control, and realizes decoupling of task planning and physical execution through an explicit high-level to bottom layer one-way task target interface and a bottom layer to high-level state and reward feedback interface, so that the high-level reinforcement learning strategy is not affected by the bottom layer complex dynamics equation, and the learning efficiency and migration ability of the strategy are significantly improved.

[0017] 2、The application can effectively capture the cross-dependent relationship between the left and right arms and balance early sufficient exploration and late stable convergence, thereby greatly improving the action consistency and force coordination of the dual-arm collaborative operation.

[0018] 3、The bottom layer multi-objective collaborative dynamics control law contains a main task control item, a collaborative control item and a null space control item, and the gain matrix in the collaborative control item is dynamically adjusted according to the task type, and the null space control item ensures that the joints are always away from mechanical limits and singular configurations, realizing high-precision, high-robustness and high-safety dual-arm collaborative operation. BRIEF DESCRIPTION OF DRAWINGS

[0019] Fig. 1 A general framework structure diagram of a dual-arm collaborative operation control method based on intelligent coordination control is provided for the application. Fig. 2 A high-level hybrid reinforcement learning module (PPO+SAC) structure diagram of a dual-arm collaborative operation control method based on intelligent coordination control is provided for the application. Fig. 3 A bottom layer multi-objective collaborative dynamics control law principle diagram of a dual-arm collaborative operation control method based on intelligent coordination control is provided for the application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the application will be described below in conjunction with the drawings, obviously, the described embodiments are only part of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0021] Embodiment one Reference Figs. 1-3 In the first embodiment of the application, the application provides a dual-arm collaborative operation control method based on intelligent coordination control, which adopts a hierarchical control architecture of high-level task strategy learning and bottom layer multi-objective collaborative dynamics control, including the following steps: S1, system state acquisition and initialization, acquiring the joint angle, joint speed, end pose, contact force, object attitude and external disturbance data of the dual-arm robot, forming a system state vector and completing the initialization of control parameters and high-level reinforcement learning network; S2, high-level task policy learning, generating a task space target according to the system state vector through a hybrid reinforcement learning policy network embedded with a dual-arm multi-head collaborative attention mechanism, wherein the hybrid reinforcement learning policy network is trained in a dynamic weighted fusion manner of PPO algorithm and SAC algorithm; S3, low-level multi-objective collaborative dynamics control, calculating joint control torque according to the task space target output by the high-level task policy learning step, using a multi-objective collaborative dynamics control law containing a main task control item, a collaborative control item and a null space control item; S4, feedback and adaptive update, collecting reward signals according to the actual state after execution, and updating the hybrid reinforcement learning policy network in the high-level task policy learning step online or offline through a multi-objective reward function, forming a closed-loop adaptive control.

[0022] Specifically, system state acquisition and initialization acquires the joint angle, joint velocity, end six-dimensional pose, end six-dimensional contact force, six-dimensional pose of the operated object, six-dimensional velocity of the operated object and external disturbance estimation value of the left arm and the right arm, and all the collected data constitute a system state vector. All state quantities are normalized by zero mean and unit variance and input to the high-level policy network. The initialization stage completes the setting of reinforcement learning network parameters, control gain coefficients, experience pool and all hyperparameters.

[0023] The high-level task policy learning receives the normalized system state vector and generates a task space target through a hybrid reinforcement learning policy network. The hybrid reinforcement learning policy network is composed of a PPO algorithm branch and a SAC algorithm branch, and the two branches share the same set of policy network parameters. During training, the loss weight of the SAC algorithm branch takes a larger value in the early training to enhance the exploration ability, and decreases linearly with the increase of training rounds; the loss weight of the PPO algorithm branch increases linearly to ensure smooth policy update. The loss weights of the two branches are summed and then back-propagated to update the policy network parameters.

[0024] The high-level policy network outputs an action vector, which is linearly mapped through a fixed gain matrix to generate a task space target. The task space target includes the left arm desired end pose, the right arm desired end pose, the left arm desired contact force, the right arm desired contact force and the overall desired motion trajectory of the operated object.

[0025] The low-level multi-objective collaborative dynamics control receives the task space target output by the high-level task policy learning step and calculates the joint control torque of the left and right arms. The joint control torque is composed of four parts: a main task control item, a collaborative control item, a null space control item and a dynamics compensation item.

[0026] The main task control item forms an impedance control relationship in the task space according to the task space position error, the speed error and the expected acceleration, generates a task space expected torque, and is mapped to the joint space through the respective task space Jacobian matrix transposition of the left arm and the right arm.

[0027] The synergic control item realizes the balance adjustment of the contact forces between the two arms by respectively acting on the two arms through the Jacobian matrix transposition of the left arm and the right arm after multiplying the difference between the actual contact force of the left arm and the actual contact force of the right arm and the deviation of each from the expected contact force by an adaptive gain coefficient. The null space control item converges the joints to the preset intermediate posture while suppressing the joint speed without affecting the main task torque, realizes posture optimization, obstacle avoidance and joint limit protection. The dynamics compensation item calculates and completely compensates the rigid body inertia matrix, the Coriolis force and centrifugal force term, the gravity term and the joint friction of the two arms in real time. The feedback and adaptive update collect the new state and the immediate reward after the end of each control cycle. The immediate reward is composed of the weighted sum of the square of the task position error, the square of the contact force error, the square of the sum of the contact forces of the two arms, the joint limit and singularity penalty term and the energy consumption term, and each weight coefficient remains fixed during the entire training process. The four-tuple composed of the current state, the action, the reward and the new state is stored in the experience pool, which is used for batch gradient update or real-time single-step update of the mixed reinforcement learning strategy network, so as to realize the continuous optimization of the strategy network parameters, form a closed-loop adaptive control, and realize the high-precision and high-robustness collaborative operation control of the dual-arm robot in a complex contact environment.

[0028] Further, the specific process of the dual-arm multi-head synergic attention mechanism embedded in the high-level task strategy learning is as follows: First, the left arm state feature vector and the right arm state feature vector are respectively subjected to feature extraction; Then, the cross-dependent relationship between the left arm features and the right arm features is calculated through multiple attention heads; Subsequently, the outputs of all attention heads are fused and spliced; Finally, a global feature vector containing dual-arm synergic information is generated through linear mapping.

[0029] Specifically, the system state vector is first split into a left arm state sub-vector and a right arm state sub-vector. The left arm state sub-vector contains the left arm end six-dimensional pose, the left arm end six-dimensional contact force, the left arm joint angular velocity, and the left arm end velocity. The right arm state sub-vector contains the right arm end six-dimensional pose, the right arm end six-dimensional contact force, the right arm joint angular velocity, and the right arm end velocity. The six-dimensional pose and the six-dimensional velocity of the object being operated are simultaneously spliced into the left arm state sub-vector and the right arm state sub-vector. The left arm state sub-vector is subjected to feature extraction through two fully connected networks to obtain a left arm feature vector with a dimension of 256. The right arm state sub-vector is also subjected to feature extraction through two fully connected networks to obtain a right arm feature vector with a dimension of 256. Both of the two fully connected networks use a ReLU activation function, and the weight initialization uses the Xavier method.

[0030] Subsequently, the left arm feature vector and the right arm feature vector are sent to an eight-head collaborative attention module. Each attention head respectively takes the left arm feature vector as the query vector, the right arm feature vector as the key vector and the value vector, and calculates the one-way attention weight of the left arm to the right arm. At the same time, the right arm feature vector is taken as the query vector, and the left arm feature vector is taken as the key vector and the value vector to calculate the one-way attention weight of the right arm to the left arm. The scaling dot product calculation is performed inside each attention head, and the scaling factor is the reciprocal of the square root of the feature dimension. The outputs of the eight attention heads are respectively subjected to independent feedforward networks and then spliced to form a cross feature vector with a dimension of 2048. The cross feature vector is respectively subjected to residual connection with the original left arm feature vector and the right arm feature vector, and is subjected to layer normalization processing to obtain a left arm global feature vector and a right arm global feature vector containing the collaborative information of the two arms. The left arm global feature vector and the right arm global feature vector are spliced again and reduced to 512 dimensions through a fully connected network to form a final global state feature vector.

[0031] The global state feature vector is then sent to the subsequent value evaluation network and policy output network of the hybrid reinforcement learning policy network to ensure that the policy decision depends on the local state of the left and right arms and the cross-dependent relationship between the two, so as to realize the consistency of the two-arm actions and the coordination of the forces in generating the task space target.

[0032] The eight-head collaborative attention module adopts the standard implementation method of the multi-head attention mechanism in actual implementation. All the attention heads are calculated in parallel, and the calculation is directly spliced after completion, without the need for an additional gating mechanism to realize the effective fusion of the left and right arm states.

[0033] Further, the specific process of training the hybrid reinforcement learning policy network in a dynamic weighted fusion manner of PPO algorithm and SAC algorithm is as follows: In the early stages of training, the loss weight of the SAC algorithm branch is increased to enhance the exploration capability, while in the later stages of training, the loss weight of the PPO algorithm branch is increased to ensure the smoothness of policy updates. The loss functions of the two algorithm branches are weighted and summed by a dynamic weight coefficient that adapts to the number of training rounds, and then jointly optimize the network parameters of the same policy.

[0034] Specifically, the hybrid reinforcement learning policy network comprises a shared policy network and two independent evaluation networks. The policy network receives a 512-dimensional global state feature vector as input and outputs the normal distribution mean μ and diagonal covariance matrix Σ of the 24-dimensional action vector. The two evaluation networks, denoted as Q1 and Q2, estimate the soft Q-values ​​of the current state-action pairs, respectively. An independent value network V is also included to estimate the state value function.

[0035] The training process uses dynamic weight coefficients. α Control the ratio of the two-branch loss. At the start of training... α Setting the value to 0.9 allows the SAC branch loss to dominate. This branch loss consists of three parts: the mean squared error regression loss of the Q1 and Q2 networks, the policy entropy maximization term, and the regression loss of the value network V, thus maintaining a high policy entropy and enabling extensive exploration.

[0036] As the number of training epochs increases from 0 to the total number of epochs... The dynamic weighting coefficient α decays linearly. ; in, epoch This indicates the current training round number, and the total number of rounds indicates the total number of iterations required to train the preset policy network. α These are dynamic weighting coefficients used to balance the proportions of PPO algorithm branch loss and SAC algorithm branch loss in the total loss function. α The initial value is set to a large value at the beginning of training to enhance the policy's exploration ability. epoch The increase, α By gradually decreasing the linear value, the strategy training process gradually transitions from an emphasis on exploration to an emphasis on stable convergence.

[0037] when epoch When 70% of the total number of rounds is reached, α When the value drops below 0.31, the loss weight of the PPO branch becomes dominant. The PPO branch implements a conservative policy update by pruning the importance sampling ratio r_t within the range of 0.8 to 1.2.

[0038] In each training step, the SAC branch loss is first calculated separately. and PPO branch loss Then press the current button. α The total loss is obtained by weighted summation: ; in, L The total loss function is used to jointly optimize the hybrid reinforcement learning policy network. This represents the policy optimization loss constructed based on the PPO algorithm, used to constrain the policy update magnitude to ensure the stability of the training process; This represents the policy optimization loss constructed based on the SAC algorithm, used to enhance the policy's exploration capability through entropy regularization; α These are dynamic weighting coefficients used for adjustment. and The relative weight in the total loss, 1 α To and α The corresponding complementary weights.

[0039] Total loss A gradient descent update is performed on the shared policy network parameters θ. The Q1 network, Q2 network, and value network V are updated independently according to their respective branch rules, without interfering with each other.

[0040] when α When the entropy regularization coefficient λ decays to below 0.2, it also decays linearly from the initial value of 0.005 to 0.0005, further reducing randomness and making the strategy gradually tend toward deterministic output.

[0041] Through the aforementioned dynamic weighted fusion training method, the hybrid reinforcement learning policy network completes sufficient exploration in the early stage of training, achieves stable convergence in the later stage of training, and finally outputs a stable task space target for use by the underlying control module.

[0042] Furthermore, the multi-objective cooperative dynamics control law used in the underlying multi-objective cooperative dynamics control includes the following three specific control terms: The main task control item is used to generate the corresponding desired control torque based on the task's spatial location and force error. The cooperative control term is used to adjust the force balance between the two arms based on the deviation between the actual contact force difference and the expected force difference between the left and right arms and the adaptive gain coefficient. Zero-space control is used to achieve joint posture optimization, obstacle avoidance, and joint limit protection without affecting the execution of the main task.

[0043] Specifically, the main task control item realizes independent compliant tracking of the left and right arms to the expected end pose and the expected contact force issued by the upper layer. The left arm first calculates the end position error and the end velocity error, the position error being equal to the difference between the expected end pose of the left arm and the actual end pose, and the velocity error being equal to the difference between the expected end velocity and the actual end velocity; the position error is multiplied by a diagonal position gain matrix, and the velocity error is multiplied by a diagonal velocity gain matrix, and the sum of the two is added to the expected contact force of the left arm to obtain a six-dimensional generalized force in the task space of the left arm; the six-dimensional generalized force is mapped to the joint space through the transpose of the current Jacobian matrix of the left arm to form the main task control item of the left arm. The right arm is calculated in the same way to obtain the main task control item of the right arm.

[0044] The cooperative control item realizes force balance between the two arms. The vector sum of the actual six-dimensional contact force of the left arm and the actual six-dimensional contact force of the right arm is calculated in real time to obtain the current internal force; the expected internal force is set to a zero vector or a preset constant vector according to the task type; the internal force error is equal to the difference between the current internal force and the expected internal force; the internal force error is multiplied by a diagonal cooperative gain matrix to obtain a cooperative adjustment force; the cooperative adjustment force is simultaneously mapped to the joint space of the left arm in the positive direction through the transpose of the Jacobian matrix of the left arm and to the joint space of the right arm in the negative direction through the transpose of the Jacobian matrix of the right arm, forming a pair of joint torques with equal size and opposite directions, which realizes rapid elimination of the internal force of the two arms.

[0045] The null space control item completes joint pose optimization, velocity damping and joint limit protection without interfering with the main task. First, a null space projection matrix is calculated, which is equal to the unit matrix minus the product of the transpose of the weighted generalized inverse of the combined Jacobian of the left and right arms and the Jacobian itself; then three parts of secondary torques are calculated: The first part multiplies the difference between the current joint angle and the preset intermediate joint angle by a diagonal pose gain to drive the joint to converge to a comfortable pose; The second part multiplies the current joint velocity by a diagonal damping coefficient to suppress unnecessary high-speed motion; The third part generates a repulsive torque when the remaining angle of any joint and the mechanical limit is less than 8 degrees, and the repulsive torque is inversely proportional to the square of the remaining angle, and saturates to a maximum value when the remaining angle is less than 3 degrees; The sum of the three parts of the secondary torque is multiplied by the null space projection matrix to obtain the final null space control item.

[0046] The final joint control torque is the sum of the main task control item, the cooperative control item, the null space control item, and the gravity compensation, the Coriolis force and centrifugal force compensation, and the friction force compensation calculated based on the dynamics model of the dual-arm, which is directly output to each joint torque servo driver.

[0047] Further, the multi-objective reward function used in the feedback and adaptive update is composed of the following multiple sub-reward items: The task completion degree reward sub-item, the two-arm cooperation reward sub-item, the force control precision reward sub-item, the safety reward sub-item, and the execution efficiency reward sub-item, the weight coefficients of each sub-reward item remain fixed or are dynamically adjusted according to the task stage in the training process.

[0048] Specifically, the task completion degree reward sub-item is equal to the negative square of the Euclidean distance between the actual pose of the left arm end and the expected pose, plus the negative square of the Euclidean distance between the actual pose of the right arm end and the expected pose, plus the negative square of the difference between the actual attitude of the operated object and the overall expected motion trajectory attitude of the object.

[0049] The two-arm cooperation reward sub-item is equal to the negative square of the module length of the vector sum of the actual six-dimensional contact force of the left arm and the actual six-dimensional contact force of the right arm, which directly punishes the internal force of the two arms.

[0050] The force control precision reward sub-item is equal to the negative square of the module length of the difference between the actual contact force of the left arm and the expected contact force of the left arm, plus the negative square of the module length of the difference between the actual contact force of the right arm and the expected contact force of the right arm.

[0051] The safety reward sub-item consists of two parts: The first part is a negative punishment generated when the closest distance between all joint angles and the respective mechanical limit is less than 10 degrees, and the punishment value is inversely proportional to the square of the distance; The second part is a negative punishment generated when the condition number of any arm Jacobian matrix is greater than 20, and the punishment value is proportional to the value after the condition number is reduced by 20, which is used to avoid configurations close to singularity.

[0052] The execution efficiency reward sub-item is equal to the negative value of the sum of the squares of all joint torques of the left and right arms in the current control period, which is used to suppress unnecessary energy consumption.

[0053] The final instantaneous reward is equal to the sum of the above five sub-reward items multiplied by the corresponding weight coefficients w1, w2, w3, w4, and w5, where w1 is 1.0, w2 is 0.8, w3 is 0.5, w4 is 2.0, and w5 is 0.1.

[0054] Further, the task space target output by the high-level task strategy learning includes the expected end pose of the left arm, the expected end pose of the right arm, the expected contact force of the left arm, the expected contact force of the right arm, and the overall expected motion trajectory of the object, and the task space target is converted into the input reference signal of the bottom layer control through an adjustable gain coefficient.

[0055] The system state vector specifically includes: the left arm end pose, the right arm end pose, the left arm contact force, the right arm contact force, the operated object attitude, the operated object velocity, and the external disturbance estimation value, all state quantities are normalized before inputting the high-level task strategy learning step.

[0056] Specifically, the task space target output by the high-level task policy learning step finally includes the left arm desired end six-dimensional pose, the right arm desired end six-dimensional pose, the left arm desired six-dimensional contact force, the right arm desired six-dimensional contact force, and the whole object desired six-dimensional motion trajectory.

[0057] The 24-dimensional original action vector output by the policy network is directly scaled by a fixed diagonal gain matrix to obtain the above five target parts, wherein the object whole desired motion trajectory takes the average of the left arm desired pose and the right arm desired pose, the diagonal elements of the gain matrix are fixed after training, and all target quantities are ensured to be within the physical executable range.

[0058] The system state vector is composed of 68-dimensional data spliced by the left arm current end six-dimensional pose, the left arm current end six-dimensional velocity, the left arm current six-dimensional contact force, the right arm current end six-dimensional pose, the right arm current end six-dimensional velocity, the right arm current six-dimensional contact force, the current six-dimensional attitude of the object to be operated, the current six-dimensional linear velocity and angular velocity of the object, and the six-dimensional disturbance estimation value. All 68-dimensional state quantities are uniformly normalized to zero mean and unit variance before inputting into the high-level task policy learning step, and the normalization statistical parameters are calculated once during the training stage and are fixed for use during deployment.

[0059] The left arm desired end pose, the right arm desired end pose, the left arm desired contact force, the right arm desired contact force, and the object whole desired motion trajectory are directly sent to the bottom layer dynamics controller through the real-time Ethernet interface, and are used as reference input signals for multi-target collaborative dynamics control law calculation without additional transformation.

[0060] Further, when the method is applied to an assembly task, the high-level task policy learning preferentially learns the relative pose coordination strategy of the two arms, and the collaborative control term in the bottom layer multi-target collaborative dynamics control preferentially ensures the force balance in the insertion direction. When the method is applied to a collaborative carrying task, the high-level task policy learning preferentially learns the common holding strategy of the two arms on the object attitude, and the collaborative control term in the bottom layer multi-target collaborative dynamics control preferentially eliminates the internal force circulation between the two arms.

[0061] The hierarchical control architecture achieves the decoupling of task planning and physical execution through the explicit one-way task target interface from high layer to bottom layer and the state and reward feedback interface from bottom layer to high layer.

[0062] Specifically, when applied to the precise shaft-hole assembly task, the high-level task strategy learning sets the weight coefficients of the assembly direction relative position error and the relative attitude error to the maximum, so that the strategy network preferentially outputs the expected trajectory that keeps the relative poses of the two-arm ends highly consistent, while automatically learning the coordination strategy of gradually reducing the forward speed and fine-tuning the relative attitude in the contact stage; the coordination control term in the bottom-layer multi-target collaborative dynamics control dynamically increases the component of the coordination gain matrix in the insertion axis direction to 3 times of the regular value, while keeping the regular value in the other five degrees of freedom directions, so as to preferentially ensure the force balance in the insertion direction.

[0063] When applied to the collaborative handling task of a large-mass object, the high-level task strategy learning sets the weight coefficient of the overall object attitude error to the maximum, so that the strategy network preferentially outputs the expected pose that keeps the object attitude unchanged by the left arm and the right arm together, while learning the strategy of automatically distributing the expected contact force of the two arms according to the object centroid offset; the coordination control term in the bottom-layer multi-target collaborative dynamics control uniformly increases the coordination gain matrix in the three force directions and the three moment directions to 2 times of the regular value, and explicitly sets the expected internal force to a zero vector, so as to preferentially eliminate the internal force circulation between the two arms.

[0064] The hierarchical control architecture only transmits five groups of data, i.e., the left-arm expected end pose, the right-arm expected end pose, the left-arm expected contact force, the right-arm expected contact force, and the overall object expected motion trajectory, through the one-way task target interface from the high level to the bottom level, and only feeds back the normalized 68-dimensional system state vector and the instant reward scalar from the bottom level to the bottom level, so as to realize the complete decoupling of task planning and physical execution, and run the two in different computing nodes independently through the real-time Ethernet.

[0065] On the basis of the above-mentioned embodiment one, the application further provides another more specific implementation mode, which details the internal structure and working principle of the hierarchical control architecture, the high-level task strategy learning module, the bottom-layer multi-target collaborative dynamics control law, and the feedback updating mechanism, so that those skilled in the art can more clearly understand the technical details of the application and implement them.

[0066] Embodiment two Please refer to Figs. 1-3 In another preferred embodiment of the application, the specific hierarchical architecture and implementation details of each module of the dual-arm collaborative operation control method based on intelligent coordination control are as follows: Hierarchical control architecture: The whole control system adopts a double-layer hierarchical architecture of high-level learning and low-level execution.

[0067] The high-level strategy layer is responsible for task target generation and strategy learning, and the bottom-layer dynamics layer is responsible for trajectory tracking, force control, and attitude coordination. The decoupling of task targets and execution signals is realized through the interlayer interface: From top to bottom: ; in, High-level initial policy network; High-level reinforcement learning strategy networks; : The system state vector at the current moment; : Task space target generator; : Expected task space pose; : Expected task space speed; From bottom to top: ; in, : The actual task space pose at the current moment.

[0068] : The actual task space velocity at the current moment.

[0069] : The final calculated joint torque.

[0070] This hierarchical design makes the high-level strategy unaffected by the complexity of the dynamic equations, improving policy transferability and learning efficiency, while the low-level control can independently optimize execution accuracy and system stability.

[0071] The technical solution of the present invention includes the following structural design: Step 1: System Status Acquisition and Initialization Data such as joint angles, end-effector pose, contact force, object orientation, and external disturbances of the dual-arm robotic arms are collected to construct the system state vector. ,in, The left and right arm distal positions, For their respective contact forces, These represent the attitude and velocity of the object being manipulated. Control parameters are set during the system initialization phase. and the initial weights of high-level reinforcement learning networks .

[0072] Step Two: High-Level Task Strategy Learning Module The high-level module learns the cooperative strategy between the two arms in the task space using reinforcement learning algorithms. This module consists of three parts: 1. State Coding and Feature Extraction Layer For input state Normalization and feature extraction are performed to form a left arm feature vector and a right arm feature vector . .

[0073] 2. Multi-head collaborative attention mechanism for double arms In order to enhance the double-arm collaborative perception ability of high-level strategy, the multi-head collaborative attention mechanism is embedded in the reinforcement learning network to realize cross-arm information fusion. The specific implementation is as follows: Input left and right arm state feature vectors Each attention head calculates the cross-dependent relationship: ; Wherein, : the attention weight vector of the mth attention head; : the state feature vector of the left arm at the current moment; : the state feature vector of the right arm at the current moment; : the query linear transformation matrix of the mth attention head; : the key linear transformation matrix of the mth attention head; : the dimension of the feature vector; : the dot product result is row normalized to obtain the attention weight; Fusion output result: ; Wherein, : the output feature vector of the mth attention head; : the attention weight vector of the mth head; : the value linear transformation matrix of the mth attention head; After the multi-head output is spliced, the global feature is generated through linear mapping: ; Wherein, : the global feature vector finally generated containing double-arm collaborative information; : the multi-head attention output linear transformation matrix; : The concatenation of all M attention head output feature vectors; This mechanism can capture the cross-arm spatial dependence and force coordination relationship, so that the policy network considers the dynamic state of both arms when making decisions, achieving task consistency and force coordination.

[0074] 3. Hybrid reinforcement learning strategy optimization The PPO+SAC combined structure is adopted, the SAC part provides a high-entropy policy, increases the exploratory nature in the early stage, and maximizes the soft value function Improve sample utilization. The PPO part constrains the policy change rate in the later stage to ensure the smooth convergence of the policy gradient, and together realize the balance of policy stability and exploratory nature. The loss functions of the two branches are dynamically weighted and fused: ; Among them, The training stage is self-adaptive, which meets . Early stage biased towards SAC, later stage biased towards PPO.

[0075] The task space target of the high-level output is:

[0076] Among them, is the action vector output by the policy network, , is the gain coefficient.

[0077] Step three: multi-objective cooperative dynamics control law The bottom control module adopts a multi-objective cooperative dynamics control law to calculate the control torque according to the expected target output by the high-level, achieving trajectory tracking, force balance and attitude stability.

[0078] Control law definition: ; Among them, : The final output joint control torque vector.

[0079] : Main task control item; : Cooperative control item; : Null space control item; : Inertial torque item; : Coriolis and centrifugal force item; : Gravity compensation item; (1) Task control item This term essentially establishes the mapping relationship between the task space error and the expected control torque, enabling the system to maintain compliant tracking of the trajectory under external disturbances or nonlinear effects.

[0080] wherein, : transpose of the manipulator Jacobian matrix; : position / posture stiffness gain diagonal matrix; : high-level issued task space desired end pose; : current actual task space end pose; : damping gain diagonal matrix; : desired task space velocity; : actual task space velocity; (2) Cooperative control term

[0081] wherein, represents the actual force difference between the two arms, is the desired force difference (usually set to zero or a constant), is the adaptive gain coefficient. This control term adjusts the force balance between the two arms; (3) Null space control term is used to achieve posture optimization, obstacle avoidance and joint limit protection without affecting the execution of the main task, wherein, : null space projection matrix; : internal secondary control torque.

[0082] Step four: feedback and adaptive update module The system collects state feedback and reward signals after execution, through a multi-objective reward function: ; wherein, is the weight coefficient of each sub-reward term, used to balance the importance of different behavior goals, including task completion reward , dual-arm cooperativity reward , force control reward , safety reward and efficiency reward , etc.

[0083] Finally, it should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art will appreciate that the technical solutions described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalent features, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for controlling coordinated operation of dual arms based on intelligent coordinated control, characterized in that, The method adopts a hierarchical control architecture of high-level task policy learning and low-level multi-objective collaborative dynamics control, and comprises the following steps: S1, system state acquisition and initialization, acquiring joint angle, joint speed, end pose, contact force, object pose and external disturbance data of the dual-arm robot, constructing a system state vector and completing initialization of control parameters and a high-level reinforcement learning network; S2, high-level task policy learning, generating a task space target through a hybrid reinforcement learning policy network embedded with a dual-arm multi-head collaborative attention mechanism according to the system state vector, wherein the hybrid reinforcement learning policy network is trained in a dynamic weighted fusion manner of PPO algorithm and SAC algorithm; S3, low-level multi-objective collaborative dynamics control, calculating joint control torque by using a multi-objective collaborative dynamics control law containing a main task control item, a collaborative control item and a null space control item according to the task space target output by the high-level task policy learning step; S4, feedback and adaptive update, acquiring a reward signal according to the actual state after execution, and updating the hybrid reinforcement learning policy network in the high-level task policy learning step online or offline through a multi-objective reward function to form a closed-loop adaptive control. 2.The dual-arm collaborative operation control method based on intelligent coordinated control according to claim 1, wherein The specific process of the dual-arm multi-head collaborative attention mechanism embedded in the high-level task policy learning is as follows: First, feature extraction is performed on the left arm state feature vector and the right arm state feature vector respectively; Then, the cross-dependent relationship between the left arm features and the right arm features is calculated through multiple attention heads respectively; Subsequently, the outputs of all attention heads are fused and spliced; Finally, a global feature vector containing dual-arm collaborative information is generated through linear mapping. 3.The dual-arm collaborative operation control method based on intelligent coordinated control according to claim 1, wherein, The specific process of training the hybrid reinforcement learning policy network in a dynamic weighted fusion manner of PPO algorithm and SAC algorithm is as follows: In the early stage of training, the loss weight of the SAC algorithm branch is increased to enhance the exploration ability, and in the later stage of training, the loss weight of the PPO algorithm branch is increased to ensure the smoothness of policy update, and the loss functions of the two algorithm branches are weighted and summed through a dynamic weight coefficient that changes adaptively with the training round number to optimize the same policy network parameters.

4. The dual-arm collaborative operation control method based on intelligent coordinated control according to claim 1, characterized in that, The multi-objective collaborative dynamics control law used in the low-level multi-objective collaborative dynamics control comprises the following three specific control items: The main task control item is used to generate the corresponding expected control torque according to the task space position and force error; The collaborative control item is used to adjust the force balance between the left arm and the right arm according to the deviation of the actual contact force difference and the expected force difference between the left arm and the right arm and the adaptive gain coefficient; The null space control item is used to achieve joint pose optimization, obstacle avoidance and joint limit protection without affecting the execution of the main task.

5. The dual-arm collaborative operation control method based on intelligent coordinated control according to claim 1, characterized in that, The multi-objective reward function used in the feedback and adaptive update is composed of the following multiple sub-reward items: Task completion degree reward sub-item, dual-arm collaboration reward sub-item, force control accuracy reward sub-item, safety reward sub-item and execution efficiency reward sub-item, and the weight coefficients of each sub-reward item remain fixed or are dynamically adjusted according to the task stage during the training process.

6. The dual-arm collaborative operation control method based on intelligent coordinated control according to claim 1, characterized in that, The high-level task policy learning outputs a task space target including a left arm desired end pose, a right arm desired end pose, a left arm desired contact force, a right arm desired contact force, and a whole object desired motion trajectory, and the task space target is converted into an input reference signal of the low-level control through an adjustable gain coefficient.

7. The dual-arm collaborative operation control method based on intelligent coordinated control according to claim 1, characterized in that, The system state vector specifically includes a left arm end pose, a right arm end pose, a left arm contact force, a right arm contact force, an operated object pose, an operated object velocity, and an external disturbance estimation value, and all state quantities are normalized before inputting a high-level task policy learning step. 8.The dual-arm collaborative operation control method based on intelligent coordinated control of claim 1, wherein, When the method is applied to an assembly task, the high-level task policy learning preferentially learns a double-arm relative pose coordination strategy, and a collaborative control term in the low-level multi-target collaborative dynamics control preferentially ensures force balance in an insertion force direction. 9.The dual-arm collaborative operation control method based on intelligent coordinated control of claim 1, wherein, When the method is applied to a collaborative carrying task, the high-level task policy learning preferentially learns a double-arm common object pose holding strategy, and the collaborative control term in the low-level multi-target collaborative dynamics control preferentially eliminates an internal force circulation between the double arms. 10.The dual-arm collaborative operation control method based on intelligent coordinated control according to claim 1, wherein, The hierarchical control architecture achieves decoupling of task planning and physical execution through an explicit high-level to low-level one-way task target interface and a low-level to high-level state and reward feedback interface.

Images