A dual-arm visual language action control method based on reinforcement learning correction

Abstract

This invention discloses a dual-arm visual-language action control method based on reinforcement learning correction, comprising: acquiring multimodal input information required for a dual-arm robot to perform a task; uniformly encoding the multimodal input information to obtain a fused feature vector and a unified state representation; performing task semantic parsing on natural language task instructions and dynamically assigning dual-arm roles; based on the fused feature vector and the results of the dynamic assignment of dual-arm roles, calling a pre-trained visual-language-action candidate action sequence generation model to output a dual-arm candidate action sequence; constructing a cross-arm constraint consistency mathematical model for dual-arm operations and performing joint evaluation of the cross-arm constraint consistency of the dual-arm candidate action sequences; constructing a dual-arm residual reinforcement learning policy network to enhance the selected action sequences that pass the joint evaluation; performing uncertainty evaluation on the current enhanced action; and sending the enhanced action that passes the evaluation to the dual-arm robot actuator and obtaining real-time feedback on the execution results.

Description

Technical Field

[0001] This invention relates to the field of dual-arm robot control technology, specifically a dual-arm visual language action control method based on reinforcement learning correction. Background Technology

[0002] As robotics technology expands from single-structured workstations to open, unstructured, and multi-tasking environments, dual-arm robots are gradually becoming key execution agents in complex operational scenarios. Compared to single-arm robots, dual-arm robots can accomplish tasks that a single arm cannot perform independently through the collaborative relationship between their left and right robotic arms. Examples include jointly lifting large objects, performing delicate operations after fixing objects, aligning and assembling components, deploying flexible materials, and simultaneously manipulating tools and manipulated objects. Therefore, the control problem for dual-arm robots is not merely a matter of increasing the number of robotic arms, but also a matter of comprehensively increasing the complexity of task structure, motion coupling relationships, and control mechanisms.

[0003] Existing dual-arm robot control methods can be broadly categorized into rule-based modeling methods, teaching-based learning methods, reinforcement learning methods, and multimodal large model-based methods.

[0004] The first category comprises rule-based modeling and motion planning methods. These methods typically set motion trajectories, grasping postures, and temporal logic for the left and right arms, given a known environment model, object structure, and task flow. This type of method has advantages in scenarios with clear environmental structures, fixed tasks, and predictable object forms, and it offers strong interpretability. However, this method relies heavily on manual modeling and expert experience. When faced with new objects, new tasks, or complex interaction scenarios, its rules are often difficult to transfer quickly, resulting in insufficient system flexibility. Especially in dual-arm operations, the master-slave relationship, force relationship, and temporal relationship between the left and right arms dynamically change with the task stage. If fixed rules are still used to describe these relationships, problems such as cumbersome planning, poor adaptability, and high maintenance costs can easily arise.

[0005] The second category comprises methods based on instruction-based learning or imitation learning. These methods learn the collaborative movement patterns of both arms from expert trajectories by collecting demonstrations of human operation. Compared to rule-based methods, instruction-based learning reduces the manual design work of complex control strategies and is suitable for tasks with high repetition and relatively stable movement patterns. However, this type of method heavily relies on high-quality instruction data, and the instruction cost is high with limited coverage. Once the position of the task object shifts, its appearance changes, or the environmental state changes, the strategy obtained solely from instruction data often struggles to generalize stably. Furthermore, the role switching, synchronization relationships, and contact control in dual-arm operations are often implicit in the demonstration; the model may not be able to explicitly learn the true collaborative mechanism between the two arms, leading to problems such as unstable execution, movement conflicts, or accumulated failures in complex new scenarios.

[0006] The third category is reinforcement learning-based methods. Reinforcement learning can optimize strategies through trial and error, giving it a natural advantage in complex continuous motion control problems. Compared to pure teaching methods, reinforcement learning can continuously correct actions using task rewards, theoretically possessing stronger adaptive capabilities. However, in bi-arm manipulation tasks, reinforcement learning faces challenges such as a large state space, high action dimensions, sparse rewards and punishments, unstable training, and difficulty in ensuring safety. Especially in bi-arm systems, the left and right arms are not two independent control objects; their relative pose, mechanical coupling, contact state, temporal synchronization, and object stability all affect the task outcome. If only ordinary single-agent or multi-agent reinforcement learning methods are used without explicit design of the division of labor and cooperative constraints between the two arms, it is easy for one robotic arm to dominate the action while the other arm follows ineffectively, or for the two robotic arms to interfere with each other, ultimately leading to low training efficiency and insufficient task success rate.

[0007] The fourth category comprises methods based on vision-language-action models. These methods integrate visual input, verbal commands, and motion trajectories into a unified model, enabling robots to directly generate motion control signals based on natural language tasks. This approach significantly improves the robot's semantic understanding and generalization capabilities for open tasks, demonstrating strong potential, particularly in the unified modeling of "what to see, what to understand, and what to execute." However, most existing vision-language-action methods focus on single-arm tasks or general end-to-end motion generation, neglecting the common challenges in dual-arm systems such as task role division, cross-arm constraint representation, fine-grained motion correction, and execution safety arbitration. Dual-arm tasks exhibit stronger multimodal coupling and greater motion diversity compared to single-arm tasks. While directly treating dual-arm motions as simple, high-dimensional motion outputs can achieve motion generation to some extent, it often lacks explicit modeling of "why the arms are divided in this way, when to switch roles, and how the two arms jointly satisfy object stability and contact constraints," thus affecting the stability and success rate of actual execution. Summary of the Invention

[0008] This invention provides a dual-arm visual language action control method based on reinforcement learning correction, which overcomes the problems in the control of dual-arm robots in the prior art, such as the difficulty in directly mapping natural language tasks to dual-arm collaborative actions, unclear division of roles between the two arms, difficulty in uniformly modeling cross-arm constraints, insufficient action control precision, and poor execution stability and insufficient safety in complex environments. It can significantly improve the task understanding ability, collaborative control ability, action precision and execution robustness of dual-arm robots in open environments, multi-task and multi-object conditions.

[0009] This invention provides a dual-arm visual-language action control method based on reinforcement learning correction, comprising:

[0010] Step 1: Obtain the multimodal input information required for the dual-arm robot to perform the task, encode the multimodal input information in a unified manner to obtain the fused feature vector, and construct a unified state representation for dual-arm cooperative control;

[0011] Step 2: Perform task semantic parsing on the natural language task instructions and dynamically assign roles to both arms;

[0012] Step 3: Based on the fused feature vector and the dynamic allocation result of the two-arm role, call the pre-trained vision-language-action candidate action sequence generation model to output the two-arm candidate action sequence that meets the semantic constraints of the task;

[0013] Step 4: Construct a mathematical model for cross-arm constraint consistency for dual-arm operation, and perform a joint evaluation of cross-arm constraint consistency for dual-arm candidate action sequences;

[0014] Step 5: Construct a two-arm residual reinforcement learning policy network to enhance the action sequence selected through joint evaluation of cross-arm constraint consistency;

[0015] Step 6: Perform an uncertainty assessment on the current enhancement action;

[0016] Step 7: Send the enhanced actions that have passed the evaluation to the dual-arm robot actuator for control execution, and obtain the execution result feedback in real time.

[0017] The present invention provides a dual-arm visual-language action control method based on reinforcement learning correction, which has the following beneficial effects:

[0018] (1) This invention encodes environmental visual observation information, dual-arm ontological state information, natural language task instructions and historical execution context information in a unified manner, and on this basis, performs task-object-role joint decomposition on task instructions, so that natural language tasks are no longer just parsed as abstract action sequences, but are further parsed as staged sub-goals and role-based task structures for dual-arm execution, thereby improving the mapping ability between natural language tasks and dual-arm collaborative actions, and enhancing the system's ability to understand complex tasks under open scenes, multi-task and multi-object conditions.

[0019] (2) This invention introduces a dynamic role allocation mechanism for the two arms. Based on the current sub-target requirements, the position and posture of the target object, the reachability of the left and right arms, the degree of end-effector posture matching, obstacle distribution, and historical execution results, the left and right arms are assessed for adaptation under multiple candidate role combinations, and the optimal role configuration is selected. This allows the two arms to dynamically switch between different roles such as main operating arm, auxiliary support arm, fixed arm, gripping arm, alignment arm, propulsion arm, or collaborative transport arm at different task stages. By adopting this mechanism, the problem of rigid collaboration caused by long-term fixed division of labor or simple symmetrical control of the two arms in the prior art is overcome, and the rationality of the division of labor and the efficiency of collaboration of the two arms in complex task processes are improved.

[0020] (3) This invention generates candidate action sequences for both arms using a vision-language-action prior model, and combines this with a cross-arm constraint consistency model to jointly evaluate the relative pose of the two arm ends, the synchronization of action time, the stability of the target object's posture, contact coordination, and collision safety. This makes the actions of both arms no longer a simple splicing of independent actions of the left and right robotic arms, but a holistic candidate action sequence generated and screened under unified task semantics and dual-arm collaborative constraints. As a result, it can effectively improve the ability of objects to remain stable, the timing of actions to be coordinated, and the contact relationship to be reasonable during dual-arm collaborative operation, and reduce the risks of dual-arm action conflicts, target slippage, and collisions.

[0021] (4) After the candidate action passes the cross-arm constraint consistency evaluation, the present invention further employs a dual-arm residual reinforcement learning strategy to continuously correct the candidate action. By outputting the left arm correction amount, right arm correction amount, and cross-arm coupling correction amount, fine-grained corrections are made for grasping deviation, alignment error, insertion angle deviation, synchronization error, and contact instability. By adopting this approach, the advantages of the vision-language-action prior model in task understanding and action generalization are retained, while environmental feedback is used to improve the control accuracy of dual-arm actions. This improves the problem of large control error and insufficient execution success rate caused by relying solely on candidate action sequence generation in the prior art.

[0022] (5) This invention establishes an uncertainty-aware safety arbitration mechanism to comprehensively evaluate semantic matching confidence, action value function stability, target object state deviation, contact risk, and double-arm collision risk. When uncertainty is low, the action is executed directly; when uncertainty is in the intermediate range, a conservative execution mode is switched; and when uncertainty is high, role reassignment, sub-target replanning, or safe rollback is triggered. This mechanism improves the system's safety and robustness in complex real-world environments, preventing high-risk actions from being executed directly, thereby reducing the occurrence of double-arm collisions, target slippage, and mission failure. Attached Figure Description

[0023] Figure 1This is a flowchart of a dual-arm visual language action control method based on reinforcement learning correction according to the present invention. Detailed Implementation

[0024] like Figure 1 As shown, the present invention provides a dual-arm visual-language action control method based on reinforcement learning correction, comprising:

[0025] Step 1: Obtain the multimodal input information required for the dual-arm robot to perform the task, encode the multimodal input information in a unified manner to obtain the fused feature vector, and construct a unified state representation for dual-arm cooperative control.

[0026] The multimodal input information includes: environmental visual observation information. Information on the status of both arms Historical execution context information and natural language task instructions .

[0027] The above multimodal input can be represented as:

[0028]

[0029] Environmental visual observation information includes: image information of the target object, spatial positional relationships, distribution of obstacles in the scene, and structural information of the operating area; dual-arm body state information includes: joint states of the left and right arms, end effector pose, gripper state, and motion state information; historical execution context information includes: previous action, completed sub-objectives, execution success or failure markers, and environmental change information; natural language task instructions are used to describe the operational objectives and task constraints that the dual-arm robot needs to perform.

[0030] In step 1, the multimodal input information is uniformly encoded to obtain a fused feature vector, and a unified state representation for dual-arm cooperative control is constructed, specifically as follows:

[0031] Using a visual encoder visual observation information Extracting visual semantic features Use a language encoder Instructions for natural language tasks Extracting semantic features of the task Employing a body state encoder Information on the state of the two arms Extracting motion state features A timing encoder is used. Historical execution context information Extracting temporal correlation features Its expression is:

[0032]

[0033]

[0034]

[0035]

[0036] Among them, the timing encoder Sequence encoding is performed on the actions of the previous moment, completed sub-objectives, success or failure markers, and environmental change information to extract temporal correlation features that reflect the progress of task execution and the impact of historical actions.

[0037] The extracted features are then fused to obtain a fused feature vector. :

[0038]

[0039] in, The network represents a feature fusion network, and [ ] represents feature concatenation or joint mapping operations; the unified state is represented as:

[0040]

[0041] The unified state representation Used for subsequent task decomposition, role assignment, candidate action generation, reinforcement learning correction, and arbitration.

[0042] This step enables the system to simultaneously possess environmental understanding, task understanding, dual-arm state awareness, and temporal context modeling capabilities, providing basic inputs for subsequent control processes.

[0043] Step 2: Perform task semantic parsing on the natural language task instructions and dynamically assign roles to both arms, specifically:

[0044] Step 2.1: Obtaining a unified state representation Then, the natural language task instructions are subjected to task semantic parsing, decomposing the original task into multiple staged sub-goals with execution order:

[0045] The phased sub-target sequence is represented as follows:

[0046]

[0047] in, Indicates the first Each stage-specific sub-goal; represented as:

[0048]

[0049] in, Indicates the semantics of stage actions. This represents target object information composed of target object image information and target object pose information. This represents the constraint information of the two arms working together; thus constructing a structured representation of "task sub-goal - target object - two arm roles".

[0050] For example, the task of "using one hand to hold a container and the other hand to open the lid and remove the contents" can be broken down into multiple stages, such as approaching the target, holding with one arm, opening with one arm, deploying with both arms in coordination, grasping with one arm, and withdrawing with both arms. This approach not only completes the semantic decomposition of the task but also simultaneously clarifies the target object, the division of labor between the two arms, and the coordination constraints for each stage.

[0051] Step 2.2: Based on the phased sub-goals, and combined with the current target object position and posture, left and right arm reachability, gripper posture matching degree, obstacle spatial distribution, and historical execution status collected by the sensors, the two arms are dynamically matched and scored under each candidate role combination to obtain the role allocation result for the current phase.

[0052] Step 2.2.1: Based on the current stage sub-goals, target object information, and dual-arm collaborative constraint information, construct a candidate role configuration set; the candidate roles include at least one or more of the following: main operating arm, auxiliary support arm, fixed arm, execution arm, gripping arm, alignment arm, and collaborative transport arm.

[0053] Step 2.2.2: Eliminate candidate configurations that do not meet the constraints of left and right arm reachability, gripper posture matching, obstacle safety distance and task role, and obtain a feasible candidate set.

[0054] Step 2.2.3: Calculate the role allocation adaptation index for each candidate role in the feasible candidate set. , is represented as:

[0055]

[0056] Among them, A( , R( represents the fit between the candidate role configuration and the action of the stage sub-objective). , () indicates that the left and right arms are in a unified state. Accessibility and ontological state fit under the given conditions, M( , C( ) represents the degree of matching between the gripper posture and the target object. , O() represents the degree to which the candidate role configuration satisfies the dual-arm collaboration constraint. () indicates the cost of obstacles and collision risks. to Preset weights.

[0057] Step 2.2.4: Calculate the RASI value for all candidate role configurations, sort them from highest to lowest score, and select the candidate role configuration that satisfies the constraints and has the highest score as the role allocation result for the current stage.

[0058]

[0059] Step 2.2.5: If none of the candidate role configurations meet the constraints, trigger the sub-goal replanning or safe rollback operation.

[0060] Furthermore, by combining the historical successful role allocation results stored in the experience memory module, the initial ranking of candidate role configurations can be prioritized to improve search efficiency and role allocation stability. Through this dynamic role allocation strategy, the system can adaptively adjust the responsibilities of the left and right arms according to changes in task stages and environmental disturbances, thereby solving the problems of rigid responsibilities and inflexible collaboration between the two arms in complex tasks.

[0061] Step 3: Based on the fused feature vectors and the dynamic allocation results of the two-arm roles, call the pre-trained vision-language-action candidate action sequence generation model to output a two-arm candidate action sequence that satisfies the task semantic constraints, specifically:

[0062] Step 3.1: Pre-train or fine-tune the pre-training based on the existing robot operation dataset to obtain a vision-language-action candidate action sequence generation model; during the model training phase, a robot operation dataset including dual-arm visual observations, natural language task instructions, dual-arm states, target action sequences, and task success markers is used for training; the training objective is to ensure that the candidate action sequences output by the model are consistent with the target action sequences in the samples and satisfy the task semantics and role constraints, and its training loss is expressed as:

[0063]

[0064] in, Represents a candidate action sequence. This represents the sequence of exemplary actions in the dataset. This represents the task semantic consistency loss. This represents the loss of consistency due to role constraints. and These are the weight parameters.

[0065] The existing datasets include one or more of the following: Open X-Embodiment dataset, DROID dataset, BridgeData V2 dataset, RH20T dataset, RoboSet dataset, and CALVIN linguistic conditional robot operation dataset.

[0066] The Open X-Embodiment dataset contains various robot bodies and large-scale robot maneuver trajectories, suitable for pre-training vision-language-action models across robot bodies. The DROID, BridgeData V2, RH20T, and RoboSet datasets contain real robot maneuver demonstration data, which can be used to enhance the model's ability to generate actions in real-world scenarios. The CALVIN dataset contains long-term robot maneuver tasks under language conditions, which can be used to improve the model's ability to understand phased task instructions.

[0067] Step 3.2: Fuse the feature vectors Phased sub-goals And role allocation results The input is a vision-language-action candidate action sequence generation model, which outputs candidate action sequences for the left and right arms at the current stage. Its expression is:

[0068]

[0069] in, This represents a visual-language-action candidate action sequence generation model. Indicates the first Candidate arm movement sequences for the phase.

[0070] Role allocation results during the above candidate action sequence process The input as an action generation condition is used to the candidate action sequence generation model. In the process, the action generation conditions include at least the left arm role label, the right arm role label, the current stage sub-target, the target object information, and the dual-arm coordination constraint information, which are used to limit the types of actions that the prior model generates for the left and right arms respectively in the current stage; the candidate action sequence generation model According to phased sub-goals The execution order generates a sequence of candidate actions in stages.

[0071] If the current role assignment result is "left arm fixed, right arm operating", then in the action generation conditions, the left arm role label is fixed arm, and the right arm role label is execution arm. The prior model prioritizes generating left arm stable support actions and right arm fine operation actions. If the current role assignment result is "two arms cooperative carrying" or "two arms jointly unfolding", then in the action generation conditions, both the left and right arms are marked as cooperative execution roles. The prior model prioritizes generating candidate action sequences with spatial symmetry and temporal synchronization. By introducing role constraints, the system-generated two-arm actions are no longer a simple superposition of two single-arm actions, but rather a task-oriented candidate action sequence with a clear cooperative structure.

[0072] The candidate action sequence includes at least the changes in the end-arm pose, the increments in joint motion, the changes in the gripper state, and the action synchronization parameters; expressed as:

[0073]

[0074] in, and These represent the changes in pose at the distal ends of the left and right arms, respectively. and These represent the increments in joint movement for the left and right arms, respectively. and These represent the changes in the gripper states of the left and right arms, respectively. This indicates the synchronization parameters for both arm movements.

[0075] Step 4: Construct a mathematical model for cross-arm constraint consistency in dual-arm manipulation, and perform a joint evaluation of cross-arm constraint consistency for candidate dual-arm action sequences, specifically:

[0076] Step 4.1: Construct a mathematical model for cross-arm constraint consistency for dual-arm operations, including at least the following constraint terms:

[0077] 1) Relative pose constraints at the end effectors of both arms are used to limit the relative spatial relationship required by the arms during grasping, unfolding, handling, and assembly; the relative pose error of the end effectors is expressed as:

[0078]

[0079] in, and These represent the distal positions of the left and right arms, respectively. This indicates the relative pose of the target.

[0080] 2) Motion timing synchronization constraints are used to ensure that key movements of both arms are executed synchronously or with a specified phase difference in time; the motion timing synchronization error is expressed as:

[0081]

[0082] in, and These represent the key moments of the left and right arm movements, respectively. This indicates the preset phase difference.

[0083] 3) Target object stability constraints are used to ensure that the attitude, position, and force state of the target object remain within allowable limits under the action of both arms; the target object stability error is expressed as:

[0084]

[0085] in, Indicates the current position of the target object. Indicates the desired position of the target object. Indicates the current pose of the target object. Indicates the desired pose of the target object. This indicates the current force state of the target object. This represents the desired force state of the target object, and μ represents the force error weight.

[0086] 4) Contact coordination constraints are used to avoid excessive force on one side, insufficient support, or conflicting contact methods; the contact coordination error is expressed as:

[0087]

[0088] in, and These represent the contact forces of the left and right arms, respectively. and η represents the contact state of the left and right arms, respectively, and η represents the contact state error weight.

[0089] 5) Collision safety constraints are used to limit the collision risk between the two arms and between the two arms and the environment; the collision risk error is expressed as:

[0090]

[0091] in, Indicates the safe distance threshold. This indicates the minimum distance between the arms or between the arms and the environment.

[0092] Step 4.2: Based on the above constraint consistency mathematical model and candidate action sequences, calculate the relative pose error of the two arm ends, the action timing synchronization error, the target object stability error, the contact coordination error, and the collision risk error, and calculate the joint evaluation cost function:

[0093]

[0094] in, to These are the weighting parameters for joint evaluation.

[0095] If the relative pose error of the two arm ends, the timing synchronization error of the action, the stability error of the target object, the contact coordination error, the collision risk error, and the joint evaluation cost function of the candidate action sequence are all less than the corresponding preset thresholds, it is determined that it meets the cross-arm constraint consistency requirements; otherwise, it is removed.

[0096] Step 5: Construct a two-arm residual reinforcement learning policy network to enhance the action sequence selected through joint evaluation of cross-arm constraint consistency, specifically as follows:

[0097] Step 5.1: During the execution of the current stage, represent the unified state of the current task. Role allocation results Environmental feedback information and the candidate action sequence generated in step 3 Input a two-arm residual reinforcement learning strategy network.

[0098] Step 5.2: The dual-arm residual reinforcement learning strategy network determines whether there are problems such as grasping bias, alignment error, synchronization error, contact instability, or posture drift in the candidate action sequence based on the current environmental feedback, and outputs the action residual correction amount. Its expression is:

[0099]

[0100] in, This represents a residual reinforcement learning policy network. This represents environmental feedback information consisting of visual feedback, force / contact feedback, arm status feedback, and task progress feedback.

[0101] The motion residual correction includes the output of the left arm motion correction through the left arm correction branch, the right arm correction branch, and the cross-arm coupling correction branch. Right arm movement correction amount And the correction amount ΔAc for the coupled action of both arms; the residual correction amount is superimposed on the candidate action sequence to obtain the enhanced action for the current stage:

[0102]

[0103] in, This represents the residual correction amount of the reinforcement learning policy network output. This indicates the enhanced action after residual correction.

[0104] Unlike learning arm movements directly from scratch, the reinforcement learning strategy in this step uses the candidate movement sequence generated in step 3 as input and only learns how to fine-tune and compensate for it, thereby reducing training difficulty and improving movement correction efficiency.

[0105] Step 5.3: When training the dual-arm residual reinforcement learning policy network, the reward function includes task completion rewards. Smooth motion reward Cross-arm collaborative rewards Safety constraints and rewards and performance-based rewards , is represented as:

[0106]

[0107] in, to The reward weights are used to describe how the dual-arm residual reinforcement learning strategy network learns how to perform residual compensation based on the candidate action sequence through the above reward function, so that the enhanced action achieves a balance between task completion, motion smoothness, dual-arm coordination, safety and execution efficiency.

[0108] Step 6: Perform an uncertainty assessment on the current enhancement action, specifically:

[0109] Step 6.1: Uncertainties include: uncertainty in task semantic matching, uncertainty in action value assessment, uncertainty in target object state deviation, uncertainty in contact risk, and uncertainty in the risk of conflict between the two arms.

[0110] Task semantic matching uncertainty is calculated based on the matching confidence between the language task and the augmented action, and is expressed as follows:

[0111]

[0112] in, This represents the confidence level of semantic matching between the language task and the augmentation action.

[0113] The uncertainty in action value assessment is calculated based on the variance of the reinforcement learning value network output, and is expressed as:

[0114]

[0115] in, This indicates that the reinforcement learning value network represents the current unified state. and enhance action The value estimate is represented by Var(·), which indicates the variance calculation.

[0116] The uncertainty of the target object's state deviation is calculated based on the deviation between the observed state and the predicted state of the target object, and is expressed as:

[0117]

[0118] in, and These represent the observation position and observation attitude of the target object, respectively. and These represent the predicted position and predicted orientation of the target object, respectively.

[0119] The uncertainty of contact risk is calculated based on the contact force deviation and the degree of abnormality in the gripper condition, and is expressed as follows:

[0120]

[0121] in, Indicates the current contact force. Indicates the desired contact force. Indicates an abnormal gripper status. This indicates the weight of abnormal gripper status.

[0122] The uncertainty of the risk of conflict between the two arms is calculated based on the minimum distance between the two arms and their relative motion trend, and is expressed as follows:

[0123]

[0124] in, Indicates the safe distance threshold. This indicates the minimum distance between the arms or between the arms and the environment. Indicates the speed at which the arms move away from or relative to an obstacle; when A value less than 0 indicates that the two tend to approach each other. This represents the relative motion risk weight.

[0125] By weighting and summing the above uncertainties, we obtain the overall uncertainty:

[0126]

[0127] in, to The weighting parameter is used; through the above formula, semantic matching, value estimation, target object status, contact risk and double-arm conflict risk are uniformly quantified into an overall uncertainty U, which is used for subsequent arbitration and safety rollback judgment.

[0128] Step 6.2: Based on the overall uncertainty U obtained in Step 6.1 and the evaluation results of each individual uncertainty, arbitrate control is applied to the execution process of the current enhancement action; a first threshold is set. Second threshold ,and < .

[0129] If U≤ If the uncertainty of each individual item does not exceed the corresponding safety threshold, then the reliability of the current enhancement action is judged to be high, and the enhancement action is executed directly.

[0130] like <U≤ If a single uncertainty is close to the corresponding safety threshold, the current enhanced action is judged to have a medium risk and enters a conservative execution mode, which includes reducing the step length of the action, reducing the execution speed, enhancing the support behavior of the auxiliary arm, increasing the frequency of visual feedback detection, or recalculating the local action correction amount.

[0131] If U> If any single uncertainty exceeds the corresponding safety threshold, the current enhancement action is judged to have a high risk, triggering a recalculation of the role allocation result, a replanning of the current sub-target, or a safety rollback operation.

[0132] Through this step, the system can classify and arbitrate enhanced actions based on overall uncertainty and individual risk indicators, enabling low-risk actions to be executed directly, medium-risk actions to be executed conservatively, and high-risk actions to be stopped or reversed, thereby improving the stability and safety of the dual-arm robot in complex environments.

[0133] Step 7: The enhanced actions that pass the evaluation are sent to the dual-arm robot actuator for control execution, and the execution results are obtained in real time. Specifically:

[0134] Step 7.1: The execution result feedback includes: visual feedback, dual-arm status feedback, and task status feedback.

[0135] Visual feedback parameters include: the current position of the target object. Current pose of the target object Target object appearance status Target object occlusion marker and the distance between the target object and the target area .

[0136] The bi-arm status feedback parameters include: the joint positions of the left and right arms. , End effector pose , Gripper open / closed state , terminal velocity , and contact status , .

[0137] Task status feedback parameters include: current sub-target number i, target area location. Capture stability indicators Support stability indicators Exception interruption flag and execution time .

[0138] Based on the above feedback parameters, construct a sub-target completion determination function:

[0139]

[0140] in, This represents environmental feedback information consisting of visual feedback, force / contact feedback, arm status feedback, and task progress feedback. Φ represents the sub-goal of the current stage, and Φ represents the sub-goal completion determination function.

[0141] Specifically, based on the current stage sub-goals The type selects the corresponding completion judgment condition. If the current sub-target is a near target, then when the distance between the target object and the end effector is less than a preset proximity threshold... When the target is close, it is determined that the task is nearing completion; if the current sub-target is the grasping target, then the gripper is in the open / closed state. or Achieving closure requirements and contact state or Effective and stable capture metrics Greater than the preset threshold When the target is successfully captured, determine if the capture is complete; if the current sub-target is a moving or placing target, then determine the target object's current position. Location of the target area The distance between them is less than the location threshold And the current pose of the target object. With expected posture The attitude error between them is less than the attitude threshold. When the target is moved or placed, it is determined that the task is complete; if the current sub-target is a fixed or supported target, then when the contact state is reached... or Effective and supportive stability indicators Greater than the preset threshold When the change in the pose of the target object is less than the stability threshold, it is determined that the fixed or supported target has been completed.

[0142] If the visual feedback condition, the dual-arm status feedback condition, and the task status feedback condition corresponding to the current sub-target are all satisfied... The current sub-objective has been determined to be completed, and the process moves on to the next sub-objective.

[0143] If any critical feedback condition is not met, or an abnormal interruption is detected... When =1, If the current sub-goal is not completed, return to the corresponding step for processing based on the reason for failure: if the reason for failure is related to the mismatch of the two-arm role allocation, return to step 2.2 to recalculate the role allocation result; if the reason for failure is related to the candidate action sequence not meeting the current task state, return to step 3 to regenerate the candidate action sequence; if the reason for failure is related to execution deviation, synchronization error or contact instability, return to step 5 to perform reinforcement learning correction again.

[0144] Step 7.2: After the task is completed, store the effective task decomposition results, role allocation result sequence, cross-arm constraint parameters, candidate action sequence, enhanced actions, sub-objective completion judgment results, and execution success or failure information in the experience memory module.

[0145] Each experience record in the experience memory module includes: task semantic information, target object category, environmental structure information, phased sub-target sequence, role allocation results for each phase, candidate action sequence for each phase, enhanced actions for each phase, cross-arm constraint parameters, sub-target completion judgment results, and task execution success or failure markers.

[0146] This invention first performs task semantic decomposition on natural language task instructions, parsing the user-input task description into multiple staged sub-goals. Each sub-goal incorporates target object information and the role allocation information of the two arms, expanding the task representation from a simple action description to a structured representation including "task object—execution stage—two-arm division of labor." In specific task scenarios, such as fixing, opening, unfolding, grasping, and supporting, the role requirements of the left and right arms are labeled, enabling the system to distinguish whether the two arms are performing symmetrical collaboration or master-slave cooperation. Based on this task semantic decomposition result, this invention further scores the left and right arms under different role combinations according to the current sub-goal, object geometry, reachable areas of the left and right arms, end-effector posture adaptability, environmental obstacle layout, and historical execution success and failure information. It then automatically selects the current optimal role configuration, allowing the two arms to dynamically switch between different roles such as main operating arm, auxiliary support arm, fixing arm, grasping arm, alignment arm, propulsion arm, or collaborative transport arm during task progression, achieving a transformation from static division of labor to dynamic collaboration.

[0147] After completing task decomposition and role determination, this invention utilizes a vision-language-action prior model to generate candidate action sequences for both arms based on environmental visual observations, task semantic information, arm state information, and role allocation results. This candidate action sequence is not generated separately for the left and right arms and then spliced ​​together; instead, it generates the actions of both arms as a whole working collaboratively under unified task semantic constraints, ensuring that the action intent of one robotic arm is consistent with that of the other. For example, in a collaborative handling task, it generates dual-arm actions with symmetrical approach and synchronous lifting characteristics; in a fixed-operation task, it generates dual-arm actions with one side stable and the other flexible. In this way, the system can quickly provide dual-arm candidate action sequences that conform to the task semantics when facing new objects, new layouts, and new language tasks, thereby reducing the cost of planning and searching from scratch.

[0148] Because of the significant coupling between the two robotic arms during dual-arm operation, this invention further constructs a cross-arm constraint consistency reinforcement mechanism after candidate actions are generated, jointly evaluating the candidate actions from an overall collaborative perspective. This mechanism uniformly constrains the relative pose relationship of the two arm ends, the timing relationship of actions, the stability of the target object's posture, contact coordination, and collision safety, ensuring that the two arms meet task requirements during gripping, handling, deployment, and assembly. After passing the cross-arm constraint evaluation, this invention further employs a dual-arm residual reinforcement learning strategy to continuously correct the candidate actions, outputting left arm correction, right arm correction, and cross-arm coupling correction based on the candidate action sequence, to correct gripping deviation, insertion angle deviation, timing error, and contact instability issues in real time based on environmental feedback. This approach preserves the semantic rationality of the candidate action sequence, improves the accuracy of dual-arm motion control, and avoids the high learning cost and large sample requirements associated with training a reinforcement learning strategy from scratch.

[0149] To make the system applicable to real-world complex environments, this invention also constructs an uncertainty-aware safety arbitration mechanism to evaluate the executability of the current enhanced actions. This mechanism integrates factors such as semantic matching reliability, action value estimation stability, object state observation bias, contact risk, and dual-arm action conflict risk. When uncertainty is low, the corrected dual-arm actions are executed directly; when uncertainty is at a moderate level, a conservative control mode is switched to improve execution reliability by reducing the main operating arm's step size, increasing the stability of the auxiliary support arm, and reducing the overall action speed; when uncertainty is too high, role reassignment, sub-target replanning, or safety rollback operations are triggered to prevent high-risk actions from being executed directly.

[0150] Furthermore, this invention incorporates a continuous experience enhancement mechanism. After each task is completed, the effective task decomposition results, dual-arm role switching sequence, cross-arm constraint parameters, key action segments, and execution feedback are stored in the experience memory module. When encountering similar task semantics, similar object categories, or similar environmental structures subsequently, historical successful experiences are preferentially invoked to assist in the initialization of role allocation, action generation, and correction strategies, thereby reducing redundant exploration and improving long-term operational efficiency and task success rate.

[0151] Through the above mechanism, this invention achieves unified closed-loop control from high-level task understanding, dynamic allocation of dual-arm roles, generation of dual-arm candidate actions, cross-arm constraint consistency evaluation, fine correction of residual reinforcement learning, to uncertainty perception, safe execution, and continuous enhancement of experience, thereby significantly improving the practicality, stability, and deployability of dual-arm robots in complex operation scenarios.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the ideas of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for controlling a dual-arm visual language action based on reinforcement learning correction, characterized in that, include: Step 1: Obtain the multimodal input information required for the dual-arm robot to perform the task, encode the multimodal input information in a unified manner to obtain the fused feature vector, and construct a unified state representation for dual-arm cooperative control; Step 2: Perform task semantic parsing on the natural language task instructions and dynamically assign roles to both arms; Step 3: Based on the fused feature vector and the dynamic allocation result of the two-arm role, call the pre-trained vision-language-action candidate action sequence generation model to output the two-arm candidate action sequence that meets the semantic constraints of the task; Step 4: Construct a mathematical model for cross-arm constraint consistency for dual-arm operation, and perform a joint evaluation of cross-arm constraint consistency for dual-arm candidate action sequences; Step 5: Construct a two-arm residual reinforcement learning policy network to enhance the action sequence selected through joint evaluation of cross-arm constraint consistency; Step 6: Perform an uncertainty assessment on the current enhancement action; Step 7: Send the enhanced actions that have passed the evaluation to the dual-arm robot actuator for control execution, and obtain the execution result feedback in real time.

2. The dual-arm visual-language action control method based on reinforcement learning correction according to claim 1, characterized in that, The multimodal input information in step 1 includes: environmental visual observation information. Information on the status of both arms Historical execution context information and natural language task instructions ; Environmental visual observation information includes: image information of the target object, spatial positional relationships, distribution of obstacles in the scene, and structural information of the operating area; dual-arm body state information includes: joint states of the left and right arms, end effector pose, gripper state, and motion state information; historical execution context information includes: previous action, completed sub-objectives, execution success or failure markers, and environmental change information; natural language task instructions are used to describe the operational objectives and task constraints that the dual-arm robot needs to perform.

3. The dual-arm visual-language action control method based on reinforcement learning correction according to claim 1, characterized in that, In step 1, the multimodal input information is uniformly encoded to obtain a fused feature vector, and a unified state representation for dual-arm cooperative control is constructed, specifically as follows: Using a visual encoder visual observation information Extracting visual semantic features Use a language encoder Instructions for natural language tasks Extracting semantic features of the task Employing a body state encoder Information on the state of the two arms Extracting motion state features A timing encoder is used. Historical execution context information Extracting temporal correlation features ; Its expression is: Among them, the timing encoder Sequence encoding is performed on the actions of the previous moment, completed sub-objectives, success or failure markers, and environmental change information to extract temporal correlation features that reflect the progress of task execution and the impact of historical actions; The extracted features are then fused to obtain a fused feature vector. : in, The network represents a feature fusion network, and [ ] represents feature concatenation or joint mapping operations; the unified state is represented as: The unified state representation Used for subsequent task decomposition, role assignment, candidate action generation, reinforcement learning correction, and arbitration.

4. The dual-arm visual-language action control method based on reinforcement learning correction according to claim 3, characterized in that, Step 2 specifically involves: Step 2.1: Obtaining a unified state representation Then, the natural language task instructions are subjected to task semantic parsing, decomposing the original task into multiple staged sub-goals with execution order: The phased sub-target sequence is represented as follows: in, Indicates the first Each stage-specific sub-goal; represented as: in, Indicates the semantics of stage actions. This represents target object information composed of target object image information and target object pose information. This indicates the coordination constraint information between the two arms; Step 2.2: Based on the phased sub-goals, and combined with the current target object position and posture, left and right arm reachability, gripper posture matching degree, obstacle spatial distribution, and historical execution status collected by the sensors, the two arms are dynamically matched and scored under each candidate role combination to obtain the role allocation result for the current phase. Step 2.2.1: Based on the current stage sub-goals, target object information, and dual-arm collaborative constraint information, construct a candidate role configuration set; the candidate roles include at least one or more of the following: main operating arm, auxiliary support arm, fixed arm, execution arm, gripping arm, alignment arm, and collaborative transport arm; Step 2.2.2: Eliminate candidate configurations that do not meet the constraints of left and right arm reachability, gripper posture matching, obstacle safety distance, and task role to obtain a feasible candidate set; Step 2.2.3: Calculate the role allocation adaptation index for each candidate role in the feasible candidate set. , is represented as: Among them, A( , R( represents the fit between the candidate role configuration and the action of the stage sub-objective). , () indicates that the left and right arms are in a unified state. Accessibility and ontological state fit under the given conditions, M( , C( ) represents the degree of matching between the gripper posture and the target object. , O() represents the degree to which the candidate role configuration satisfies the dual-arm collaboration constraint. () indicates the cost of obstacles and collision risks. to Preset weights; Step 2.2.4: Calculate the RASI value for all candidate role configurations, sort them from highest to lowest score, and select the candidate role configuration that satisfies the constraints and has the highest score as the role allocation result for the current stage. Step 2.2.5: If none of the candidate role configurations meet the constraints, trigger the sub-goal replanning or safe rollback operation.

5. The dual-arm visual-language action control method based on reinforcement learning correction according to claim 3, characterized in that, Step 3 specifically involves: Step 3.1: Pre-train or fine-tune the pre-training based on the existing robot operation dataset to obtain a vision-language-action candidate action sequence generation model; during the model training phase, a robot operation dataset including dual-arm visual observations, natural language task instructions, dual-arm states, target action sequences, and task success markers is used for training; the training objective is to ensure that the candidate action sequences output by the model are consistent with the target action sequences in the samples and satisfy the task semantics and role constraints, and its training loss is expressed as: in, Represents a candidate action sequence. This represents the sequence of exemplary actions in the dataset. This represents the task semantic consistency loss. This represents the loss of consistency due to role constraints. and These are weight parameters; The existing datasets include one or more of the following: Open X-Embodiment dataset, DROID dataset, BridgeData V2 dataset, RH20T dataset, RoboSet dataset, and CALVIN language-conditional robot manipulation dataset; Step 3.2: Fuse the feature vectors Phased sub-goals And role allocation results The input is a vision-language-action candidate action sequence generation model, which outputs candidate action sequences for the left and right arms at the current stage. Its expression is: in, This represents a visual-language-action candidate action sequence generation model. Indicates the first Candidate arm movement sequences for the phase; Role allocation results during the above candidate action sequence process The input as an action generation condition is used to the candidate action sequence generation model. In the process, the action generation conditions include at least the left arm role label, the right arm role label, the current stage sub-target, the target object information, and the dual-arm coordination constraint information, which are used to limit the types of actions that the prior model generates for the left and right arms respectively in the current stage; the candidate action sequence generation model According to phased sub-goals The execution order generates a candidate action sequence in stages; The candidate action sequence includes at least the changes in the end-arm pose, the increments in joint motion, the changes in the gripper state, and the action synchronization parameters; expressed as: in, and These represent the changes in pose at the distal ends of the left and right arms, respectively. and These represent the increments in joint movement for the left and right arms, respectively. and These represent the changes in the gripper states of the left and right arms, respectively. This indicates the synchronization parameters for both arm movements.

6. The dual-arm visual-language action control method based on reinforcement learning correction according to claim 1, characterized in that, Step 4 specifically involves: Step 4.1: Construct a mathematical model for cross-arm constraint consistency for dual-arm operations, including at least the following constraint terms: 1) Relative pose constraints at the end effectors of both arms are used to limit the relative spatial relationship required by the arms during grasping, unfolding, handling, and assembly; the relative pose error of the end effectors is expressed as: in, and These represent the distal positions of the left and right arms, respectively. Indicates the relative pose of the target; 2) Motion timing synchronization constraints are used to ensure that key movements of both arms are executed synchronously or with a specified phase difference in time; the motion timing synchronization error is expressed as: in, and These represent the key moments of the left and right arm movements, respectively. Indicates the preset phase difference; 3) Target object stability constraints are used to ensure that the attitude, position, and force state of the target object remain within allowable limits under the action of both arms; the target object stability error is expressed as: in, Indicates the current position of the target object. Indicates the desired position of the target object. Indicates the current pose of the target object. Indicates the desired pose of the target object. This indicates the current force state of the target object. This represents the desired force state of the target object, and μ represents the force error weight. 4) Contact coordination constraints are used to avoid excessive force on one side, insufficient support, or conflicting contact methods; the contact coordination error is expressed as: in, and These represent the contact forces of the left and right arms, respectively. and η represents the contact state of the left and right arms respectively, and η represents the contact state error weight. 5) Collision safety constraints are used to limit the collision risk between the two arms and between the two arms and the environment; the collision risk error is expressed as: in, Indicates the safe distance threshold. This indicates the minimum distance between the arms or between the arms and the environment. Step 4.2: Based on the above constraint consistency mathematical model and candidate action sequences, calculate the relative pose error of the two arm ends, the action timing synchronization error, the target object stability error, the contact coordination error, and the collision risk error, and calculate the joint evaluation cost function: in, to For joint evaluation weighting parameters; If the relative pose error of the two arm ends, the timing synchronization error of the action, the stability error of the target object, the contact coordination error, the collision risk error, and the joint evaluation cost function of the candidate action sequence are all less than the corresponding preset thresholds, it is determined that it meets the cross-arm constraint consistency requirements; otherwise, it is removed.

7. The dual-arm visual-language action control method based on reinforcement learning correction according to claim 1, characterized in that, Step 5 specifically involves: Step 5.1: During the execution of the current stage, represent the unified state of the current task. Role allocation results Environmental feedback information and the candidate action sequence generated in step 3 Input a dual-arm residual reinforcement learning policy network; Step 5.2: The dual-arm residual reinforcement learning strategy network determines whether there are problems such as grasping bias, alignment error, synchronization error, contact instability, or posture drift in the candidate action sequence based on the current environmental feedback, and outputs the action residual correction amount. Its expression is: in, This represents a residual reinforcement learning policy network. This represents environmental feedback information consisting of visual feedback, force / contact feedback, arm status feedback, and task progress feedback. The motion residual correction includes the output of the left arm motion correction through the left arm correction branch, the right arm correction branch, and the cross-arm coupling correction branch. Right arm movement correction amount And the correction amount ΔAc for the coupled action of the two arms; the residual correction amount is superimposed on the candidate action sequence to obtain the enhanced action of the current stage; in, This represents the residual correction amount of the reinforcement learning policy network output. This indicates the enhanced action after residual correction; Step 5.3: When training the dual-arm residual reinforcement learning policy network, the reward function includes task completion rewards. Smooth motion reward Cross-arm collaborative rewards Safety constraints and rewards and performance-based rewards , is represented as: in, to The reward weights are used to describe how the dual-arm residual reinforcement learning strategy network learns how to perform residual compensation based on the candidate action sequence through the above reward function, so that the enhanced action achieves a balance between task completion, motion smoothness, dual-arm coordination, safety and execution efficiency.

8. The dual-arm visual-language action control method based on reinforcement learning correction according to claim 1, characterized in that, Step 6 specifically involves: Step 6.1: Uncertainties include: uncertainty in task semantic matching, uncertainty in action value assessment, uncertainty in target object state deviation, uncertainty in contact risk, and uncertainty in the risk of conflict between the two arms. Task semantic matching uncertainty is calculated based on the matching confidence between the language task and the augmented action, and is expressed as follows: in, Indicates the semantic matching confidence between the language task and the augmentation action; The uncertainty in action value assessment is calculated based on the variance of the reinforcement learning value network output, and is expressed as: in, This indicates that the reinforcement learning value network represents the current unified state. and enhance action The value estimate is represented by Var(·), which indicates the variance calculation. The uncertainty of the target object's state deviation is calculated based on the deviation between the observed state and the predicted state of the target object, and is expressed as: in, and These represent the observation position and observation attitude of the target object, respectively. and These represent the predicted position and predicted pose of the target object, respectively. The uncertainty of contact risk is calculated based on the contact force deviation and the degree of abnormality in the gripper condition, and is expressed as follows: in, Indicates the current contact force. Indicates the desired contact force. Indicates an abnormal gripper status. Indicates the weight of abnormal gripper status; The uncertainty of the risk of conflict between the two arms is calculated based on the minimum distance between the two arms and their relative motion trend, and is expressed as follows: in, Indicates the safe distance threshold. This indicates the minimum distance between the arms or between the arms and the environment. Indicates the speed at which the arms move away from or relative to an obstacle; when A value less than 0 indicates that the two tend to approach each other. Indicates relative motion risk weight; By weighting and summing the above uncertainties, we obtain the overall uncertainty: in, to The above formula is used to quantify semantic matching, value estimation, target object status, contact risk, and double-arm conflict risk into an overall uncertainty U, which is used for subsequent arbitration and safety rollback judgment. Step 6.2: Based on the overall uncertainty U obtained in Step 6.1 and the evaluation results of each individual uncertainty, arbitrate control is applied to the execution process of the current enhancement action; a first threshold is set. Second threshold ,and < ; If U≤ If the uncertainty of each individual item does not exceed the corresponding safety threshold, then the reliability of the current enhancement action is judged to be high, and the enhancement action is executed directly. like <U≤ If there is a single uncertainty that is close to the corresponding safety threshold, the current enhanced action is judged to have a medium risk and enters a conservative execution mode, including reducing the step length of the action, reducing the execution speed, enhancing the support behavior of the auxiliary arm, increasing the frequency of visual feedback detection, or recalculating the local action correction amount. If U> If any single uncertainty exceeds the corresponding safety threshold, the current enhancement action is judged to have a high risk, triggering a recalculation of the role allocation result, a replanning of the current sub-target, or a safety rollback operation.

9. The dual-arm visual-language action control method based on reinforcement learning correction according to claim 1, characterized in that, Step 7 specifically involves: Step 7.1: The execution result feedback includes: visual feedback, dual-arm status feedback, and task status feedback; Visual feedback parameters include: the current position of the target object. Current pose of the target object Target object appearance status Target object occlusion marker and the distance between the target object and the target area ; The bi-arm status feedback parameters include: the joint positions of the left and right arms. , End effector pose , Gripper open / closed state , terminal velocity , and contact status , ; Task status feedback parameters include: current sub-target number i, target area location. Capture stability indicators Support stability indicators Exception interruption flag and execution time ; Based on the above feedback parameters, construct a sub-target completion determination function: in, This represents environmental feedback information consisting of visual feedback, force / contact feedback, arm status feedback, and task progress feedback. Φ represents the sub-objective of the current stage, and Φ represents the sub-objective completion determination function. If the visual feedback condition, the dual-arm status feedback condition, and the task status feedback condition corresponding to the current sub-target are all satisfied... Determine that the current stage sub-objective has been completed, and proceed to the next stage sub-objective; If any critical feedback condition is not met, or an abnormal interruption is detected... When =1, If the current sub-objective is not completed, return to the corresponding step for processing based on the reason for failure: if the reason for failure is related to the mismatch of the two-arm role allocation, return to step 2.2 to recalculate the role allocation result; if the reason for failure is related to the candidate action sequence not meeting the current task state, return to step 3 to regenerate the candidate action sequence; if the reason for failure is related to execution deviation, synchronization error or contact instability, return to step 5 to perform reinforcement learning correction again. Step 7.2: After the task is completed, store the effective task decomposition results, role allocation result sequence, cross-arm constraint parameters, candidate action sequence, enhanced actions, sub-objective completion judgment results, and execution success or failure information in the experience memory module. Each experience record in the experience memory module includes: task semantic information, target object category, environmental structure information, phased sub-target sequence, role allocation results for each phase, candidate action sequence for each phase, enhanced actions for each phase, cross-arm constraint parameters, sub-target completion judgment results, and task execution success or failure markers.

Images