A shared control method for dual-arm teleoperation for asymmetric tasks

Abstract

The present invention discloses a shared control method for dual-arm teleoperation for asymmetric tasks. Based on shared control, the method constructs a sub-target guidance and predicted trajectory guidance framework for asymmetric tasks. The framework contains four modules: a sub-target generation module, a trajectory prediction module, a task allocation module, and a speed synthesis module. The sub-target generation module generates respective sub-targets for the left and right arms in the asymmetric task under image and language conditions. The trajectory prediction module predicts the future trajectory based on the past motion trajectory of the manipulator to improve the compliance in shared control. The speed synthesis module organically combines the sub-target speed, the predicted trajectory speed, and the operator's control speed to generate the final control speed of the manipulator. This method effectively solves the problems of the operator easily ignoring the intermediate process of the task during teleoperation, lacking target guidance, and shared control compliance, effectively reduces the number of fine-tuning times during teleoperation, and improves the efficiency and success rate of task completion.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot operation control, and in particular relates to a dual-arm teleoperation sharing control method for asymmetric tasks. Background Art

[0002] Asymmetric tasks, introduced in the paper "A bimanual manipulation taxonomy," are an important research direction in robotic manipulation. Due to the nature of the tasks, they require cooperation between two robotic arms. For example, one robotic arm is responsible for stabilization while the other handles manipulation. Despite significant progress in robotic manipulation, challenges remain in handling asymmetric tasks due to dual-arm collaboration and longer task objectives.

[0003] For asymmetric tasks, autonomous manipulation is the preferred approach, and numerous research results have been achieved. These can be categorized as motion path guidance and sub-goal guidance. "Planning with diffusion for flexible behavior synthesis" and "Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation" directly generate motion paths based on generative models, while "Skilldiffuser: Interpretable hierarchical planning via skill abstractions indiffusion-based task execution" plans long-term goals as sub-goals through LLM and employs additional path planning methods. These autonomous manipulation methods can ensure optimal operational control and deterministic motion for asymmetric tasks. Both of these rely heavily on the accuracy of sub-goal generation or motion path planning, significantly reducing the success rate of robotic arms in asymmetric tasks. For challenging tasks such as "handing over an object," the success rate in "Voxact-b: Voxel-based acting and stabilizing policy for bimanual manipulation" is less than 20%. This phenomenon is particularly pronounced in the real world.

[0004] Another effective approach to operating asymmetric tasks is teleoperation, based on human experience. The operator's intent is transmitted from the master device to the slave actuator, naturally establishing closed-loop control by introducing a human element. This ensures more flexible movement and enhanced obstacle avoidance. However, during the operation of complex tasks, operators often focus on achieving the ultimate goal while neglecting important intermediate sub-goals or failing to determine the optimal sub-goal, resulting in the selection of a locally optimal path. Summary of the Invention

[0005] The purpose of the present invention is to provide a dual-arm teleoperation shared control method for asymmetric tasks, mainly to solve the problems of existing dual-arm teleoperation methods in asymmetric tasks, such as neglect of task intermediate states, lack of target guidance and insufficient shared control flexibility.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A dual-arm teleoperation shared control method for an asymmetric task comprises the following steps:

[0008] S1, based on the RGB-D image of the scene and the final task language instructions, generates the sub-goals of the left and right robotic arms respectively through the sub-goal generation module;

[0009] S2, based on the historical motion trajectory of the robot arm, the trajectory prediction module predicts and generates the predicted trajectory of the left and right robot arms, and generates the predicted trajectory guidance speed for the left and right robot arms according to the predicted trajectory;

[0010] S3, based on the predicted trajectory and the sub-goals of the left and right robotic arms, the task assignment module determines the roles of the left and right robotic arms according to the relative distance from the predicted trajectory to the sub-goal, and generates sub-goal guidance speeds for each robotic arm through the sub-goal;

[0011] S4, based on the sub-target guidance speed, the predicted trajectory guidance speed and the operator control speed, the speed synthesis module forms the final speed of the left and right robotic arms to control the movement of the robotic arms.

[0012] Furthermore, in step S1, the sub-goal generation process is as follows:

[0013] S11, inputs the text query and the RGB image from the front camera into the open vocabulary object detector OWL-ViT to detect objects;

[0014] S12 uses the image segmentation model Segment Anything to crop the target object and generate language instructions for the left and right robotic arms;

[0015] S13, encodes the cropped target object and generated language instructions through 3D CNN and CLIP respectively;

[0016] S14, based on the encoded input, PerceverIO Transformer extracts high-dimensional embeddings, aggregates key information through a cross-attention mechanism, and uses a transposed convolution solver for decoding to obtain the corresponding sub-goals of the left and right robotic arms.

[0017] Furthermore, in the step S12, the clipping of the image segmentation model Segment Anything is implemented based on the voxel grid clipping unit, and the voxel grid clipping unit has an initial size of L × W × H The voxel grid is cropped, and the cropping ratio is determined by the parameter λ. After cropping, a new voxel grid is formed with improved resolution.

[0018] Furthermore, in step S2, the historical motion trajectory of the robotic arm is defined as follows:

[0019]

[0020] in, p Indicates location, v Indicates speed, a Indicates acceleration; 1: T obs Indicates a past time period. Indicates data dimension;

[0021] Use LSTM module model encoding to obtain historical features And the future features are obtained by encoding the future trajectory through LSTM module training ;

[0022] Obtained through MLP and , and Obeying Bernoulli distribution, r is a latent variable, X For the historical trajectory, Y To predict the trajectory guidance speed, ;

[0023] The sampler obtains the latent variables and decodes them to obtain the probability distribution of future velocities, and generates the predicted trajectory guidance speed through GMMs sampling.

[0024] Furthermore, in step S3, the specific method for determining the roles of the left and right robotic arms is:

[0025] Based on the respective sub-goals of the left and right robotic arms and the future trajectories of the left and right robotic arms, calculate the distance from the predicted trajectories of the left and right robotic arms to the sub-goal of the stable arm Dl,s and D r,s , and the distances from the predicted trajectories of the left and right manipulators to the manipulator sub-targets D l,o and D r,o ;if Compare If the value is large, the right arm is the operating arm, otherwise it is the stable arm.

[0026] Furthermore, in step S4, the speed synthesis module dynamically determines the weights of the sub-target guidance speed and the predicted trajectory guidance speed in the final speed based on the similarity between the sub-target guidance speed, the predicted trajectory guidance speed and the operator control speed; that is:

[0027]

[0028] in, Guide the speed for the subgoal, Control speed for the operator; is the predicted trajectory guidance speed; and is the importance coefficient.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) This invention introduces a sub-goal guidance strategy that fully focuses on intermediate states, thereby improving the ability to fine-tune dual-arm collaboration to complete dual-arm symmetric tasks. By further integrating the predicted trajectory and operator motion, the smoothness of the motion is improved and the operator's sensitivity to sub-goals is reduced. As a result, it significantly improves the task success rate and the amount of adjustment during the operation.

[0031] (2) The present invention decomposes tasks into intermediate sub-goals in a refined manner, guides operators to focus on key states, avoids local optimal operations, and significantly improves the success rate of tasks; at the same time, dynamic weight adjustment Reduce operator sensitivity and reduce the number of fine-tuning times by more than 50%.

[0032] (3) The present invention uses a task allocation module to allocate the roles of the stabilizing arm / manipulating arm in real time based on the distance from the predicted trajectory to the sub-target. The state role allocation adapts to asymmetric task changes, improves the efficiency of dual-arm collaboration, and shortens the task completion time by 30%. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below with reference to the accompanying drawings and examples. The embodiments of the present invention include but are not limited to the following examples. Example

[0035] like Figure 1 As shown in the figure, the present invention discloses a shared control method for dual-arm teleoperation for asymmetric tasks. The overall framework of this method includes four core modules: a sub-goal generation module, a trajectory prediction module, a task allocation module, and a velocity synthesis module. These modules work together to generate sub-goals and predict future trajectories based on image and language information, ultimately dynamically synthesizing them to provide precise control instructions for the robotic arm. First, the sub-goal generation module receives an RGB-D image of the scene and language instructions as input. It detects the target object using the OWL-ViT model and generates a segmentation mask for the target object using the Segment Anything model. The centroid of the segmentation mask is combined with the point cloud data in the RGB-D image to calculate the pose of the target object relative to the front camera. To zoom in on the region of interest, the voxel grid cropping unit crops the initial voxel grid of size L×W×H. The cropping ratio is determined by the parameter λ. This cropping results in a new voxel grid with a smaller workspace size but higher resolution. The cropped voxel grid, along with the language command and the robot arm ID, is input into the PerceiverIO Transformer model. Six self-attention layers encode the latent variables, and a cross-attention mechanism aggregates key information. Finally, a transposed convolution solver is used for decoding to obtain the corresponding sub-goals for the stabilizing and manipulating arms. The loss function using cross entropy is expressed as:

[0036] ;

[0037] I Represents input information, including voxel representation, language instructions, and robotic arm ID. p Represents the position of the sub-goal at the next moment. Indicates a given I of p of Q value. Represents the ground truth recorded in the expert argumentation data. ,in s It is a stable arm. o It's the operating arm.

[0038] The trajectory prediction module consists of four parts: past encoder, future encoder, sampler, and decoder. First, define the historical motion trajectory of the robot arm:

[0039]

[0040] in, p Indicates location,v Indicates speed, a Indicates acceleration; 1: T obs Indicates a past time period. Indicates data dimension;

[0041] The past encoder uses its LSTM module to encode past trajectory information and output features , the future encoder encodes the future velocity v with the help of bidirectional LSTM to obtain the feature , the future encoder is only used during training. In the sampler, the feature objects are processed by a multilayer perceptron to obtain , , obeys Bernoulli distribution. Then the feature and Connected together and processed by a multilayer perceptron, we get , Obeys Bernoulli distribution. Latent variable r In training by , and during testing by The decoder is used to sample the latent variables. r ,feature and speed v Decode and get the GMMs distribution By sampling from GMMs, we can get possible future velocities:

[0042] ;

[0043] Predicted trajectory position Obtained by the following equation:

[0044]

[0045] For the task assignment module, the first sub-targets of the stabilizing arm and the manipulating arm are generated based on the current RGB-D image and language instructions. In the initial stage of teleoperation, the trajectory prediction module predicts the future trajectory of the manipulator based on historical motion information. Then, the distance from the predicted trajectory of the left and right manipulator arms to the stabilizing arm sub-target is calculated separately. D l,s and D r,s , and the distances from the predicted trajectories of the left and right manipulators to the manipulator sub-targets D l,o and D r,o ;if Compare If the right arm is larger, the right arm becomes the operating arm, otherwise it becomes the stabilizing arm. The task allocation module will not be used after it determines which of the left and right arms is the stabilizing arm and which is the operating arm.

[0046] Speed ​​synthesis module: We predict the speed of the trajectory , sub-target guidance speed and operator movement speed Combined to establish the final command speed , expressed as:

[0047]

[0048] in, Guide the speed for the subgoal, Control speed for the operator; is the predicted trajectory guidance speed; and is the importance coefficient. Determine the weight of the first and second items, and the two together determine the weight of the last item.

[0049] Generate by using two attractors through artificial potential field and , the expression is:

[0050] ;

[0051] j represents the attraction constant, P represents the location of the sub-target or predicted trajectory, P ee Represents the position of the robot arm's end effector. D=|| P − P ee ||, ε is a small positive value. In addition, and Will be adjusted dynamically over time and can be determined separately by speed similarity as follows:

[0052] ;

[0053] The proposed dual-arm shared control framework introduces a subgoal guidance strategy that focuses on intermediate states, thereby improving the ability to achieve refined dual-arm collaboration for dual-arm symmetric tasks. By further integrating predicted trajectories with operator motion, it improves motion smoothness and reduces operator sensitivity to subgoals. Consequently, it significantly improves the success rate of asymmetric tasks and the amount of adjustment required during operation.

[0054] In actual application scenarios, such as when two arms collaborate to complete object grasping and placement tasks, this method can effectively solve the problem that operators tend to ignore the intermediate states of the task. When the operator focuses on the final goal, the sub-goal generation module generates specific intermediate sub-goals through images and language instructions to ensure the fineness of task decomposition. At the same time, the trajectory prediction module predicts future trajectories by analyzing historical trajectories, providing smooth motion instructions for the robot arm and reducing the operator's control burden. The task allocation module dynamically allocates the roles of the left and right robot arms according to the sub-goals and predicted trajectories, enhancing the flexibility of dual-arm collaboration. The speed synthesis module generates the final control instructions by dynamically combining the sub-goal guidance speed, the predicted trajectory guidance speed and the operator control speed, ensuring the accuracy and flexibility of the robot arm's movement.

[0055] The above embodiment is only one of the preferred implementation methods of the present invention and should not be used to limit the scope of protection of the present invention. Any changes or modifications that have no substantive meaning made to the main design concept and spirit of the present invention, as long as the technical problems solved are still consistent with the present invention, should be included in the scope of protection of the present invention.

Claims

1. A dual-arm teleoperation shared control method for asymmetric tasks, characterized in that: The following steps are involved: S1, based on the RGB-D image of the scene and the final task language instructions, generates sub-goals for the left and right robotic arms through the sub-goal generation module; the specific process is as follows: S11, inputs the text query and the RGB image from the front camera into the open vocabulary object detector OWL-ViT to detect objects; S12, use the image segmentation model Segment Anything to cut out the target object and generate language instructions for the left and right robotic arms; the image segmentation model Seg Anything’s cutting is based on the voxel grid cutting unit, and the voxel grid cutting unit has an initial size of L × W × H The voxel grid is clipped, and the clipping ratio is determined by the parameter Decision, after cropping, a new voxel grid is formed with an increased resolution; S13, encodes the cropped target object and generated language instructions through 3D CNN and CLIP respectively; S14, based on the encoded input, PerceverIO Transformer extracts high-dimensional embeddings, aggregates key information through a cross-attention mechanism, and uses a transposed convolution solver for decoding to obtain the corresponding sub-goals of the left and right robotic arms; S2, based on the historical motion trajectory of the robot arm, the trajectory prediction module predicts and generates the predicted trajectory of the left and right robot arms, and generates the predicted trajectory guidance speed for the left and right robot arms according to the predicted trajectory; S3, based on the predicted trajectory and the sub-goals of the left and right robotic arms, the task assignment module determines the roles of the left and right robotic arms according to the relative distance from the predicted trajectory to the sub-goal, and generates sub-goal guidance speeds for each robotic arm through the sub-goal; S4, based on the sub-target guidance speed, the predicted trajectory guidance speed and the operator control speed, the speed synthesis module forms the final speed of the left and right robotic arms to control the movement of the robotic arms.

2. A dual-arm teleoperation shared control method for asymmetric tasks according to claim 1, characterized in that: In step S2, the historical motion trajectory of the robotic arm is defined as follows: in, Indicates location, Indicates speed, represents acceleration; Indicates past time. Indicates data dimension; Use LSTM module model encoding to obtain historical features And the future features are obtained by encoding the future trajectory through LSTM module training ; Obtained through MLP and , and Obeying Bernoulli distribution, is a latent variable, X For the historical trajectory, Y To predict the trajectory guidance speed, ; The sampler obtains the latent variables and decodes them to obtain the probability distribution of future velocities, and generates the predicted trajectory guidance speed through GMMs sampling.

3. A dual-arm teleoperation shared control method for asymmetric tasks according to claim 2, characterized in that: In step S3, the specific method for determining the roles of the left and right robotic arms is: Based on the respective sub-goals of the left and right robotic arms and the future trajectories of the left and right robotic arms, calculate the distance from the predicted trajectories of the left and right robotic arms to the sub-goal of the stable arm and , and the distances from the predicted trajectories of the left and right manipulators to the manipulator sub-targets and ;if Compare If the value is large, the right arm is the operating arm, otherwise it is the stable arm.

4. A dual-arm teleoperation shared control method for asymmetric tasks according to claim 3, characterized in that: In step S4, the speed synthesis module dynamically determines the weights of the sub-target guidance speed and the predicted trajectory guidance speed in the final speed based on the similarity between the sub-target guidance speed, the predicted trajectory guidance speed, and the operator control speed; that is, in, Guide the speed for the subgoal, Control speed for the operator; is the predicted trajectory guidance speed; and is the importance coefficient.

Images