Shared control method for asymmetric task two-arm teleoperation

Abstract

The invention discloses a two-arm teleoperation sharing control method for an asymmetric task, and the method comprises the steps: constructing a sub-target guiding and prediction track guiding frame in the asymmetric task based on sharing control; the framework comprises four modules: a sub-target generation module, a trajectory prediction module, a task allocation module and a speed synthesis module. The sub-target generation module generates respective sub-targets for the left arm and the right arm in the asymmetric task under image and language conditions. The trajectory prediction module predicts a future trajectory based on a past motion trajectory of the mechanical arm so as to improve the flexibility in shared control. And the speed synthesis module organically combines the sub-target speed, the predicted trajectory speed and the operator control speed to generate the final control speed of the mechanical arm. According to the method, the problems that an operator easily neglects a task intermediate process in a teleoperation process, target guidance and sharing control flexibility are lacked and the like are effectively solved, the fine tuning frequency during teleoperation is effectively reduced, and the task completion efficiency and the success rate are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot operation control, and specifically relates to a shared control method for bimanual teleoperation for asymmetric tasks. Background Art

[0002] Asymmetric tasks were proposed in the paper "A bimanual manipulation taxonomy" and are an important research direction in the field of robot manipulation. Due to the nature of the tasks, cooperation between the two arms is required to complete asymmetric tasks. For example, one robot arm is responsible for stabilization while the other is responsible for operation. Despite the meaningful progress made in robot operation, there are still challenges in handling asymmetric tasks due to the bimanual cooperation and longer task objectives.

[0003] For asymmetric tasks, autonomous operation is the preferred method and many research results have been achieved. They can be divided into motion path guidance and sub-goal guidance. "Planning with diffusion for flexible behavior synthesis" and "Chained diffuser: Unifying trajectory diffusion and key pose prediction for robotic manipulation" directly generate motion paths based on generative models, while "Skill diffuser: Interpretable hierarchical planning via skill abstractions in diffusion-based task execution" plans long-term goals as sub-goals through LLM and adopts an additional path planning method. The above autonomous operation methods can ensure optimal operation control and deterministic motion for asymmetric tasks, both of which rely heavily on the accuracy of sub-goal generation or motion path planning, but instead greatly reduce the operation success rate of the robot arm in asymmetric tasks. For challenging tasks such as "handing over an object", the success rate is less than 20% in the paper "Voxact-b: Voxel based acting and stabilizing policy for bimanual manipulation". This phenomenon is particularly obvious in the real world.

[0004] Another effective method for operating asymmetric tasks is teleoperation based on human experience. The operator's intention is transferred from the master device to the slave actuator, and a closed-loop control is naturally established by introducing human factors, which ensures more flexible movement and stronger obstacle avoidance ability. However, during the operation of complex tasks, the operator always focuses on the achievement of the final goal, ignoring important intermediate sub-goals or unable to determine the optimal sub-goal, resulting in the selection of a locally optimal path. Summary of the Invention

[0005] The purpose of the present invention is to provide a shared control method for dual-arm teleoperation in asymmetric tasks, mainly solving the problems of neglecting the intermediate state of tasks, lacking target guidance, and insufficient compliance of shared control in existing dual-arm teleoperation methods in asymmetric tasks.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows: A shared control method for dual-arm teleoperation in asymmetric tasks, comprising the following steps: S1, based on the scene RGB-D image and the final task language instruction, generate the respective sub-goals of the left and right robotic arms through the sub-goal generation module; S2, based on the historical motion trajectories of the robotic arms, predict and generate the predicted trajectories of the left and right robotic arms through the trajectory prediction module, and generate predicted trajectory guiding speeds for the left and right robotic arms according to the predicted trajectories; S3, based on the predicted trajectories and the sub-goals of the left and right robotic arms, the task allocation module determines the roles of the left and right robotic arms according to the relative distances from the predicted trajectories to the sub-goals, and generates sub-goal guiding speeds for their respective robotic arms through the sub-goals; S4, based on the sub-goal guiding speed, the predicted trajectory guiding speed, and the operator control speed, form the final speeds of the left and right robotic arms through the speed synthesis module to control the movement of the robotic arms.

[0007] Further, in the step S1, the generation process of the sub-goal is as follows: S11, input the text query and the RGB image from the front camera into the open-vocabulary object detector OWL-ViT to detect objects; S12, use the image segmentation model Segment Anything to crop out the target object and generate the language instructions for the tasks of the left and right robotic arms; S13, encode the cropped target object and the generated language instructions through 3D CNN and CLIP respectively; S14, based on the encoded input, the PerceverIO Transformer extracts high-dimensional embeddings, aggregates key information through the cross-attention mechanism, and uses the transposed convolutional decomposer for decoding to obtain the corresponding sub-goals of the left and right robotic arms.

[0008] Further, in the step S12, the cropping of the image segmentation model Segment Anything is implemented based on a voxel grid cropping unit, and the voxel grid cropping unit crops a voxel grid with an initial size of L × W × H . The cropping ratio is determined by the parameter λ, and a new voxel grid is formed after cropping with the resolution improved.

[0009] Further, in the step S2, the historical motion trajectory of the robotic arm is defined as: wherein, p represents the position, v represents the velocity, a represents the acceleration; 1: T obs represents the past time period, represents the data dimension; The historical features are obtained by encoding with the LSTM module model, and the future features are obtained by training and encoding the future trajectory with the LSTM module; Through the MLP, and are obtained. and follow the Bernoulli distribution, r is the latent variable, X is the historical trajectory, Y is the predicted trajectory guiding velocity, ; The sampler obtains the latent variable and decodes to obtain the probability distribution of the future velocity, and generates the predicted trajectory guiding velocity through GMMs sampling.

[0010] Further, in the step S3, the specific method for determining the roles of the left and right robotic arms is as follows: Based on the respective sub-goals of the left and right robotic arms and the future trajectories of the left and right robotic arms, calculate the distances D l,s and D r,s from the predicted trajectories of the left and right robotic arms to the sub-goal of the stabilizing arm, and the distances D l,o and D r,o from the predicted trajectories of the left and right robotic arms to the sub-goal of the operating arm respectively; if is larger than , then the right arm is the operating arm, otherwise it is the stabilizing arm.

[0011] Further, in the step S4: The speed synthesis module dynamically determines the weights of the sub-goal guidance speed and the predicted trajectory guidance speed in the final speed according to the similarity of the sub-goal guidance speed, the predicted trajectory guidance speed, and the operator control speed; that is: Wherein, is the sub-goal guidance speed, is the operator control speed; is the predicted trajectory guidance speed; wherein and are importance coefficients.

[0012] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention introduces a sub-goal guidance strategy, fully focuses on the intermediate state, thereby improving the ability to complete the double-arm symmetric task with high precision in double-arm collaboration. By further integrating the predicted trajectory and the operator's movement, the smoothness of the movement is improved, and the operator's sensitivity to the sub-goal is reduced. Therefore, it significantly improves the task success rate and the adjustment amount during the operation process.

[0013] (2) The present invention decomposes the task into intermediate sub-goals with high precision, guides the operator to focus on the key state, avoids local optimal operations, and significantly improves the task success rate; at the same time, the dynamic weight adjustment reduces the operator's sensitivity and reduces the number of fine-tuning times by more than 50%.

[0014] (3) The present invention assigns the roles of the stable arm / operating arm in real time based on the distance from the predicted trajectory to the sub-goal through the task allocation module. The dynamic role allocation adapts to the changes in asymmetric tasks, improves the double-arm collaboration efficiency, and shortens the task completion time by 30%. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] The present invention will be further described below with reference to the accompanying drawings and embodiments. The implementation manners of the present invention include but are not limited to the following embodiments. Embodiment

[0017] As Figure 1As shown in the figure, a shared control method for asymmetric task dual-arm teleoperation disclosed by the present invention. The overall framework of this method includes four core modules: a sub-goal generation module, a trajectory prediction module, a task allocation module, and a speed synthesis module. These modules cooperate together to generate sub-goals and predict future trajectories through image and language information, and finally dynamically synthesize precise control instructions for the robotic arms. First, the sub-goal generation module receives the RGB-D image of the scene and language instructions as inputs, detects the target object through the OWL-ViT model, and uses the Segment Anything model to generate the segmentation mask of the target object. The centroid of the segmentation mask is combined with the point cloud data in the RGB-D image to calculate the pose of the target object relative to the front camera. In order to magnify the region of interest, the voxel grid cropping unit crops the voxel grid with an initial size of L×W×H, and the cropping ratio is determined by the parameter λ. After cropping, a new voxel grid is formed, whose working space size is reduced but the resolution is improved. The cropped voxel grid, together with the language instructions and the robotic arm ID, is input into the PerceiverIO Transformer model, encodes the latent variables through 6 self-attention layers, and aggregates key information through the cross-attention mechanism. Finally, a transposed convolutional decomposer is used for decoding to obtain the corresponding sub-goals of the stable arm and the operating arm. The loss function using cross-entropy is expressed as: ; I represents the input information, including voxel representation, language instructions, and robotic arm ID. p represents the position of the sub-goal at the next moment. represents given I of p of Q value. represents the ground truth recorded in the expert demonstration data. , where s is the stable arm, o is the operating arm.

[0018] The trajectory prediction module consists of four parts: a past encoder, a future encoder, a sampler, and a decoder. First, define the historical motion trajectory of the robotic arm: Among them, p represents the position, v represents the speed, a represents the acceleration; 1: T obs represents the past time period, represents the data dimension; The past encoder uses its LSTM module to encode the past trajectory information and outputs the feature , the future encoder encodes the future speed v with the help of a bidirectional LSTM to obtain features , and the future encoder is only used during training. In the sampler, the feature object is processed by a multi-layer perceptron to obtain , , which follows a Bernoulli distribution. Then the feature and are concatenated and processed together by a multi-layer perceptron to obtain , , which follows a Bernoulli distribution. The latent variable r is obtained by during training and sampled by during testing. The decoder decodes the latent variable r , the feature and the speed v to obtain that follows a GMMs distribution. By sampling from the GMMs, possible future speeds can be obtained: ; The position of the predicted trajectory is obtained by the following equation: For the task assignment module, the first sub-goals of the stabilizing arm and the operating arm are generated based on the current RGB-D image and the language instruction. In the initial stage of teleoperation, the trajectory prediction module predicts the future trajectory of the robotic arm according to the historical motion information. Then, the distances D l,s and D r,s from the predicted trajectories of the left and right robotic arms to the sub-goal of the stabilizing arm, and the distances D l,o and D r,o from the predicted trajectories of the left and right robotic arms to the sub-goal of the operating arm are calculated respectively; if is larger than , the right arm is the operating arm, otherwise it is the stabilizing arm. The task assignment module is no longer used after determining which arm is the stabilizing arm and which is the operating arm at the beginning.

[0019] Speed synthesis module: We combine the trajectory prediction speed , the sub-goal guidance speed and the operator motion speed to establish the final command speed , expressed as: where is the sub-goal guidance speed, For the operator to control the speed; For the predicted trajectory to guide the speed; where and are importance coefficients. Where Determines the weights of the first and second terms, and both together determine the weight of the last term.

[0020] Through the artificial potential field, two attractors are used to generate and , and the expression is: ; j Represents the attraction constant, P Represents the position of the sub-goal or predicted trajectory, P ee Represents the position of the end effector of the robotic arm. D = || P − P ee ||, and ε is a small positive value. In addition, and Will be dynamically adjusted over time and can be determined separately by velocity similarity as follows: ; The proposed dual-arm sharing control framework mentioned above, by introducing a sub-goal guidance strategy, fully focuses on the intermediate state, thereby improving the ability to complete dual-arm symmetric tasks with refined dual-arm collaboration. By further integrating the predicted trajectory and the operator's movement, the smoothness of the movement is improved, and the sensitivity of the operator to the sub-goal is reduced. Therefore, it significantly improves the success rate of asymmetric tasks and the amount of adjustment during operation.

[0021] In practical application scenarios, such as when the dual arms collaborate to complete object grasping and placement tasks, this method can effectively solve the problem that the operator is prone to ignoring the intermediate state of the task. When the operator focuses on the final goal, the sub-goal generation module generates specific intermediate sub-goals through image and language instructions to ensure the fineness of task decomposition. At the same time, the trajectory prediction module predicts the future trajectory through the analysis of historical trajectories, provides smooth movement instructions for the robotic arm, and reduces the control burden on the operator. The task allocation module dynamically allocates the roles of the left and right robotic arms according to the sub-goal and the predicted trajectory, enhancing the flexibility of dual-arm collaboration. The speed synthesis module generates the final control instruction by dynamically combining the sub-goal guidance speed, the predicted trajectory guidance speed, and the operator control speed, ensuring the accuracy and compliance of the robotic arm movement.

[0022] The above embodiments are only one of the preferred embodiments of the present invention and should not be used to limit the protection scope of the present invention. Any modifications or polishings made without substantial significance based on the main design concept and spirit of the present invention, as long as the technical problems solved are still consistent with those of the present invention, should be included within the protection scope of the present invention.

Claims

1. An asymmetric task dual-arm teleoperation shared control method, characterized in that, It includes the following steps: S1. Based on the scene-based RGB-D image and the final task language instruction, the sub-goal generation module generates the respective sub-goals of the left and right robotic arms; S2. Based on the historical motion trajectories of the robotic arms, the trajectory prediction module predicts and generates the predicted trajectories of the left and right robotic arms, and generates the predicted trajectory guiding speeds for the left and right robotic arms according to the predicted trajectories; S3. Based on the predicted trajectories and the sub-goals of the left and right robotic arms, the task allocation module determines the roles of the left and right robotic arms according to the relative distances from the predicted trajectories to the sub-goals, and generates the sub-goal guiding speeds for their respective robotic arms through the sub-goals; S4. Based on the sub-goal guiding speed, the predicted trajectory guiding speed, and the operator control speed, the speed synthesis module forms the final speeds of the left and right robotic arms to control the movement of the robotic arms.

2. The asymmetric task two-armed teleoperation shared control method according to claim 1, wherein In the step S1, the generation process of the sub-goal is as follows: S11. Input the text query and the RGB image from the front camera into the open-vocabulary object detector OWL-ViT to detect objects; S12. Use the image segmentation model Segment Anything to crop out the target object and generate the language instructions for the tasks of the left and right robotic arms; S13. Encode the cropped target object and the generated language instructions through 3D CNN and CLIP respectively; S14. Based on the encoded inputs, the PerceverIO Transformer extracts high-dimensional embeddings, aggregates key information through the cross-attention mechanism, and decodes using the transposed convolutional decomposer to obtain the corresponding sub-goals of the left and right robotic arms.

3. The asymmetric task dual-arm teleoperation shared control method according to claim 2, characterized in that In the step S12, the cropping of the image segmentation model Segment Anything is implemented based on a voxel grid cropping unit, and the voxel grid cropping unit crops a voxel grid with an initial size of L × W × H . The cropping ratio is determined by the parameter . After cropping, a new voxel grid is formed and the resolution is improved.

4. A method for shared control of asymmetric task dual-arm teleoperation according to claim 3, characterized in that, In the step S2, the historical motion trajectories of the robotic arms are defined: Among them, represents position, represents speed, represents acceleration; represents the past time period, represents the data dimension; Encode historical features using the LSTM module model and train and encode future trajectories through the LSTM module to obtain future features ; Obtained through the MLP and , and follow the Bernoulli distribution, is the latent variable, X is the historical trajectory, Y is the predicted trajectory guiding speed, ; The sampler obtains the latent variables, decodes to obtain the probability distribution of the future speeds, and generates the predicted trajectory guiding speeds through GMMs sampling.

5. A method for shared control of asymmetric task dual-arm teleoperation according to claim 4, characterized in that In the step S3, the specific method for determining the roles of the left and right robotic arms is: Calculate the distances from the predicted trajectories of the left and right robotic arms to the sub-goals of the stabilizing arm, based on the respective sub-goals of the left and right robotic arms and their future trajectories and , as well as the distances from the predicted trajectories of the left and right robotic arms to the sub-goals of the operating arm respectively and ; If is greater than , then the right arm is the operating arm, otherwise it is the stabilizing arm 6. The asymmetric task dual-arm teleoperation shared control method according to claim 5, characterized in that, In the step S4: The speed synthesis module dynamically determines the weights of the sub-goal guiding speed and the predicted trajectory guiding speed in the final speed according to the similarities of the sub-goal guiding speed, the predicted trajectory guiding speed, and the operator control speed; that is: Among them, is the sub-goal guiding speed, is the operator control speed; is the predicted trajectory guiding speed; among them and are importance coefficients.

Images