Double-arm smart teleoperation method and system based on motion redirection and adaptive force

Abstract

The invention discloses a two-arm smart teleoperation method and system based on motion redirection and self-adaption, and relates to the technical field of robots, and the method comprises the steps: determining the posture of a human hand and the posture of a human arm according to a human body image; the robot is redirected according to the human hand posture and the human arm posture through the space attention gating map convolutional network, and the joint angle of a mechanical hand of the robot and the joint angle of a mechanical arm of the robot are correspondingly obtained; the mechanical arm is driven according to the joint angle of the mechanical arm so that the mechanical arm can grab the target object; and target force needed for grabbing the target object is determined, and the mechanical arm is driven to grab the target object according to the target force and the joint angle of the mechanical arm. According to the method, topology modeling and cross-platform action redirection between the human body and the robot are achieved through the space attention gating map convolutional network, pairing data dependence is eliminated, and therefore the high precision of action reproduction and the safety and flexibility of the grabbing process are both considered in the complex dynamic environment.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a dual-arm dexterous telemanipulation method and system based on motion redirection and adaptive force. Background Technology

[0002] Robot teleoperation technology aims to enable human operators to efficiently control robots to perform complex tasks through remote interfaces, and is one of the important supporting technologies for realizing human-robot collaboration and remote operation. In this process, human-robot motion redirection and adaptive force control constitute the key foundation for achieving natural, precise, and safe operation. The former determines the accuracy and naturalness of the mapping of human movements on the robot platform, while the latter relates to the compliance and stability of the robot when interacting with the environment.

[0003] In the area of ​​human-robot motion redirection, existing research primarily employs supervised learning methods. These methods typically collect motion data pairs between humans and robots, using a supervised learning model to learn the mapping relationship between them, thereby enabling the transfer of human-robot actions. While this approach can achieve good operational results for specific tasks and on a single platform, its generalization ability is limited. Due to significant differences in kinematic structure (including the number of joints, link lengths, and degree-of-freedom constraints) between different robot platforms, supervised learning models often fail to transfer accurately across different platforms. Furthermore, each time a new robot platform is added, human-robot pairing data needs to be recollected, resulting in high training costs and low efficiency, making it difficult to meet the practical needs of multi-platform teleoperation.

[0004] In the field of dexterous hand telemanipulation, existing grasping control strategies are mainly divided into two categories. The first category is position-based control, which directly maps the posture of human fingers to the angle of robot fingers to achieve the imitation of human hand movements. The second category is fixed-threshold force control, which completes object grasping by setting a fixed grasping force or force threshold. The former can reproduce the gesture shape well, but lacks real-time perception of the object's contact state; the latter, although introducing force feedback, has a fixed grasping force, making it difficult to cope with different object shapes, material properties, and dynamically changing environments. Summary of the Invention

[0005] The main objective of this application is to propose a dual-arm dexterous teleoperation method and system based on motion redirection and adaptive force, so as to achieve accurate remote control operation of robot arms.

[0006] To achieve the above objectives, one aspect of this application proposes a dual-arm dexterous teleoperation method based on motion redirection and adaptive force, the method comprising the following steps: Determine the posture of a person's hand and arm based on human images; A spatial attention-gated graph convolutional network is used to redirect the robot based on the poses of the human hand and the human arm, thereby obtaining the joint angles of the robot's manipulator and the joint angles of the manipulator. The robotic arm is driven according to the joint angle of the robotic arm so that the robotic hand can grasp the target object; Determine the target force required to grasp the target object, and drive the robotic arm to grasp the target object based on the target force and the joint angle of the robotic arm.

[0007] In some embodiments, determining the posture of a person's hand and arm based on a human image includes the following steps: Acquire the human body image; The human body image is subjected to three-dimensional motion capture using a visual capture algorithm to obtain the joint information of the hand; The joint information of the hand is processed into a 3D hand pose; wherein the 3D hand pose includes the human hand pose and the human arm pose.

[0008] In some embodiments, the method of using a spatial attention-gated graph convolutional network to redirect the robot based on the pose of the human hand and the pose of the human arm, thereby obtaining the joint angles of the robot's manipulator and the joint angles of the manipulator, includes the following steps: The human body and robot skeleton are modeled as a weighted graph structure; where, the... The motion skeleton of a frame is represented as , Let N be a set of nodes, and let N be the total number of nodes. These nodes correspond to joints in a human body or robot. This is the set of skeleton connecting edges, used to represent the topological connection relationships between joints; The dynamic attention weight matrix, where each element represents the spatial attention weight between nodes, is adaptively generated by the spatial attention gated graph convolutional network during forward propagation; each node... Corresponding feature vector From three-dimensional position vector Quaternion rotation features Composition, in which Represents three-dimensional Euclidean space. Represents a quaternion vector space; any two adjacent nodes and The edge features between are defined as follows The edge features are used to represent the local skeleton geometric relationship between node i and node j, where i and j are node indices and i ≠ j; The spatial attention-gated graph convolutional network is used to normalize the positional features to eliminate the influence of scaling; wherein, the normalized positional features are calculated as follows: , , ; in, This represents the normalized 3D position feature vector of the i-th node in the k-th frame, where the superscript k represents the time frame index and the subscript i represents the node index. This represents the geometric center of the positions of all nodes in the motion skeleton of the k-th frame; The overall scaling factor is N; the total number of nodes is N; ||·|| represents the Euclidean norm. The normalized position features and the original rotation features are concatenated to form the node input of the weighted graph structure, which is then fed into the spatial attention module for message propagation. Attention weights are dynamically calculated based on spatial and pose similarity between nodes, and the features of neighboring nodes are weighted and aggregated to obtain aggregated information; the specific calculation of the aggregated information is as follows: ; in, m k,i This represents the aggregated information of node i in the k-th frame. Represents the set of neighboring nodes of node i; This represents a message coding function consisting of two fully connected layers. To represent neighboring nodes in the k-th frame For nodes Attention weight coefficient; The spatial attention-gated graph convolutional network is used to update nodes by fusing the aggregated information with the original features through gated residual units. The calculation formula includes: , ; in, This represents the original node feature vector of the i-th node in the k-th frame; This is the gate vector, used to control the fusion ratio of aggregated information and residual information; This represents the Sigmoid activation function; U and U are trainable weight matrices. It is the bias vector; This represents element-wise multiplication. This represents the updated node feature vector; The joint angles of the robotic hand and the robotic arm are determined based on the updated nodes.

[0009] In some embodiments, determining the target force required to grasp the target object includes the following steps: The target force required to grasp the target object is predicted based on the scene image of the target object using a visual language model.

[0010] In some embodiments, driving the robotic arm to grasp the target object based on the target force and the joint angle of the robotic arm includes the following steps: The predicted force is determined based on the joint angles of the robotic arm; The model predictive control is used to drive the robotic arm to grasp the target object based on the predicted force and the target force.

[0011] In some embodiments, determining the predicted force based on the joint angles of the robotic arm includes the following steps: A force prediction model is constructed based on a pre-defined dataset D; The preset dataset D is represented as follows: ; in, This is the joint control command angle. The joint angle of the robotic arm. For historical force data, M The total dimension of the joints; k Joint dimension; The joint control command angle and the joint angle of the robot are combined as the input to the force prediction model to obtain the predicted force output by the force prediction model.

[0012] In some embodiments, the step of using model predictive control to drive the robotic arm to grasp the target object based on the predicted force and the target force includes the following steps: In each control cycle, the error between the predicted force and the target force is calculated; wherein, the optimization objective of the error calculation is defined as: ; in, For the optimization objective, This refers to the joint command from the previous moment. For the target force, Used to adjust the predicted force to the target force. As a regularization mechanism, it suppresses angle jumps. Used to control smoothness and response speed; Among them, PyTorch is used to... A differentiable parameter is established, and gradient descent updates are achieved through several iterations using the Adam optimizer. The calculation formula includes: ; Updated optimal joint commands As the control output for the next moment.

[0013] To achieve the above objectives, another aspect of this application proposes a dual-arm dexterous teleoperation system based on motion redirection and adaptive force, the system comprising: The pose determination unit is used to determine the pose of a person's hand and arm based on a human image; The redirection unit is used to redirect the robot according to the pose of the human hand and the pose of the human arm using a spatial attention-gated graph convolutional network, thereby obtaining the joint angles of the robot's manipulator and the joint angles of the manipulator. A robotic arm drive unit is used to drive the robotic arm according to the joint angle of the robotic arm so that the robotic hand can grasp the target object; A robotic arm drive unit is used to determine the target force required to grasp the target object, and drive the robotic arm to grasp the target object based on the target force and the joint angle of the robotic arm.

[0014] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0015] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0016] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0017] The embodiments of this application include at least the following beneficial effects: This application provides a dual-arm dexterous telemanipulation method and system based on motion redirection and adaptive force. The solution determines the human hand and arm postures from a human image; it then uses a spatial attention-gated graph convolutional network to redirect the robot's motion based on these postures, resulting in the joint angles of the robot's manipulator and arm; the manipulator is driven according to these joint angles to allow it to grasp a target object; the target force required to grasp the object is determined, and the manipulator is driven to grasp the object based on this force and the manipulator's joint angles. This application achieves topology modeling and cross-platform motion redirection between the human and robot through a spatial attention-gated graph convolutional network, eliminating paired data dependencies and thus balancing high accuracy in motion reproduction with safety and compliance in the grasping process in complex dynamic environments. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating the dual-arm dexterous teleoperation method based on motion redirection and adaptive force provided in this application embodiment; Figure 2 An example flowchart of a dual-arm dexterous teleoperation method based on motion redirection and adaptive force provided in an embodiment of this application; Figure 3 Example diagram of a redirection network provided in an embodiment of this application; Figure 4 Example diagram of the adaptive force control module provided in the embodiments of this application; Figure 5 A schematic diagram of the structure of a dual-arm dexterous teleoperation device based on motion redirection and adaptive force provided in an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] Before providing a detailed description of the embodiments of this application, some related technologies involved in the embodiments of this application will be described first, as follows: In recent years, with the development of deep learning and graph structure modeling, some studies have attempted to introduce Graph Neural Networks (GNNs) into human-robot motion retargeting. By modeling the human body and robot as topological graph structures, these studies aim to capture their spatial relationships and joint constraints, thereby improving the generalization ability of motion mapping to some extent. Meanwhile, other studies have combined visual perception with Model Predictive Control (MPC) for dynamic force adjustment and optimization during hand grasping. However, these methods are mostly optimized for single sub-tasks (such as improving motion retargeting accuracy or improving grasping force adjustment), lacking a unified system framework. Therefore, it is difficult to achieve coordination and balance between cross-platform motion transfer and adaptive force control.

[0023] Therefore, there is an urgent need for a new dual-arm dexterous teleoperation system architecture that can achieve cross-platform high-precision motion redirection and multimodal perception-based adaptive force control within a unified framework, so as to significantly improve the naturalness, stability and safety of teleoperated robots in complex task scenarios.

[0024] Existing human-robot motion retargeting methods primarily convert human movements into robot motion through paired data or kinematic mapping. Traditional sensor-based motion capture methods (such as VR devices, data gloves, or optical tagging systems), while highly accurate, suffer from large size, high cost, and inconvenient operation, making them difficult to implement in everyday remote control environments. To reduce hardware dependence, vision-based non-contact methods have emerged in recent years, capturing human motion through RGB images and 3D pose estimation. However, these methods often focus on single body parts (such as hands or arms), lacking a unified model of the whole-body topology, leading to decreased motion retargeting accuracy in multi-degree-of-freedom systems. While some learning-based cross-platform retargeting methods can achieve good results on a single robot platform, they still rely on human-robot paired datasets, have limited generalization ability, and are difficult to extend to different robot structures.

[0025] On the other hand, in robotic dexterity hand manipulation, existing adaptive force control strategies mostly employ position control or fixed threshold force control. While these methods can achieve basic grasping, they struggle to adjust in real-time based on object material, shape, and contact state, easily leading to object damage or grasping failure. Although some research has introduced force sensors or impedance control to enhance stability, these often require precise dynamic modeling, resulting in complex implementation and difficulty in generalizing to multi-object scenarios. Some methods combining machine learning and MPC have achieved force prediction and online optimization, but still lack understanding of the semantic features of the target object and task-level adaptability.

[0026] To address the aforementioned shortcomings, this application aims to propose a robot teleoperation scheme that integrates visual semantic reasoning and force feedback optimization. This scheme utilizes a Spatial Attention Gated Graph Convolutional Network (SAG-GCN) to achieve topological modeling and cross-platform motion redirection between the human and robot, eliminating pairing data dependencies. Simultaneously, it combines a Visual Language Model (VLM) and Model Predictive Control (MPC) to achieve intelligent force prediction and real-time adaptive force control based on object semantics, thereby balancing high accuracy in motion reproduction with safety and compliance during the grasping process in complex dynamic environments.

[0027] In dual-arm dexterous teleoperation, the cross-platform generalization capability of human-robot motion redirection and the interactive adaptability of the grasping process are key to achieving natural and precise operation. Through motion redirection, robots can mimic human movement trajectories to perform remote operation tasks; through adaptive force control, robots can adjust the grasping force according to the properties and contact state of different objects to ensure stability and safety. However, existing teleoperation systems still face the following problems: First, there are structural and kinematic differences between different robot platforms. Existing motion redirection methods mostly use direct mapping, which lacks the ability to generalize to heterogeneous structures across multiple platforms, resulting in insufficient mapping accuracy and unnatural motion reproduction; second, existing systems generally lack an adaptive force control mechanism based on the fusion of vision and force feedback, making it difficult to dynamically adjust the grasping force according to real-time perception results, which can easily lead to unstable grasping or damage to the target in complex interactive scenarios.

[0028] This application aims to provide a dual-arm dexterous teleoperation system based on motion redirection and adaptive force control. The system achieves high-precision motion redirection of human movements through the fusion modeling of visual perception and motion coding; simultaneously, it combines a visual language model (VLM) and model predictive control (MPC) to achieve adaptive force control based on object semantics and tactile feedback, thereby enabling natural, stable, and safe dual-arm teleoperation across multiple platforms and scenarios.

[0029] This application provides a dual-arm dexterous telemanipulation method and system based on motion redirection and adaptive force, relating to the field of robotics. The dual-arm dexterous telemanipulation method and system based on motion redirection and adaptive force provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited thereto; the server can be configured as an independent physical server, a server cluster composed of multiple physical servers, or a distributed system; it can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the server can also be a node server in a blockchain network; the software can be an application implementing the dual-arm dexterous telemanipulation method based on motion redirection and adaptive force, but is not limited to the above forms.

[0030] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0031] Reference Figure 1 This application provides a dual-arm dexterous teleoperation method based on motion redirection and adaptive force. This method may include, but is not limited to, steps S100 to S130, as follows: S100: Determines the posture of a person's hand and arm based on a human body image; S110: Using a spatial attention-gated graph convolutional network, the robot is redirected according to the pose of the human hand and the pose of the human arm, thereby obtaining the joint angles of the robot's manipulator and the joint angles of the manipulator. S120: Drive the robotic arm according to the joint angle of the robotic arm so that the robotic hand can grasp the target object; S130: Determine the target force required to grasp the target object, and drive the robotic arm to grasp the target object based on the target force and the joint angle of the robotic arm.

[0032] Optionally, determining the posture of a person's hand and arm based on a human image includes the following steps: Acquire the human body image; The human body image is subjected to three-dimensional motion capture using a visual capture algorithm to obtain the joint information of the hand; The joint information of the hand is processed into a 3D hand pose; wherein the 3D hand pose includes the human hand pose and the human arm pose.

[0033] Optionally, the step of using a spatial attention-gated graph convolutional network to redirect the robot based on the pose of the human hand and the pose of the human arm, thereby obtaining the joint angles of the robot's manipulator and the joint angles of the manipulator, includes the following steps: The human body and robot skeleton are modeled as a weighted graph structure; where, the... The motion skeleton of a frame is represented as , Let N be a set of nodes, and let N be the total number of nodes. These nodes correspond to joints in a human body or robot. This is the set of skeleton connecting edges, used to represent the topological connection relationships between joints; The dynamic attention weight matrix, where each element represents the spatial attention weight between nodes, is adaptively generated by the spatial attention gated graph convolutional network during forward propagation; each node... Corresponding feature vector From three-dimensional position vector Quaternion rotation features Composition, in which Represents three-dimensional Euclidean space. Represents a quaternion vector space; any two adjacent nodes and The edge features between are defined as follows The edge features are used to represent the local skeleton geometric relationship between node i and node j, where i and j are node indices and i ≠ j; The spatial attention-gated graph convolutional network is used to normalize the positional features to eliminate the influence of scaling; wherein, the normalized positional features are calculated as follows: , , ; in, This represents the normalized 3D position feature vector of the i-th node in the k-th frame, where the superscript k represents the time frame index and the subscript i represents the node index. This represents the geometric center of the positions of all nodes in the motion skeleton of the k-th frame; The overall scaling factor is N; the total number of nodes is N; ||·|| represents the Euclidean norm. The normalized position features and the original rotation features are concatenated to form the node input of the weighted graph structure, which is then fed into the spatial attention module for message propagation. Attention weights are dynamically calculated based on spatial and pose similarity between nodes, and the features of neighboring nodes are weighted and aggregated to obtain aggregated information; the specific calculation of the aggregated information is as follows: ; in, m k,i This represents the aggregated information of node i in the k-th frame. Represents the set of neighboring nodes of node i; This represents a message coding function consisting of two fully connected layers. To represent neighboring nodes in the k-th frame For nodes Attention weight coefficient; The spatial attention-gated graph convolutional network is used to update nodes by fusing the aggregated information with the original features through gated residual units. The calculation formula includes: , ; in, This represents the original node feature vector of the i-th node in the k-th frame; This is the gate vector, used to control the fusion ratio of aggregated information and residual information; This represents the Sigmoid activation function; U and U are trainable weight matrices. It is the bias vector; This represents element-wise multiplication. This represents the updated node feature vector; The joint angles of the robotic hand and the robotic arm are determined based on the updated nodes.

[0034] Optionally, determining the target force required to grasp the target object includes the following steps: The target force required to grasp the target object is predicted based on the scene image of the target object using a visual language model.

[0035] Optionally, the step of driving the robotic arm to grasp the target object based on the target force and the joint angle of the robotic arm includes the following steps: The predicted force is determined based on the joint angles of the robotic arm; The model predictive control is used to drive the robotic arm to grasp the target object based on the predicted force and the target force.

[0036] Optionally, determining the predicted force based on the joint angles of the robotic arm includes the following steps: A force prediction model is constructed based on a pre-defined dataset D; The preset dataset D is represented as follows: ; in, This is the joint control command angle. The joint angle of the robotic arm. For historical force data, M The total dimension of the joints; k Joint dimension; The joint control command angle and the joint angle of the robot are combined as the input to the force prediction model to obtain the predicted force output by the force prediction model.

[0037] Optionally, the step of using model predictive control to drive the robotic arm to grasp the target object based on the predicted force and the target force includes the following steps: In each control cycle, the error between the predicted force and the target force is calculated; wherein, the optimization objective of the error calculation is defined as: ; in, For the optimization objective, This refers to the joint command from the previous moment. For the target force, Used to adjust the predicted force to the target force. As a regularization mechanism, it suppresses angle jumps. Used to control smoothness and response speed; Among them, PyTorch is used to... A differentiable parameter is established, and gradient descent updates are achieved through several iterations using the Adam optimizer. The calculation formula includes: ; Updated optimal joint commands As the control output for the next moment.

[0038] The following sections will provide a detailed description and explanation of some optional embodiments of this application, using specific application examples.

[0039] 1. Overall architecture of the dual-arm dexterous remote control system.

[0040] The dual-arm dexterous telescopic system proposed in this solution tightly integrates vision-based motion redirection with adaptive force control, enabling cross-platform high-precision motion reproduction and intelligent grasping. The overall system flow is as follows: Figure 2 As shown, firstly, a visual capture algorithm (Frank Mocap) is used to perform 3D motion capture on the operator, obtaining complete information on key joints of the human body and hands, and processing it into 3D body and hand poses. The captured arm and hand movements are then separated for differentiated processing based on their respective characteristics: the arm has a large range of motion and fewer joints; the hand has a small range of motion, more joints, and higher precision requirements.

[0041] The separated motion data is input into a Spatial Attention Gated Graph Convolutional Network (SAG-GCN) for motion redirection. The network models the topological relationship between the human and robot joints through a graph structure and uses an attention mechanism to weight key nodes, thereby achieving high-precision, cross-platform human-robot motion mapping. The joint angles of the arm can be directly mapped to the robot arm for control after redirection, while the joint angles of the hand are input to the adaptive force control module for optimization and adjustment before physical grasping.

[0042] The adaptive force control module combines Visual Language Model (VLM) inference with Model Predictive Control (MPC)-based force adjustment to achieve intelligent grasping. The VLM analyzes scene images captured by an external camera to infer the grasping characteristics of the target object and the required grasping force. The initially estimated grasping force is combined with real-time force feedback from the dexterous hand and optimized online via MPC, enabling dynamic adjustment of joint commands and grasping force. This ensures the stability, compliance, and safety of the grasping action under different object and environmental conditions.

[0043] Through this process, the dual-arm dexterous teleoperation system can simultaneously achieve high-precision redirection and intelligent adaptive force control of arm and hand movements, providing a unified and reliable solution for dual-arm teleoperation in complex task scenarios.

[0044] 2. Motion redirection.

[0045] Motion redirection framework: The motion redirection network proposed in this embodiment adopts a symmetric encoder-decoder design, with each component consisting of three layers, such as... Figure 3 As shown, the encoder's first two layers employ a dual-stream structure to process arm and hand movements separately, addressing the differences in motion scale and joint complexity between the two. Arm movements have a large span and coarse granularity, while hand movements have a small span, many joints, and high precision requirements. The encoder's third layer is a feature fusion layer, responsible for integrating arm and hand features to achieve efficient cross-scale information interaction. The decoder also uses a dual-stream plus fusion structure, mapping the encoded features back to the corresponding arm and hand motion outputs to achieve high-fidelity motion redirection.

[0046] The core modules of the network include Spatial Basic Blocks (SBBs) and Gated Residual Blocks (GRBs). SBBs are primarily used for extracting skeleton topological features. Through the connection and message propagation of node features and edge attributes, they efficiently encode the spatial relationships between human and robot joints, providing fundamental feature representations for real-time motion retargeting. GRBs introduce attention mechanisms and gated residual structures, enhancing network feature selectivity while effectively suppressing noise accumulation, thus improving the stability and accuracy of the retargeting process. Through dual-stream input / output, cross-scale feature fusion, and the collaborative design of SBBs and GRBs, this network can simultaneously ensure the overall coordination of coarse-grained arm movements and the high-precision reproduction of fine hand movements when handling complex dual-arm motions, thereby significantly improving the accuracy of cross-platform motion retargeting and overall task execution efficiency.

[0047] Redirection Networks: In redirection networks, the human body and robot skeleton are modeled as a unified weighted graph structure to characterize the topological relationships and spatial constraints between joints. Frame motion skeleton representation ,in Let N be a set of nodes, with a total of N nodes, where each node corresponds to a joint in a human or robot. This is the set of skeleton connecting edges, used to represent the topological connection relationships between joints; This is a dynamic attention weight matrix, where each element represents the spatial attention weights between nodes, adaptively generated by the network during forward propagation. Each node... Corresponding feature vector From three-dimensional position vector Quaternion rotation features Composition, in which Represents three-dimensional Euclidean space. Represents a quaternion vector space; any two adjacent nodes and The edge features between are defined as follows The edge features are used to represent the local skeleton geometry relationship between node i and node j, where i and j are node indices and i ≠ j.

[0048] In the feature encoding stage, the core of the redirection network is a gated graph convolutional module based on spatial attention. Considering the significant scale differences between humans and robots, the network first normalizes the positional features to eliminate the influence of scale, while maintaining the original distribution of rotational features to preserve their physical meaning. Specifically, the normalized positional features are calculated as follows: , , ; in, This represents the normalized 3D position feature vector of the i-th node in the k-th frame, where the superscript k represents the time frame index and the subscript i represents the node index. This represents the geometric center of the positions of all nodes in the motion skeleton of the k-th frame; is the overall scale coefficient; N is the total number of nodes; ||·|| represents the Euclidean norm. The normalized position features are concatenated with the original rotation features to form the node input, which is then fed into the spatial attention module for message propagation. The attention weights are dynamically calculated based on the spatial-pose similarity between nodes, and the features of neighboring nodes are weighted and aggregated. ; in, m k,i This represents the aggregated information of node i in the k-th frame. Represents the set of neighboring nodes of node i; This represents a message coding function consisting of two fully connected layers. To represent neighboring nodes in the k-th frame For nodes The attention weight coefficients are then used. Subsequently, the network updates node representations by fusing aggregated messages with original features through gated residual units: , ; in, This represents the original node feature vector of the i-th node in the k-th frame; This is the gate vector, used to control the fusion ratio of aggregated information and residual information; This represents the Sigmoid activation function; U and U are trainable weight matrices. It is the bias vector; This represents element-wise multiplication. This represents the updated node feature vector. This gating mechanism effectively suppresses noise accumulation in deep networks and enhances the stability of feature propagation. 3. Adaptive force control module.

[0049] To achieve flexible and adaptive grasping of dexterous hands in multi-object scenarios, we designed a force feedback optimization module that combines large model inference with MPC-based principles. This module uses "target object recognition—force target estimation—online joint command optimization" as its core process, realizing a closed-loop mapping from visual perception to adaptive force control. For example... Figure 4As shown, the entire module can be divided into three parts: the large model inference stage, the joint angle-force prediction model construction stage, and the online adaptive optimization stage.

[0050] (1) Large-scale model-driven target force reasoning: At the beginning of the grasping task, the system first acquires images of the current grasping scene through a camera installed near the end effector. The images are input to the visual language large-scale model for rapid semantic understanding, thereby outputting two types of information: the semantic category of the target object and the appropriate clamping force range for the object. Unlike the traditional force threshold setting based on sensors or experience tables, this study uses the large-scale model to directly map the object category to the force requirement using its rich knowledge base. For example, when the object in hand is identified as a "plastic bottle", the model will give the conclusion of "recommended clamping force of about 400 grams", providing an initial value reference for subsequent adaptive force optimization. In this process, the key advantage of large-scale model reasoning lies in knowledge-driven prior force estimation. In complex multi-object scenarios, manual calibration or fixed thresholds often fail to balance safety and stability; while the large-scale model can directly give a reasonable target force based on visual input and background knowledge. This ensures that the system has good task feasibility during the initialization phase.

[0051] (2) Construction of Joint Angle-Force Prediction Model: After obtaining the target force, the system needs to be able to predict the current hand force based on the joint angles in order to evaluate the grasping effect in closed-loop optimization. This study adopts a data-driven approach, training the joint angle-force prediction model based on the collected historical motion-force data. The dataset format is as follows: ; in Joint control command angle For actual joint angle feedback, This corresponds to the six-dimensional force sensing data. During the modeling phase, the system uses the concatenation of control angle and actual angle features as input: ; The output is the predictive power. This embodiment uses a random forest regressor as the basic prediction model to capture nonlinear mapping relationships while balancing training efficiency and online inference speed. After training, the prediction model... It can quickly provide force estimates without relying on actual force sensor readings, providing a differentiable approximation for subsequent gradient optimization. This predictive model is equivalent to providing a "mechanical approximation model" for the MPC optimizer, enabling the system to directly infer the force response within the instruction space, thereby achieving online force control without the need for cumbersome dynamic modeling.

[0052] (3) Adaptive force optimization based on MPC: In the actual grasping process, the system needs to continuously correct the joint commands based on real-time feedback so that the force output by the hand smoothly approaches the target value. To this end, we designed an online optimization module with MPC style, embedding the prediction model into the gradient iteration loop to form a lightweight closed-loop control.

[0053] In each control cycle, the system starts from the joint command of the previous moment. Starting point, considering the current actual joint angles Calculate predictive power through predictive models The error is calculated between the target force and the objective force. The optimization objective is defined as: ; The first term ensures that the predictive power is close to the target power, while the second term acts as regularization to suppress large angle jumps. The system balances smoothness with response speed. In implementation, it utilizes PyTorch... By establishing a differentiable parameter and iterating through the Adam optimizer for several steps, a lightweight gradient descent update can be achieved. ; Updated optimal joint commands This serves as the control output for the next moment. Through this rolling optimization, the system possesses the characteristics of MPC: it can dynamically correct the deviation between the predicted force and the target force while ensuring motion continuity, achieving smooth and controllable adaptive force grasping.

[0054] In summary, this embodiment includes the following key technical solutions: (1) A dual-arm dexterous teleoperation system is proposed to solve the problems of cross-platform motion redirection and adaptive grasping in remote operation. It can perform accurate motion redirection across robots, while supporting stable and flexible object grasping, thus overcoming the limitations of existing methods.

[0055] (2) A vision-based motion redirection module is proposed. This module uses a spatial attention-gated graph convolutional network (SAG-GCN) to simulate the human-computer topology and adopts a dual-stream input-output design to simulate arm and hand movements, thereby achieving accurate cross-platform motion redirection.

[0056] (3) An adaptive grasping module is proposed to achieve flexible grasping. By integrating VLM with MPC, it can predict appropriate grasping forces and perform online optimization for stable and adaptive object grasping.

[0057] Traditional human-robot motion retargeting methods rely on costly motion capture equipment or large-scale paired data, making it difficult to achieve natural and accurate motion reproduction in cross-platform scenarios. Therefore, this embodiment introduces a Spatial Attention Gated Graph Convolutional Network (SAG-GCN) to decouple the arm and hand and model them as a graph structure, achieving end-to-end motion mapping through latent spatial encoding. This not only reduces the dependence on paired training samples but also effectively improves the naturalness, generalization, and real-time performance of motion retargeting.

[0058] Furthermore, addressing the issues of traditional grasping control relying on manual or predefined models for target force setting and lacking adaptive adjustment capabilities, this embodiment combines a Visual Language Model (VLM) with Model Predictive Control (MPC) to construct an adaptive grasping module with semantic understanding and force feedback optimization capabilities. The object category and characteristics are obtained through VLM inference to predict the target grasping force, and MPC is used for online optimization and adjustment, thereby achieving safe and compliant force-controlled grasping in different objects and complex environments. In summary, this embodiment significantly outperforms existing technologies in terms of motion redirection accuracy, adaptability, and grasping safety.

[0059] Reference Figure 5 This application also provides a dual-arm dexterous teleoperation system based on motion redirection and adaptive force, which can realize the above-mentioned dual-arm dexterous teleoperation method based on motion redirection and adaptive force. The system includes: The pose determination unit is used to determine the pose of a person's hand and arm based on a human image; The redirection unit is used to redirect the robot according to the pose of the human hand and the pose of the human arm using a spatial attention-gated graph convolutional network, thereby obtaining the joint angles of the robot's manipulator and the joint angles of the manipulator. A robotic arm drive unit is used to drive the robotic arm according to the joint angle of the robotic arm so that the robotic hand can grasp the target object; A robotic arm drive unit is used to determine the target force required to grasp the target object, and drive the robotic arm to grasp the target object based on the target force and the joint angle of the robotic arm.

[0060] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0061] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method of this application. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0062] It is understood that the content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the methods of this application, and the beneficial effects achieved are the same as those achieved by the methods of this application.

[0063] Figure 6 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 101 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 102 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 102 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 102 and is called and executed by the processor 101. Input / output interface 103 is used to implement information input and output; The communication interface 104 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 105 transmits information between various components of the device (e.g., processor 101, memory 102, input / output interface 103, and communication interface 104); The processor 101, memory 102, input / output interface 103 and communication interface 104 are connected to each other within the device via bus 105.

[0064] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of this application.

[0065] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0066] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0067] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0068] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0069] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0070] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0071] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0072] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0073] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0074] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0075] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0077] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A dual-arm dexterous teleoperation method based on motion reorientation and adaptive force, characterized in that, The method comprises the following steps: determine the human hand posture and the human arm posture according to the human body image; use a spatial attention gate graph convolution network to reorient the robot according to the human hand posture and the human arm posture, and correspondingly obtain the joint angle of the robot's mechanical hand and the joint angle of the mechanical arm; drive the mechanical arm according to the joint angle of the mechanical arm, so that the mechanical hand can grasp the target object; determine the target force required to grasp the target object, and drive the mechanical hand to grasp the target object according to the target force and the joint angle of the mechanical hand.

2. The dual-arm dexterous teleoperation method based on motion reorientation and adaptive forces of claim 1, wherein, The human hand posture and the human arm posture are determined according to the human body image, which comprises the following steps: obtain the human body image; use a visual capture algorithm to perform three-dimensional motion capture on the human body image to obtain joint information of the hand; correspondingly process the joint information of the hand into a 3D hand posture; wherein the 3D hand posture comprises the human hand posture and the human arm posture.

3. The dual-arm dexterous teleoperation method based on motion reorientation and adaptive forces of claim 1, wherein, The spatial attention gate graph convolution network is used to reorient the robot according to the human hand posture and the human arm posture, and correspondingly obtain the joint angle of the robot's mechanical hand and the joint angle of the mechanical arm, which comprises the following steps: modeling a human body and a robot skeleton as a weighted graph structure; wherein a first a motion skeleton representation of a frame is , is a node set, a total number of nodes is N, and the nodes correspond to joints of a human body or a robot, is a skeleton connection edge set, used to represent a topological connection relationship between joints; is a dynamic attention weight matrix, an element in the dynamic attention weight matrix representing a spatial attention weight between nodes, and being adaptively generated by the spatial attention gated graph convolution network in forward propagation; each node corresponds to a feature vector is composed of a three-dimensional position vector and a quaternion rotation feature , wherein represents a three-dimensional Euclidean space, represents a quaternion vector space; an edge feature between any two adjacent nodes is defined as , the edge feature being used to represent a local skeleton geometric relationship of a node i pointing to a node j, wherein i and j are node indices and i ≠ j;​ normalize the position feature using the spatial attention gate graph convolution network to eliminate the scale effect; wherein the normalized position feature is calculated as: ， ， ； wherein, denotes the normalized 3D position feature vector of the i-th node in the k-th frame, the superscript k denotes the time frame index, and the subscript i denotes the node index; denotes the geometric center of all node positions in the k-th frame motion skeleton; is the overall scale coefficient; N is the total number of nodes; and ||·|| denotes the Euclidean norm. The normalized position feature and the original rotation feature are spliced to form the node input of the weighted graph structure, which is sent to the spatial attention module for message propagation; dynamically calculate the attention weight through the spatial and posture similarity between nodes, and weight aggregate the neighbor node features to obtain aggregation information; the aggregation information is calculated as follows: ； wherein, m k,i denotes the aggregated information of node i in the kth frame, denotes the neighbor node set of node i; denotes a message encoding function composed of a two-layer fully connected network, denotes the neighbor node set of node i in the kth frame, the attention weight coefficient of node in the kth frame. The spatial attention gate graph convolution network fuses the aggregation information and the original feature through a gate residual unit to update the node, and the calculation formula comprises: ， ； wherein, represents the original node feature vector of the i-th node in the k-th frame; is a gating vector, which is used to control the fusion ratio of the aggregated information and the residual information; represents a Sigmoid activation function; and U are trainable weight matrices, is a bias vector; represents an element-wise multiplication operation; represents the updated node feature vector; determine the joint angle of the mechanical hand and the joint angle of the mechanical arm according to the updated node.

4. The dual-arm dexterous teleoperation method based on motion reorientation and adaptive forces of claim 1, wherein, The target force required to grasp the target object is determined, which comprises the following steps: use a visual language model to predict the target force required to grasp the target object according to the scene image of the target object.

5. The dual-arm dexterous teleoperation method based on motion reorientation and adaptive forces of claim 1, wherein, The mechanical hand is driven to grasp the target object according to the target force and the joint angle of the mechanical hand, which comprises the following steps: determine the predicted force according to the joint angle of the mechanical hand; use model predictive control to drive the mechanical hand to grasp the target object according to the predicted force and the target force.

6. The dual-arm dexterous teleoperation method based on motion reorientation and adaptive forces of claim 5, wherein, The predicted force is determined according to the joint angle of the mechanical hand, which comprises the following steps: construct a force prediction model based on a preset data set D; The preset data set D is represented as: ； wherein, is a joint control command angle, is a joint angle of the robot, is historical force data, M is a total number of joints; k is a joint dimension; concatenate the joint control instruction angle and the joint angle of the mechanical hand as the input of the force prediction model, and obtain the predicted force output by the force prediction model.

7. The dual-arm dexterous teleoperation method based on motion reorientation and adaptive forces of claim 5, wherein, The mechanical hand is driven to grasp the target object according to the predicted force and the target force using model predictive control, which comprises the following steps: In each control cycle, calculate the error between the predicted force and the target force; wherein the optimization target of error calculation is defined as: ； wherein, is the optimization target, is the joint command of the previous time instant, is the target force, for adjusting the predicted force to the target force, as a regularization to suppress angle jumps, for controlling the smoothness and response speed; where the PyTorch is used to The differentiable parameterization is established, and the gradient descent update is realized by iterating several steps through the Adam optimizer. The calculation formula includes: ； Updated optimal joint commands as the control output for the next time.

8. A dual-arm dexterous teleoperation system based on motion reorientation and adaptive force, characterized in that, The system comprises: A posture determination unit is configured to determine a human hand posture and a human arm posture according to a human body image; A redirection unit is configured to redirect a robot according to the human hand posture and the human arm posture by using a spatial attention gate graph convolution network, so as to obtain joint angles of a mechanical hand and joint angles of a mechanical arm of the robot; A mechanical arm driving unit is configured to drive the mechanical arm according to the joint angles of the mechanical arm, so that the mechanical hand grasps a target object; A mechanical hand driving unit is configured to determine a target force required for grasping the target object, and drive the mechanical hand to grasp the target object according to the target force and the joint angles of the mechanical hand.

9. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the method in any one of claims 1 to 7.

Images