Dexterous hand teleoperation method based on vision-touch fusion

Abstract

The invention discloses a dexterous hand teleoperation method based on vision-touch fusion. The method comprises the following steps: 1) vision detection; the method comprises the following steps of (1) carrying out remote control, (2) carrying out redirection, (3) carrying out tactile grabbing, (4) carrying out vision-tactile interaction, (5) carrying out a communication process and (6) carrying out dexterous hand.The method can stably map human hand actions under the complex vision condition, safe and reliable grabbing control is achieved through tactile feedback, meanwhile, multiple dexterous hand configurations and cross-platform deployment are supported, and the precision, robustness and practicability of teleoperation can be improved.

Description

Technical Field

[0001] This invention relates to human-computer interaction technology, and more particularly to a remote operation technology for multi-finger dexterous hands that achieves high robustness, high precision motion redirection and closed-loop grasping control based on a fusion of monocular visual gesture recognition, fingertip tactile feedback and adaptive error weighting mechanism. Specifically, it is a dexterous hand remote operating system and grasping method based on visual-tactile fusion. Background Technology

[0002] With the continuous development of robotics technology, dexterous hands, as end effectors with multiple degrees of freedom and high flexibility, have been widely used in various fields such as remote control, service robots, medical assistance, and hazardous environment operations. Compared with traditional grippers, dexterous hands can simulate the fine movements of human hands, realizing operations such as grasping, carrying, rotating, and inserting complex objects, significantly improving the robot's operational capabilities and intelligence level in unstructured environments.

[0003] In the telemanipulation of dexterous hands, accurately and in real-time mapping the operator's hand movements to the robot's end effector is crucial for achieving natural and efficient human-computer interaction. Current mainstream solutions primarily rely on visual perception or wearable sensing devices to acquire hand posture information and then map it to the joint space of the dexterous hand using motion redirection algorithms. While these methods achieve motion reproduction to some extent, they still face numerous challenges in practical applications.

[0004] While visual perception offers advantages such as being non-contact, low-cost, and flexible in deployment, the accuracy of key point detection significantly decreases in complex scenarios such as hand occlusion, changes in lighting, background interference, or overlapping hands. This leads to increased retargeting errors, affecting operational precision and system stability. Furthermore, existing visual teleoperation systems generally lack tactile feedback mechanisms, preventing operators from sensing contact force or object state during grasping. When grasping unknown objects (such as fragile, flexible, or slippery items), it's difficult to determine if the gripping force is appropriate, easily causing the object to slip or be damaged, thus limiting the system's practicality and safety.

[0005] Meanwhile, existing teleoperation systems also have significant shortcomings in terms of versatility and scalability. Most systems are custom-developed for specific robot platforms or task scenarios, resulting in strong hardware dependencies, high algorithm coupling, and difficulties in migration. When changing the dexterous hand model, camera configuration, or operating environment, recalibration, model training, or parameter adjustment are often required, making rapid cross-platform deployment and promotion difficult. In addition, traditional redirection algorithms typically assign equal weights to errors at all key points, failing to adequately consider detection confidence or spatial consistency. This leads to unstable mapping results in the presence of false detections, missing values, or jitter, potentially causing sudden joint changes or trajectory anomalies in the robot, affecting control smoothness and safety.

[0006] Therefore, there is an urgent need for a new type of teleoperation system that integrates visual and tactile feedback, has adaptive error adjustment capabilities, and supports multi-platform deployment, in order to improve the robustness, accuracy, and practicality of dexterous hands in complex environments and promote their widespread application and intelligent development in real-world scenarios. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a dexterous hand teleoperation method based on vision-tactile fusion. This method can stably map human hand movements under complex visual conditions and achieve safe and reliable grasping control through tactile feedback. It also supports various dexterous hand configurations and cross-platform deployment, improving the accuracy, robustness, and practicality of teleoperation.

[0008] The technical solution to achieve the objective of this invention is: A dexterous hand remote control method based on vision-tactile fusion includes the following steps: 1) Visual detection: An RGB-D camera is used to simultaneously acquire color images of the operator's hand. The images are input into the MediaPipe gesture recognition framework. Using its built-in 3D reconstruction algorithm based on geometric priors and neural networks, the coordinates of 21 three-dimensional key points of the hand are directly estimated and output. Then, using the MANO right-hand coordinate system as the reference, the coordinates of 21 three-dimensional key points with the wrist as the origin (0 point) and the palm facing upward are output for subsequent pose estimation and gesture recognition. The 21 three-dimensional key points of the hand are: points 1, 2, 3, and 4 of the thumb from bottom to top with the wrist as the origin (0 point) and the palm facing upward; points 5, 6, 7, and 8 of the index finger from bottom to top; points 9, 10, 11, and 12 of the middle finger from bottom to top; points 13, 14, 15, and 16 of the ring finger from bottom to top; and points 17, 18, 19, and 20 of the little finger from bottom to top. 2) Retargeting: Using the coordinates of 21 3D key points of the hand obtained by the visual detection algorithm as input, and different DH values ​​obtained according to the dexterous hand URDF model, the adaptive error weight retargeting algorithm AEWD-Retargeting is used to generate angle commands for each joint of the dexterous hand, including: 2-1) Calculate the error vector for each key point as shown in formula (1): (1), in Indicates the robot hand at time step The joint position is expressed in radians; It is based on the detected number of people's hands. One key point vector; It is the first A forward kinematic mapping function, whose input is the robot hand joint position. The output is the vector of the i-th key point corresponding to the dexterous hand, where a is a scaling factor used to compensate for the size difference between the human hand and the robot hand; 2-2) Smoothing term suppresses adjacent time steps The joint transitions between points make the trajectory smoother, as shown in formula (2): (2), Where β is the smoothing term coefficient, which controls the smoothing intensity; the larger β is, the smoother the trajectory. 2-3) Introduce adaptive error weights into the error term. As shown in formula (3): (3), in and The first corresponding to human hand and dexterous hand One key point vector; These are hyperparameters used to control the rate of weight decay. =1.0; To avoid small constants with a denominator of zero, ; 2-4) The weights are normalized to the mean as shown in formula (4): (4), Where N is the number of key points of the current dexterous hand; 2-5) The objective optimization function is constructed based on the normalized adaptive error weights as shown in formula (5): ;

[0009] Indicates the robot hand at time step The joint position; It is calculated based on the key points of the detected human hand. One key point vector; It is the first A forward kinematic mapping function, whose input is the robot hand joint position. The output is the vector of the i-th key point corresponding to the dexterous hand; a is a scaling factor that compensates for the size difference between the human hand and the robot hand and achieves alignment between them. These are the lower and upper limits of the joint position, respectively; This represents the normalized error weights; Indicates the coefficient of the smoothing term; 3) Tactile Grasping: Based on tactile data from the PST tactile sensor at the dexterous fingertip, it completes tactile baseline calibration, contact detection, posture approximation, and adaptive force-controlled grasping based on PI control, including: 3-1) Basic pressure calibration: The calibration process involves collecting data before... The average pressure output by the frame sensor is used to calculate the basic pressure of each finger as shown in formula (6): (6),

[0010] in This is the basic pressure applied to the fingers; It is the first The sample stress value; N is the number of samples; 3-2) Contact detection: Based on the pressure threshold, when the real-time pressure exceeds the base pressure plus the contact threshold, it is determined that the initial contact has occurred, as shown in formula (7): (7), in This represents the current real-time pressure value of the finger. This is the contact threshold. At this point, the system enters the force control preparation stage and records the pressure and joint position at the first contact. 3-3) Grabbing: The system loads a predefined object grasping posture library, which contains the target joint postures of a specific object. Taking the current object name as input, it extracts the target joint angles from the library and gradually approximates these postures. For each joint, its new joint angles are calculated. Adjust as shown in formula (8): (8),

[0011] Where Δq is the maximum joint angular displacement in a single step; This is the difference between the target position and the current position; The target location of the joint; This represents the current position of the joint. 3-4) Fine-tuning of joint angles: for each finger The PI (proportional-integral) control method is adopted as shown in formula (9): (9), in The current pressure of the tactile sensor; For target pressure; This refers to the joint angle adjustment amount, used to gradually close the fingers; These are the proportional and integral gains, respectively. (Proportional gain) The integral gain is 0.8. It is 0.15; 4) Visual-tactile interaction: When the size of a fingertip tactile data exceeds the basic pressure plus contact threshold, the system switches from visual mode to tactile mode; when the visual algorithm detects that the human hand is in an open gesture, the system switches from tactile mode to visual teleoperation. 5) Communication process: The communication process is based on the ROS distributed communication framework. Data interaction is completed by publishing and subscribing to ROS topics. The server acts as the host computer to publish joint trajectory message topics, and the dexterous hand edge device acts as the slave computer to subscribe to joint trajectory topics and publish fingertip PST tactile data topics. 6) Dexterous hand execution: Based on the joint trajectory of the redirected output and the tactile closed-loop feedback, the shadow five-finger dexterous hand completes the teleoperation and grasping control.

[0012] This technical solution overcomes the shortcomings of existing dexterous hand remote control systems in terms of visual robustness, lack of tactile feedback, insufficient versatility, and weak noise resistance. Compared with existing technologies, it has the following beneficial effects: This technical solution provides a unified vision-tactile fusion dexterous hand teleoperation system. For motion retargeting, it employs the AWD-Retargeting algorithm, using an adaptive error weighting mechanism to suppress noise and outlier interference, effectively improving robustness and accuracy in scenarios with occlusion, noise, and scale differences. For tactile interaction, it uses an adaptive tactile grasping algorithm based on a PST sensor, achieving closed-loop force-controlled grasping for the first time in monocular vision teleoperation, enhancing grasping stability and object adaptability. Simultaneously, combined with ROS multi-machine communication, it achieves distributed collaboration between visual perception, motion retargeting, and tactile control, improving the system's real-time performance and scalability.

[0013] This method can stably map human hand movements under complex visual conditions and achieve safe and reliable grasping control through haptic feedback. It also supports various dexterous hand configurations and cross-platform deployment, which can improve the accuracy, robustness and practicality of teleoperation. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the method in the embodiment; Figure 2 This is a schematic diagram illustrating the detection of 21 2D key points on the hand in the embodiment; Figure 3 This is a schematic diagram of the adaptive error weight redirection algorithm in the embodiment; Figure 4 This is a schematic diagram of the dexterous hand tactile grasping process in the embodiment; Figure 5 This is a schematic diagram of the ROS communication structure in the embodiment; Figure 6This is a URDF visualization diagram of a common dexterous hand shape in the embodiments; Figure 7 This is a schematic diagram comparing the accuracy of different redirection algorithms in the embodiment; Figure 8 This is a schematic diagram illustrating the use of a dexterous hand to perform dexterous tasks in the embodiment. Figure 9 This is a schematic diagram of a grasping scenario using a dexterous hand remote operation in the embodiment. Detailed Implementation

[0015] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0016] Example: Reference Figure 1 A dexterous hand remote control method based on vision-tactile fusion includes the following steps: 1) Visual Inspection: An RGB-D camera is used to simultaneously acquire color images of the operator's hand. These images are then input into the MediaPipe gesture recognition framework. Utilizing its built-in 3D reconstruction algorithm based on geometric priors and neural networks, the coordinates of 21 key 3D points on the hand are directly estimated and output. Figure 2 As shown, the 21 3D key points of the hand are: with the wrist as the origin (0 point) and the palm facing upwards, the thumb is set to points 1, 2, 3, and 4 from bottom to top; the index finger is set to points 5, 6, 7, and 8 from bottom to top; the middle finger is set to points 9, 10, 11, and 12 from bottom to top; the ring finger is set to points 13, 14, 15, and 16 from bottom to top; and the little finger is set to points 17, 18, 19, and 20 from bottom to top. Then, using the MANO right-hand coordinate system as the reference, the coordinates of the 21 3D key points with the wrist as the origin (0 point) and the palm facing upwards are output for subsequent pose estimation and gesture recognition. 2) Retargeting: Using the coordinates of 21 3D key points of the hand obtained by the visual detection algorithm as input, and different DH values ​​obtained according to the dexterous hand URDF model, the adaptive error weight retargeting algorithm (AEWD-Retargeting) is used to generate angle commands for each joint of the dexterous hand, such as... Figure 3 As shown, it includes: 2-1) Calculate the error vector for each key point as shown in formula (1): (1), in Indicates the robot hand at time step The joint position is expressed in radians; It is based on the detected number of people's hands. One key point vector; It is the first A forward kinematic mapping function, whose input is the robot hand joint position. The output is the vector of the i-th key point corresponding to the dexterous hand, where a is a scaling factor used to compensate for the size difference between the human hand and the robot hand; 2-2) Smoothing term suppresses adjacent time steps The joint transitions between points make the trajectory smoother, as shown in formula (2): (2), Where β is the smoothing term coefficient, which controls the smoothing intensity; the larger β is, the smoother the trajectory. 2-3) Introduce adaptive error weights into the error term. As shown in formula (3): (3), in and The first corresponding to human hand and dexterous hand One key point vector; These are hyperparameters used to control the rate of weight decay. =1.0; To avoid small constants with a denominator of zero, ; 2-4) To improve time consistency, the weights are normalized to the mean as shown in formula (4): (4), Where N is the number of key points of the current dexterous hand; 2-5) The objective optimization function is constructed based on the normalized adaptive error weights as shown in formula (5): ;

[0017] in Indicates the robot hand at time step The joint position is expressed in radians; It is calculated based on the key points of the detected human hand. One key point vector; It is the first A forward kinematic mapping function, whose input is the robot hand joint position. Specifically, depending on the definition of the dexterous hand manufacturer, the output is the vector of the i-th key point corresponding to the dexterous hand; 'a' is a scaling factor that compensates for the size difference between the human hand and the robot hand to achieve alignment between the two. These are the lower and upper limits of the joint position, respectively; This represents the normalized error weights; Indicates the coefficient of the smoothing term; 3) Tactile Grasping: Based on tactile data from the PST tactile sensor at the dexterous fingertip, it completes tactile baseline calibration, contact detection, posture approximation, and adaptive force-controlled grasping based on PI control, such as... Figure 4 As shown, pressure calibration is recorded by detecting the pressure and joint position at contact; posture approach is achieved by loading the approaching target posture from the posture library and performing tactile fine-tuning; PI force control achieves grasping by adjusting the joint angle through proportional-integral (PI) control, including: 3-1) Basic pressure calibration: The calibration process involves collecting data before... The average pressure output by the frame sensor is used to calculate the basic pressure of each finger as shown in formula (6): (6),

[0018] in This is the basic pressure applied to the fingers; It is the first The base pressure value is the sample pressure value; N is the number of samples. After calibration, the base pressure is used for subsequent contact testing and target pressure calculation. 3-2) Contact detection: Based on the pressure threshold, when the real-time pressure exceeds the base pressure plus the contact threshold, it is determined that the initial contact has occurred, as shown in formula (7): (7), in This represents the current real-time pressure value of the finger. This is the contact threshold. At this point, the system enters the force control preparation stage and records the pressure and joint position at the first contact. 3-3) After contact is established, the grasping algorithm is invoked for grasping: The system loads a predefined object grasping posture library, which contains the target joint postures of specific objects. Taking the current object name as input, the target joint angles are extracted from the library and gradually approximated to these postures. For each joint, its new joint angles are calculated. Adjust as shown in formula (8): (8),

[0019] Where Δq is the maximum joint angular displacement in a single step; This is the difference between the target position and the current position; The target location of the joint; This represents the current position of the joint. 3-4) After reaching the predefined grasping posture, fine-tune the joint angles based on the pressure error: for each finger The PI (proportional-integral) control method is adopted as shown in formula (9): (9), in The current pressure of the tactile sensor; For target pressure; This refers to the joint angle adjustment amount, used to gradually close the fingers; These are the proportional and integral gains, respectively. (Proportional gain) The integral gain is 0.8. It is 0.15; 4) Visual-tactile interaction: When the size of a fingertip tactile data exceeds the basic pressure plus contact threshold, the system switches from visual mode to tactile mode; when the visual algorithm detects that the human hand is in an open gesture, the system switches from tactile mode to visual teleoperation. 5) Communication process: such as Figure 5 As shown, the server is a host equipped with an NVIDIA 4090 graphics card, the edge device is a NUC box, the communication process is based on the ROS distributed communication framework, and the data interaction is completed by publishing and subscribing to ROS topics. The server acts as the host computer to publish joint trajectory message topics, and the dexterous hand edge device acts as the slave computer to subscribe to joint trajectory topics and publish fingertip PST haptic data topics. 6) Dexterous hand execution: Based on the joint trajectory of the redirected output and the tactile closed-loop feedback, the shadow five-finger dexterous hand completes the teleoperation and grasping control.

[0020] This example uses the overall system architecture shown in Figure 1, which includes five major components: visual detection, motion redirection, tactile grasping, ROS communication, and dexterous hand execution. All experiments in this example are conducted on the same computing platform, which is equipped with an Intel Core i9-13900 processor and an NVIDIA RTX 4080 Ti graphics card, running Ubuntu 20.04.6 LTS. The hardware platform in this example uses the ShadowHand-G3M3 five-finger dexterous hand as the execution end and is equipped with an Intel RealSense D435i depth camera to collect RGB-D data. Since the depth camera is easily affected by lighting, all experiments are conducted in an environment with uniform lighting and sufficient brightness to ensure stable key point detection and redirection performance.

[0021] To verify the noise resistance and redirection accuracy of the proposed method, multiple sets of different noise intensity conditions were selected for evaluation. A comparative experiment was designed to test and compare the proposed method with the benchmark method Dex-Retargeting. In the experiment, to simulate the input keypoint disturbances that may be caused by hand occlusion, visual detection noise, and communication link jitter, independent zero-mean Gaussian noise was added to all 3D keypoint locations. The noise standard deviation σ was set to the commonly used {0.0, 0.5, 1.0, 1.5, 2.0}, thereby evaluating the redirection accuracy performance of the two algorithms under different noise intensities. To verify the generality of the method presented in this example, the experiment was tested on six common dexterous hand models, including Shadow, Inspire, Ability, Leap, SVH, and Allegro, as shown in Figure 6. Figure 6 This is a visualization diagram of common dexterous hand types using URDF. From left to right and top to bottom, they are: Shadow, Inspire, Ability, Leap, SVH, and Allegro. For each hand type, AEWD-Retargeting and Dex-Retargeting are run respectively, and the average L2 norm error between the optimized robot hand keypoint vector and the target keypoint vector is calculated as an indicator of retargeting accuracy. To ensure the reproducibility and fairness of the experiment, low-pass filtering (low_pass_alpha = 0) and scaling factor (scaling_factor = 1.0) were disabled in all tests to focus on the efficiency of single-step optimization. To ensure the reproducibility of the experiment, the same random seed was set in all tests. like Figure 7 As shown, Figure 7 The horizontal axis represents the standard deviation of zero-mean Gaussian noise, with the intensity increasing in 0.5 increments from 0 to 2.0; the vertical axis displays the average error value, with the blue curve representing the average error of the baseline method and the red curve corresponding to the error performance of our proposed AEWD-Retargeting method. Figure 7 The retargeting accuracy of AEWD-Retargeting and the benchmark method Dex-Retargeting is compared under different Gaussian noise conditions. It can be seen that in all the dexterous hand patterns tested, the optimizer of this method is significantly better than Dex-Retargeting at all noise levels. Under low noise conditions (σ = 0 or 0.5), AEWD-Retargeting achieves near-zero error retargeting. When the noise intensity increases further (σ = 1.0, 1.5, or 2.0), the optimization accuracy of this method remains 2 to 5 times that of the baseline method. In experiments with the Inspire Hand and SVH Hand, AEWD-Retargeting exhibits near-zero error performance at all noise levels, further demonstrating the excellent generalization and stability of this method on dexterous hands with different structures and degrees of freedom.

[0022] The experimental results above fully demonstrate that the adaptive error weighting mechanism of the proposed method can effectively suppress the influence of key point perturbations under both low and high noise conditions, achieving highly stable, robust motion repositioning performance with cross-hand type versatility.

[0023] To verify the capability of the method presented in this example for dexterous manipulation tasks, two dexterous manipulation tasks are designed as follows: Figure 8 As shown, the tasks are typing on the keyboard and unscrewing the bottle cap. The typing task requires the operator to type "hello" and "hand" on a standard keyboard in sequence, and the time required to type "hello" and "hand" is calculated for each. The unscrewing task requires the operator to complete the screw-top operation of the bottle cap, repeating it 5 times and taking the average time. The average time to complete the task is used as the evaluation index.

[0024] Table 1: Comparison of average duration of dexterity tasks , The experimental results are shown in Table 1. Compared with the baseline method, the proposed method significantly shortened the task completion time in both dexterity tasks. In the keyboard typing task, the average completion time decreased from 207 seconds to 131 seconds and from 193 seconds to 90 seconds, respectively, representing improvements of approximately 36.1% and 53.4%. In the cup-twisting task, the average completion time was shortened from 90 seconds to 50.8 seconds, an improvement of 43.6%. This performance improvement is mainly due to the adaptive error weighting mechanism introduced in the proposed method, which makes the gesture mapping process more accurate and stable, thereby significantly reducing erroneous actions caused by keypoint offset or posture jitter. This advantage is particularly evident in the keyboard input task: the probability of generating incorrect keystrokes during input is significantly lower than that of the baseline method, effectively reducing the additional time overhead caused by repeated attempts and corrections. To verify the overall performance of the method presented in this example, we designed a grasping task for four typical types of objects, such as... Figure 9As shown in the figure, (a) is grasping a banana, (b) is grasping a baseball, (c) is grasping a water cup, and (d) is grasping a bottle. Before the formal test, the subjects underwent about 3 to 5 minutes of operation familiarization training to ensure that they could successfully understand and master the basic interaction mode of the system. The main evaluation indicators of this experiment were the success rate and average time of the subjects to complete the "grasp-manipulate-release" task. The experimental procedure is as follows: The subjects manipulated their real five-finger dexterity hand with open gestures to grasp the target object placed on the platform. After grasping, the subjects were required to drag, press and shake the object in multiple directions to examine the performance of this method in terms of grasping stability, pressure regulation ability and anti-slip performance. Finally, at the end of the task, the subjects were required to manipulate their dexterity hand to release the object. Four typical objects were selected as test objects, including water bottles, mugs, plastic bananas and spherical objects. Each type of object was repeated 10 times. The task judgment rules are as follows: If the subject fails to complete the grasping action within 60 seconds, the experiment is judged as a failure; if the dexterous hand has completed the grasp but the object slips during subsequent dragging, pressing, or shaking, it is also judged as a failure. Throughout the experiment, when the dexterous fingertip touches the object surface, this method automatically switches from visual mode to tactile mode; the system then loads the corresponding target posture from the predefined grasping posture library according to the object category and executes the grasping action. After the grasping is completed, when the vision detects that the operator's hand is in an open posture, the system will switch from tactile mode back to visual mode, allowing the subject to regain remote control. See Table 2 for details. Table 2 is a comparative experiment of different grasping tasks. Succ.Rate refers to the success rate, Aver. Duration refers to the average time to successfully grasp an object, "fall off" indicates that the object falls during dragging, pressing, and shaking tests, and "readjust" indicates that the grasping process has been adjusted twice. Table 2 , Table 2 summarizes the overall results of this usability experiment. Compared with the baseline method, this method achieved a higher grasping success rate and a shorter average task completion time on all test objects. Taking irregularly shaped and slippery objects such as plastic bananas as an example, the success rate of the traditional method was only 60%, while this method achieved a 100% success rate. In the grasping task of spherical objects, the success rate increased from 70% of the baseline method to 90%. For mugs with large openings and wide diameters, the success rate of the baseline method was only 50%. This is because such objects require the subject to first complete the abduction and repositioning of the thumb, and the traditional repositioning method is prone to action deviation when handling such complex posture transitions. This method improved the stability of posture mapping through an adaptive weighting mechanism, increasing the success rate by 30%.

[0025] In terms of task efficiency, the method in this example also shows a significant advantage in average completion time, reducing overall time consumption by about 20-40%, which fully demonstrates its improvement effect on operational continuity and motion stability. Furthermore, as shown in Table 3, following the AnyTeleop evaluation paradigm, this example method was compared with existing vision-based multi-finger dexterous hand teleoperations from three dimensions: (i) sensor requirements; (ii) robot-related support; and (iii) applicable scenarios. The results show that this example method is one of the few frameworks that can be adapted to multiple dexterous hands at the same time, and it is also the only system that realizes vision-tactile interaction and tactile grasping functions within a unified architecture.

[0026] Table 3: Comparison of Vision-Based Remote Operating Systems .

Claims

1. A dexterous hand remote operation method based on vision-tactile fusion, characterized in that, Includes the following steps: 1) Visual detection: An RGB-D camera is used to simultaneously acquire color images of the operator's hand. The images are input into the MediaPipe gesture recognition framework. Using its built-in 3D reconstruction algorithm based on geometric priors and neural networks, the coordinates of 21 three-dimensional key points of the hand are directly estimated and output. Then, using the MANO right-hand coordinate system as the reference, the coordinates of 21 three-dimensional key points with the wrist as the origin (0 point) and the palm facing upward are output for subsequent pose estimation and gesture recognition. The 21 three-dimensional key points of the hand are: points 1, 2, 3, and 4 of the thumb from bottom to top with the wrist as the origin (0 point) and the palm facing upward; points 5, 6, 7, and 8 of the index finger from bottom to top; points 9, 10, 11, and 12 of the middle finger from bottom to top; points 13, 14, 15, and 16 of the ring finger from bottom to top; and points 17, 18, 19, and 20 of the little finger from bottom to top. 2) Retargeting: Using the coordinates of 21 3D key points of the hand obtained by the visual detection algorithm as input, and different DH values ​​obtained according to the dexterous hand URDF model, the adaptive error weight retargeting algorithm AEWD-Retargeting is used to generate angle commands for each joint of the dexterous hand, including: 2-1) Calculate the error vector for each key point as shown in formula (1): （1）， in Indicates the robot hand at time step The joint position is expressed in radians; It is based on the detected number of people's hands. One key point vector; It is the first A forward kinematic mapping function, whose input is the robot hand joint position. The output is the vector of the i-th key point corresponding to the dexterous hand, where a is a scaling factor used to compensate for the size difference between the human hand and the robot hand; 2-2) Smoothing term suppresses adjacent time steps The joint transitions between points make the trajectory smoother, as shown in formula (2): （2）， Where β is the smoothing term coefficient, which controls the smoothing intensity; the larger β is, the smoother the trajectory. 2-3) Introduce adaptive error weights into the error term. As shown in formula (3): （3）， in and The first corresponding to human hand and dexterous hand One key point vector; These are hyperparameters used to control the rate of weight decay. =1.0; To avoid small constants with a denominator of zero, ; 2-4) The weights are normalized to the mean as shown in formula (4): （4）， Where N is the number of key points of the current dexterous hand; 2-5) The objective optimization function is constructed based on the normalized adaptive error weights as shown in formula (5): ； in Indicates the robot hand at time step The joint position is expressed in radians; It is calculated based on the key points of the detected human hand. One key point vector; It is the first A forward kinematic mapping function, whose input is the robot hand joint position. The output is the vector of the i-th key point corresponding to the dexterous hand; a is a scaling factor that compensates for the size difference between the human hand and the robot hand and achieves alignment between them. These are the lower and upper limits of the joint position, respectively; This represents the normalized error weights; Indicates the coefficient of the smoothing term; 3) Tactile Grasping: Based on tactile data from the PST tactile sensor at the dexterous fingertip, it completes tactile baseline calibration, contact detection, posture approximation, and adaptive force-controlled grasping based on PI control, including: 3-1) Basic pressure calibration: The calibration process involves collecting data before... The average pressure output by the frame sensor is used to calculate the basic pressure of each finger as shown in formula (6): （6）， in This is the basic pressure applied to the fingers; It is the first The sample stress value; N is the number of samples; 3-2) Contact detection: Based on the pressure threshold, when the real-time pressure exceeds the base pressure plus the contact threshold, it is determined that the initial contact has occurred, as shown in formula (7): （7）， in This represents the current real-time pressure value of the finger. This is the contact threshold. At this point, the system enters the force control preparation stage and records the pressure and joint position at the first contact. 3-3) Grabbing: The system loads a predefined object grasping posture library, which contains the target joint postures of a specific object. Taking the current object name as input, it extracts the target joint angles from the library and gradually approximates these postures. For each joint, its new joint angles are calculated. Adjust as shown in formula (8): （8）， Where Δq is the maximum joint angular displacement in a single step; This is the difference between the target position and the current position; The target location of the joint; This represents the current position of the joint. 3-4) Fine-tuning of joint angles: for each finger The PI (proportional-integral) control method is adopted as shown in formula (9): （9）， in The current pressure of the tactile sensor; For target pressure; This refers to the joint angle adjustment amount, used to gradually close the fingers; These are the proportional and integral gains, respectively. (Proportional gain) The integral gain is 0.

8. It is 0.15; 4) Visual-tactile interaction: When the size of a fingertip tactile data exceeds the basic pressure plus contact threshold, the system switches from visual mode to tactile mode; when the visual algorithm detects that the human hand is in an open gesture, the system switches from tactile mode to visual teleoperation. 5) Communication process: The communication process is based on the ROS distributed communication framework. Data interaction is completed by publishing and subscribing to ROS topics. The server acts as the host computer to publish joint trajectory message topics, and the dexterous hand edge device acts as the slave computer to subscribe to joint trajectory topics and publish fingertip PST tactile data topics. 6) Dexterous hand execution: Based on the joint trajectory of the redirected output and the tactile closed-loop feedback, the shadow five-finger dexterous hand completes the teleoperation and grasping control.

Images