Method and device for redirecting human hand actions to a heterogeneous robotic arm in a multi-task scenario

Abstract

The present invention provides a method and device for redirecting human hand movements to a heterogeneous robotic hand in a multi-task scenario, which relates to the fields of robotics technology and intelligent control technology. The method includes: acquiring visual data and tactile data; classifying task types using a pre-trained Perceiver IO model to obtain task type encodings; correcting the joint weights of the robotic hand fingers according to a physical constraint dynamic correction mechanism to obtain normalized finger weights; extracting key points from the human hand and calculating the vectors between each joint of the human hand through the geometric vector method; performing driven joint constraint processing according to the established joint mapping matrix, calculating the adapted Jacobian matrix, and obtaining the adapted and standardized kinematic model of the robotic hand; performing non-linear optimization on the vectors between human fingers and robotic hand fingers to obtain the joint angles of the robotic hand, so that the robotic hand completes action redirection. Using the present invention can improve the generalization ability across tasks and heterogeneous models.

Description

Technical Field

[0001] The present invention relates to the field of robotics and intelligent control technology, and in particular to a method and device for redirecting human hand movements to heterogeneous robotic hands in a multi-task scenario. Background Art

[0002] Motion redirection technology can not only map the complex movements of human hands to robot hands to achieve efficient and accurate motion transfer, but also show unique advantages in data collection and remote control. Using motion redirection technology, large-scale training data sets can be built from real data collected from natural movements, while also enabling remote operators to perform real-time operations with the help of hand movements. As an important component of the robot system, the high degree of freedom and flexibility of dexterous hands play a vital role in fine operations, complex environment interactions, and multi-task scenarios, providing key support for practical applications in various fields.

[0003] However, there are obvious physical and kinematic differences between human hands and robotic hands. The human hand has 27 degrees of freedom, while different robotic hands have huge differences in their degrees of freedom, joint restrictions and physical dimensions when they are designed. This mismatch in structure and capabilities makes it difficult to directly map human hand movements, and may cause motion distortion or even fail to achieve the expected effect even in diverse task environments. Many traditional methods rely on manually set rules or fixed mapping schemes, which show significant limitations when dealing with heterogeneous robotic systems. They often lack the necessary adaptive capabilities and find it difficult to balance the requirements for fine and coarse-grained operations in different task scenarios. In order to make up for the above shortcomings, unsupervised neural networks are currently used to map the function vectors of the key points of the human hand to the joint angles of the robot hand through energy functions, thereby trying to capture the implicit nonlinear relationship in the human hand movements. On the other hand, through rule-based kinematic mapping methods, the intermediate joint rotations are used to reconstruct the posture of the human hand, thereby achieving the mapping of finger angles to a certain extent. In addition, there is a method that uses fingertip vector optimization technology to adjust the optimization strategy to balance functionality, constraints and motion smoothness. However, the above method cannot efficiently allocate the optimization weights between fingers when facing different complex scenarios, and cannot meet the requirements of real-time dynamic adaptability in complex task scenarios. Summary of the invention

[0004] In order to solve the technical problem that the existing redirection technology in the prior art cannot efficiently allocate the optimization weights between fingers in different complex scenarios, resulting in the inability to meet the requirements of real-time dynamic applicability in task scenarios, the embodiment of the present invention provides a method and device for redirecting human hand movements to heterogeneous robotic hands in multi-task scenarios. The technical solution is as follows:

[0005] On the one hand, a method for redirecting human hand movements to a heterogeneous robotic hand in a multi-task scenario is provided. This method is implemented by a device for redirecting human hand movements to a heterogeneous robotic hand in a multi-task scenario, and the method includes:

[0006] S1. Obtain the visual data and tactile data of the robotic hand; process the visual data to obtain a visual embedding vector; process the tactile data to obtain a tactile embedding vector; perform embedding splicing on the visual embedding vector and the tactile embedding vector and input them into a pre-trained Perceiver IO model for task type classification to obtain a task type encoding;

[0007] S2. Obtain the current robotic hand URDF / XML file, query the task-weight knowledge base according to the task type encoding to obtain a basic weight template; according to the template, adopt a physical constraint dynamic correction mechanism to correct the joint weights of the robotic hand fingers, and obtain the normalized finger weights through normalization processing;

[0008] S3. Use the hand detection module of the MediaPipe framework to detect the collected human hand video to obtain the human hand area; according to the human hand area, use the key point localization module of the MediaPipe framework to extract the key points of the human hand, and calculate the vectors between each joint of the human hand by the geometric vector method;

[0009] S4. According to the URDF / XML file and the pre-configured joint parameters, perform parsing and initialization, establish a joint mapping matrix, and calculate an adapted Jacobian matrix through driven joint constraint processing; perform forward kinematic mapping and inverse kinematic mapping according to the Jacobian matrix to obtain an adapted and standardized robotic hand kinematic model;

[0010] S5. According to the kinematic model, the normalized finger weights, and the vectors between each joint of the human hand, perform non-linear optimization on the vectors between human fingers and robotic hand fingers to obtain the joint angles of the robotic hand, so that the robotic hand completes movement redirection.

[0011] Optionally, the step of performing embedding splicing on the visual embedding vector and the tactile embedding vector and inputting them into a pre-trained Perceiver IO model for task type classification to obtain a task type encoding in S1 includes:

[0012] S11. Perform embedding splicing on the visual embedding vector and the tactile embedding vector and input them into a pre-trained Perceiver IO model, perform feature fusion through the self-attention mechanism and the cross-modal query mechanism to obtain a fused multi-modal feature;

[0013] S12. The fused multi-modal features are input into the Transformer encoder layer of the Perceiver IO model for deep semantic representation to obtain deep features;

[0014] S13. The deep features are input into the fully connected classification layer and mapped to the task categories, and the task type codes are obtained through calculation by the softmax activation function.

[0015] Optionally, the process of correcting the joint weights of the modified robotic finger is represented by the following formula (1):

[0016] (1)

[0017] where represents the dynamically corrected weight; represents the basic weight of the i-th finger; represents the maximum torque of the i-th finger joint; represents the system global safety torque threshold; represents the maximum limit of the i-th finger joint angle; represents the minimum limit of the i-th finger joint angle; represents the natural movement range of the i-th finger of the human hand; represents the torque constraint correction term; represents the motion space constraint correction term.

[0018] Optionally, S4 parses and initializes according to the URDF / XML file and the pre-configured joint parameters, establishes a joint mapping matrix, and performs driven joint constraint processing, including:

[0019] S41. Use the yourdfpy toolkit to parse the URDF / XML file and extract the robot structure and parameters;

[0020] S42. Correct and supplement information for the robot structure and parameters according to the pre-configured joint parameters to obtain the processed robot structure and parameters;

[0021] S43. Establish a joint mapping matrix according to the processed robot structure and parameters;

[0022] S44. Calculate the driven joint positions according to the joint mapping matrix to complete the driven joint constraint processing; among them, the calculation formula for the driven joint positions is represented by the following formula (2):

[0023] (2)

[0024] where represents the driven joint position vector; represents the active joint position vector; represents the transmission ratio vector; represents the position offset vector; represents the element-wise multiplication of vectors.

[0025] Optionally, the process of calculating the adapted Jacobian matrix is represented by the following formula (3):

[0026] (3)

[0027] where, represents the adapted Jacobian matrix; represents the target joint Jacobian matrix; represents the Jacobian matrix of the i-th driven joint; represents the transmission ratio of the i-th driven joint.

[0028] Optionally, the calculation formula of the forward kinematic mapping is represented by the following formula (4):

[0029] (4)

[0030] where x represents the finger coordinates in Cartesian coordinates; f represents the forward kinematic function;

[0031] where the calculation formula of the inverse kinematic mapping is represented by the following formula (5):

[0032] (5)

[0033] where, represents the inverse kinematic mapping function.

[0034] Optionally, the problem of non-linearly optimizing the vectors between human fingers and machine fingers of S5 is represented by the following formula (6):

[0035] (6)

[0036]

[0037] where, represents the base weight of the i-th finger; represents the regularization term coefficient; represents the finger joint angle; represents the forward kinematic function of the i-th finger of the robotic hand, outputting the end position coordinates; represents the 3D coordinates of the end of the i-th finger of the human hand; represents the safe default joint of the robot, in the semi-open state; represents the minimum joint of the finger joint; represents the maximum joint of the finger joint; Denote the joint torque vector, which is obtained by calculating through the Jacobian matrix and the contact force; Boolean function, which detects whether the pose of the robotic arm has no collision with the environment.

[0038] On the other hand, a device for redirecting human hand movements to a heterogeneous robotic arm in a multi-task scenario is provided. This device is applied to the method of redirecting human hand movements to a heterogeneous robotic arm in a multi-task scenario. The device includes:

[0039] A task type allocation unit, which is used to obtain the visual data and tactile data of the robotic arm; process the visual data to obtain a visual embedding vector; process the tactile data to obtain a tactile embedding vector; perform embedding splicing on the visual embedding vector and the tactile embedding vector and input them into a pre-trained Perceiver IO model for task type classification to obtain a task type encoding;

[0040] A dynamic weight correction unit, which is used to obtain the current URDF / XML file of the robotic arm, query the task-weight knowledge base according to the task type encoding to obtain a basic weight template; according to the template, adopt a physical constraint dynamic correction mechanism to correct the joint weights of the robotic arm's fingers, and through normalization processing, obtain the normalized finger weights;

[0041] A human hand key point recognition unit, which is used to detect the collected human hand video by using the hand detection module of the MediaPipe framework to obtain the human hand area, and according to the human hand area; use the key point positioning module of the MediaPipe framework to extract the key points of the human hand, and calculate the vectors between the joints of the human hand through the geometric vector method;

[0042] A calculation and acquisition unit, which is used to perform parsing and initialization according to the URDF / XML file and the pre-configured joint parameters, establish a joint mapping matrix, and calculate an adapted Jacobian matrix through driven joint constraint processing; perform forward kinematic mapping and inverse kinematic mapping according to the Jacobian matrix to obtain an adapted and standardized kinematic model of the robotic arm;

[0043] An optimization unit, which is used to perform non-linear optimization on the vectors between human fingers and robotic arm fingers according to the kinematic model, the normalized finger weights, and the vectors between the joints of the human hand, to obtain the joint angles of the robotic arm, so that the robotic arm completes the action redirection.

[0044] Optionally, the task type allocation unit is used for:

[0045] Perform embedding splicing on the visual embedding vector and the tactile embedding vector and input them into a pre-trained Perceiver IO model, and perform feature fusion through the self-attention mechanism and the cross-modal query mechanism to obtain the fused multi-modal features;

[0046] The fused multi-modal features are input into the Transformer encoder layer of the Perceiver IO model for deep semantic representation to obtain deep features;

[0047] The deep features are input into the fully connected classification layer and mapped to the task categories, and the task type encoding is obtained through calculation by the softmax activation function.

[0048] Optionally, the process of correcting the joint weights of the modified robotic finger is represented by the following formula (1):

[0049] (1)

[0050] where represents the dynamic correction weight; represents the base weight of the i-th finger; represents the maximum torque of the i-th finger joint; represents the system global safety torque threshold; represents the maximum limit of the i-th finger joint angle; represents the minimum limit of the i-th finger joint angle; represents the natural movement range of the i-th finger of the human hand; represents the torque constraint correction term; represents the motion space constraint correction term.

[0051] Optionally, the calculation and acquisition unit is used for:

[0052] Using the yourdfpy toolkit to parse URDF / XML files to extract the robot structure and parameters;

[0053] According to the pre-configured joint parameters, correcting and supplementing information for the robot structure and parameters to obtain the processed robot structure and parameters;

[0054] Based on the processed robot structure and parameters, establish a joint mapping matrix;

[0055] According to the joint mapping matrix, calculate the driven joint positions to complete the driven joint constraint processing; among them, the calculation formula of the driven joint positions is represented by the following formula (2):

[0056] (2)

[0057] where represents the driven joint position vector; represents the active joint position vector; represents the transmission ratio vector; represents the position offset vector; represents the element-wise multiplication of vectors.

[0058] Optionally, the process of calculating the adapted Jacobian matrix is represented by the following formula (3):

[0059] (3)

[0060] where represents the adapted Jacobian matrix; represents the target joint Jacobian matrix; represents the Jacobian matrix of the i-th driven joint; represents the transmission ratio of the i-th driven joint.

[0061] Optionally, the calculation formula of the forward kinematic mapping is represented by the following formula (4):

[0062] (4)

[0063] where x represents the finger coordinates in Cartesian coordinates; f represents the forward kinematic function;

[0064] where the calculation formula of the inverse kinematic mapping is represented by the following formula (5):

[0065] (5)

[0066] where represents the inverse kinematic mapping function.

[0067] Optionally, the problem of nonlinearly optimizing the vectors between human fingers and machine fingers is represented by the following formula (6):

[0068] (6)

[0069]

[0070] where represents the base weight of the i-th finger; represents the regularization term coefficient; represents the finger joint angle; represents the forward kinematic function of the i-th finger of the robotic hand, outputting the end position coordinates; represents the 3D coordinates of the end of the i-th finger of the human hand; represents the safe default joint of the robot, in the semi-open state; represents the minimum joint of the finger joint; represents the maximum joint of the finger joint; represents the joint torque vector, obtained by calculating through the Jacobian matrix and the contact force; A Boolean function for detecting whether the pose of the robotic arm is collision-free with the environment.

[0071] On the other hand, a device for redirecting human hand movements to a heterogeneous robotic arm in a multi-task scenario is provided. The device for redirecting human hand movements to a heterogeneous robotic arm in the multi-task scenario includes: a processor; a memory storing computer-readable instructions, which when executed by the processor, implement any one of the methods in the above-mentioned method for redirecting human hand movements to a heterogeneous robotic arm in a multi-task scenario.

[0072] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement any one of the methods in the above-mentioned method for redirecting human hand movements to a heterogeneous robotic arm in a multi-task scenario.

[0073] The beneficial effects brought by the technical solutions provided in the embodiments of the present invention at least include:

[0074] In the embodiments of the present invention, first, the visual data and tactile data of the robotic arm are acquired; the visual data is processed to obtain a visual embedding vector; the tactile data is processed to obtain a tactile embedding vector; the visual embedding vector and the tactile embedding vector are embedded and spliced and input into a pre-trained Perceiver IO model for task type classification to obtain a task type code; the current URDF / XML file of the robotic arm is acquired, and according to the task type code, the task-weight knowledge base is queried to obtain a basic weight template; secondly, according to the template, a physical constraint dynamic correction mechanism is used to correct the joint weights of the robotic arm fingers, and through normalization processing, the normalized finger weights are obtained; the hand detection module of the MediaPipe framework is used to detect the collected human hand video to obtain the human hand area, and according to the human hand area; the key point positioning module of the MediaPipe framework is used to extract the key points of the human hand, and the vectors between the joints of the human hand are calculated by the geometric vector method;

[0075] According to the URDF / XML file and the pre-configured joint parameters, parsing and initialization are performed to establish a joint mapping matrix, and through driven joint constraint processing, an adapted Jacobian matrix is calculated; forward kinematic mapping and inverse kinematic mapping are performed according to the Jacobian matrix to obtain an adapted and standardized kinematic model of the robotic arm; finally, according to the kinematic model, the normalized finger weights, and the vectors between the joints of the human hand, non-linear optimization is performed on the vectors between the human fingers and the robotic arm fingers to obtain the joint angles of the robotic arm, so that the robotic arm completes motion redirection.

[0076] The embodiment of the present invention solves the problem of over-reliance on specific tasks and models in traditional solutions by constructing a task-weight dynamic coupling mechanism and an incremental semantic knowledge base; the present invention does not require manual adjustment of weight parameters in mixed task scenarios such as grasping, lifting, and precision assembly. The use of the present invention can improve the generalization ability across tasks and heterogeneous models, and can support zero-sample adaptation from a 2-DOF gripper to a 21-DOF bionic hand. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0078] Figure 1 It is a flow chart of a method for redirecting human hand movements to heterogeneous robotic hands in a multi-task scenario provided by an embodiment of the present invention;

[0079] Figure 2 It is a structural schematic diagram of redirecting human hand movements to heterogeneous robotic arms in a multi-task scenario provided by an embodiment of the present invention;

[0080] Figure 3 It is a block diagram of a device for redirecting human hand movements to heterogeneous robotic arms in a multi-task scenario provided by an embodiment of the present invention;

[0081] Figure 4 It is a structural schematic diagram of a device for redirecting human hand movements to heterogeneous robotic arms in a multi-tasking scenario provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0082] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0083] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0084] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when not emphasizing the difference, their intended meanings are the same. "of", "corresponding", and "corresponding" can sometimes be used interchangeably. It should be noted that when not emphasizing the difference, their intended meanings are the same.

[0085] In the embodiments of the present invention, sometimes subscripts such as W 1 may be written in a non-subscript form such as W1. When not emphasizing the difference, their intended meanings are the same.

[0086] To make the technical problems, technical solutions, and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments.

[0087] The embodiments of the present invention provide a method for redirecting human hand movements to a heterogeneous robotic arm in a multi-task scenario. This method can be implemented by a device for redirecting human hand movements to a heterogeneous robotic arm in a multi-task scenario. The device for redirecting human hand movements to a heterogeneous robotic arm in a multi-task scenario can be a terminal or a server. As Figure 1 shown in the flowchart of the method for redirecting human hand movements to a heterogeneous robotic arm in a multi-task scenario, the processing flow of this method can include the following steps:

[0088] S1. Obtain the visual data and tactile data of the robotic arm; process the visual data to obtain a visual embedding vector; process the tactile data to obtain a tactile embedding vector; perform embedding splicing on the visual embedding vector and the tactile embedding vector and input them into a pre-trained Perceiver IO model for task type classification to obtain a task type code.

[0089] Among them, an RGB-D camera is used to collect the visual point cloud data of the object and the robotic arm; an array tactile sensor is used to collect tactile time series signal data.

[0090] In a feasible implementation, downsample the visual point cloud data and perform local patch division, and obtain the processed data by adding position encoding; convert the processed data into a fixed-dimensional visual embedding vector through point cloud embedding technology.

[0091] In a feasible implementation, perform normalization processing on the tactile time series signal data, embed each signal channel using one-dimensional convolution, and add sequence position information at the same time to obtain a tactile embedding vector.

[0092] Among them, the Perceiver IO model architecture is a machine learning architecture that can process input and output data of any structure.

[0093] Optionally, the specific implementation process of S1 includes S11 - S13:

[0094] S11. Embed and splice the visual embedding vector and the tactile embedding vector, and input them into the pre - trained PerceiverIO model. Through the self - attention mechanism and the cross - modal query mechanism, feature fusion is performed to obtain the fused multi - modal features;

[0095] In a feasible implementation, the visual embedding vector and the tactile embedding vector are embedded and spliced to obtain the spliced input vector; after linear mapping and position encoding of the spliced input vector, the input data of the Perceiver IO model is obtained.

[0096] Among them, the self - attention mechanism and the cross - modal query mechanism effectively capture the associations between different modalities and obtain the fused feature fusion.

[0097] S12. The fused multi - modal features are input into the Transformer encoder layer of the Perceiver IO model for deep semantic representation to obtain deep features;

[0098] S13. The deep features are input into the fully - connected classification layer and mapped to the task category, and the task type encoding is obtained through calculation using the softmax activation function.

[0099] Among them, the task categories include grasping tasks, pinching tasks, and lifting tasks; calculation is performed using the softmax activation function to obtain the task type encoding and the corresponding semantic labels, including the determination results and confidence levels of grasping, pinching, and lifting tasks.

[0100] S2. Obtain the current robotic arm URDF / XML file. According to the task type encoding, query the task - weight knowledge base to obtain the basic weight template; according to the template, use the physical constraint dynamic correction mechanism to correct the joint weights of the robotic fingers, and through normalization, obtain the normalized finger weights.

[0101] In a feasible implementation, the process of obtaining the basic weight template, for example, when using inspire_hand to grasp a cup with a handle, the weight vector corresponding to the five fingers for the pinching task (corresponding task number) is [0.8, 0.8, 0.1, 0.1, 0.1]. This weight indicates that the thumb and index finger have larger weights in the pinching task.

[0102] Among them, the physical constraint dynamic correction mechanism can adaptively adjust the joint weights to ensure the safe operation of the mechanism; while maintaining the task - completion ability, it avoids reaching the physical limits; and realizes the dynamic balance between task requirements and hardware constraints. The physical constraints of the robotic arm include: joint limits and maximum clamping force, etc., to dynamically correct the weights.

[0103] Optionally, the process of correcting the joint weights of the robotic fingers is represented by the following formula (1):

[0104] (1)

[0105] where represents the dynamically corrected weight of the i-th finger joint; represents the base weight of the i-th finger; represents the maximum torque of the i-th finger joint; represents the system global safety torque threshold; represents the maximum limit of the angle of the i-th finger joint; represents the minimum limit of the angle of the i-th finger joint; represents the natural movement range of the i-th finger of the human hand; represents the torque constraint correction term; represents the motion space constraint correction term.

[0106] Among them, the torque constraint correction term reflects the ratio of the actual available torque of each joint to the system safety threshold; when the joint torque approaches the limit, the weight of the joint is reduced to ensure safe operation; the torque constraint correction term is used to prevent joint overload and avoid mechanical damage.

[0107] Among them, the motion space constraint correction term represents the ratio of the actual available motion range of the joint to the standard motion range of the human hand; when the motion space of the joint is limited, the weight of the joint is correspondingly reduced; the motion space constraint correction term is used to ensure the feasibility of motion planning.

[0108] In a feasible implementation, the process of obtaining the normalized finger weights is represented by the following formula (2):

[0109] (2)

[0110] where represents the normalized finger weight.

[0111] S3. Use the hand detection module of the MediaPipe framework to detect the collected human hand video to obtain the human hand area, and according to the human hand area; use the key point localization module of the MediaPipe framework to extract the key points of the human hand, and calculate the vectors between each joint of the human hand by the geometric vector method.

[0112] In a feasible implementation, use the hand detection module of the MediaPipe framework to locate the human hand area from the human hand video, and extract 21 key points of the hand by the key point localization module of the MediaPipe framework.

[0113] S4. Parse and initialize according to the URDF / XML file and pre-configured joint parameters, establish a joint mapping matrix, and calculate the adapted Jacobian matrix through driven joint constraint processing; perform forward kinematic mapping and inverse kinematic mapping based on the Jacobian matrix to obtain the adapted and standardized robotic arm kinematic model.

[0114] Among them, the pre-configured joint parameters include: active joint parameters and passive joint parameters.

[0115] Among them, the adapted and standardized robotic arm kinematic model includes: a unified forward kinematic interface, a unified inverse kinematic interface, a driven joint constraint processing mechanism, and a Jacobian matrix adaptation mechanism.

[0116] Optionally, the specific implementation process of S4 includes S41 - S44:

[0117] S41. Use the yourdfpy toolkit to parse the URDF / XML file and extract the robot structure and parameters;

[0118] S42. Correct and supplement information for the robot structure and parameters according to the pre-configured joint parameters to obtain the processed robot structure and parameters;

[0119] S43. Establish a joint mapping matrix according to the processed robot structure and parameters;

[0120] Among them, by establishing the joint mapping matrix, the legality of the joint configuration can be verified to ensure that the mechanism constraints meet the requirements.

[0121] S44. Calculate the driven joint positions based on the joint mapping matrix to complete the driven joint constraint processing; among them, the calculation formula for the driven joint positions is represented by the following formula (3):

[0122] (3)

[0123] Among them, represents the driven joint position vector; represents the active joint position vector; represents the transmission ratio vector; represents the position offset vector; represents element-wise multiplication of vectors.

[0124] Among them, a driven joint refers to a joint whose motion is determined by other joints, and its kinematic characteristics need to be specially processed.

[0125] Optionally, the process of calculating the adapted Jacobian matrix is represented by the following formula (4):

[0126] (4)

[0127] Among them, represents the adapted Jacobian matrix; represents the target joint Jacobian matrix; represents the Jacobian matrix of the i-th driven joint; represents the transmission ratio of the i-th driven joint.

[0128] Optionally, the calculation formula of the forward kinematic mapping is represented by the following formula (5):

[0129] (5)

[0130] Among them, x represents the finger coordinates in Cartesian coordinates; f represents the forward kinematic function;

[0131] In a feasible implementation manner, the specific implementation process of the forward kinematic mapping includes:

[0132] (1) Calculate the positions of the driven joints based on the positions of the active joints and the transmission ratios;

[0133] (2) Combine the positions of the active joints and the driven joints to construct a complete joint model;

[0134] (3) Based on the complete joint model, use the forward kinematic algorithm to calculate the poses of each end fingertip.

[0135] Among them, the forward kinematic algorithm is a method mastered by those skilled in the art, and the position and pose of the robot end execution are calculated through the given robot joint variables.

[0136] Among them, the calculation formula of the inverse kinematic mapping is represented by the following formula (6):

[0137] (6)

[0138] Among them, represents the inverse kinematic mapping function.

[0139] In a feasible implementation manner, the specific implementation process of the inverse kinematic mapping includes:

[0140] (1) Calculate the adapted Jacobian matrix;

[0141] (2) Solve the minimized position error according to the adapted Jacobian matrix;

[0142] (3) Based on the minimized position error, perform joint limit processing to keep the joints within the physical limit range.

[0143] S5. According to the kinematic model, the normalized finger weights, and the vectors between the joints of the human hand, nonlinearly optimize the vectors between the human fingers and the robotic fingers to obtain the joint angles of the robotic hand, enabling the robotic hand to complete motion redirection.

[0144] Optionally, the problem of nonlinearly optimizing the vectors between the human fingers and the robotic fingers in S5 is represented by the following formula (7):

[0145] (7)

[0146]

[0147] where represents the base weight of the i-th finger; represents the regularization term coefficient; represents the finger joint angle; represents the forward kinematic function of the i-th finger of the robotic hand, which outputs the end position coordinates; represents the 3D coordinates of the end of the i-th finger of the human hand; represents the safe default joint of the robot, in the semi-open state; represents the minimum joint of the finger joint; represents the maximum joint of the finger joint; represents the joint torque vector, obtained by calculating through the Jacobian matrix and the contact force; is a boolean function that detects whether the pose of the robotic hand has no collision with the environment.

[0148] where, nonlinearly optimize the vectors between the human fingers and the robotic fingers to obtain the joint angles of the robotic hand, so that the robot realizes the functions of the human hand and the robotic hand completes motion redirection.

[0149] Among them, the present invention adopts a task-semantic-driven dynamic weight allocation mechanism to identify task types such as pinching, grasping, and lifting within 0.1 second; constructs a task-weight knowledge base to dynamically set the optimization weights of each finger, solves the problem that the last three fingers are overly squeezed and deformed when grasping thin-walled objects in traditional methods, and improves the success rate of operations. After identifying the task type and determining the importance weights of the fingers, it can perform real-time redirection according to the URDF / XML file of the robotic hand of the heterogeneous robotic hand, so that different settings for different robotic hands are not required, improving the scene generalization and hardware adaptability.

[0150] Among them, the Realsense D435i RGBD camera adopted in the present invention can collect point clouds with a resolution of 1280×720, with a depth accuracy of ±1mm, and transmit them to the graphics workstation through USB3.0. An array tactile sensor is used to collect tactile data. The array tactile sensor is integrated into the fingertips of the Yinshi dexterous hand, with 5-channel pressure detection (0-10N, ±0.1N), and the sampling rate of 100Hz is output through the SPI interface; among them, the graphics workstation is an NVIDIA RTX 4090 GPU; among them, the real-time controller is an intel i9 processor for kinematic adaptation and optimization solution; the hardware adopted is the Yinshi dexterous hand, a 6-degree-of-freedom bionic hand, and the joints adopt Maxon EC-i40 motors with a control accuracy of 0.1°. The 7-degree-of-freedom robotic arm adopted has a load of 5kg, is synchronously controlled by the EtherCAT bus, and the repeat positioning accuracy of the end is ±0.05mm. The methods proposed in the present invention are used to simulate the shadow hand, allogro, and ability robotic hands.

[0151] Among them, as Figure 2 is a schematic structural diagram of redirecting human hand movements to heterogeneous robotic hands in a multi-task scenario provided by an embodiment of the present invention; obtaining the human hand video and the URDF / XML file of the robotic hand; the human hand video obtains the key points and joint angles of the human hand through the gesture recognition detection module; the URDF / XML file of the robotic hand is unified into a redirection class through the motion adapter according to the configuration file customized for different hands and redirection types; obtaining the specific task scenario, and obtaining the normalized finger weights through the multi-modal classifier and task weight dynamic allocation; according to the normalized finger weights, as well as the key points and joint angles of the human hand, non-linearly optimize the vectors between human fingers and robotic fingers to obtain the joint angles of the robotic hand, so that the robotic hand completes the motion redirection.

[0152] The embodiment of the present invention first obtains the visual data and tactile data of the robotic hand; processes the visual data to obtain the visual embedding vector; processes the tactile data to obtain the tactile embedding vector; splices the visual embedding vector and the tactile embedding vector and inputs them into the pre-trained Perceiver IO model for task type classification to obtain the task type encoding; obtains the current URDF / XML file of the robotic hand, and queries the task-weight knowledge base according to the task type encoding to obtain the basic weight template; secondly, according to the template, adopts the physical constraint dynamic correction mechanism to correct the joint weights of the robotic fingers, and obtains the normalized finger weights through normalization; uses the hand detection module of the MediaPipe framework to detect the collected human hand video to obtain the human hand area, according to the human hand area; uses the key point positioning module of the MediaPipe framework to extract the key points of the human hand, and calculates the vectors between each joint of the human hand through the geometric vector method;

[0153] According to the URDF / XML file and pre-configured joint parameters, parsing and initialization are performed to establish a joint mapping matrix. Through the processing of slave joint constraints, the adapted Jacobian matrix is calculated. According to the Jacobian matrix, forward kinematic mapping and inverse kinematic mapping are performed to obtain the adapted and standardized robotic hand kinematic model. Finally, according to the kinematic model, the normalized finger weights, and the vectors between each joint of the human hand, non-linear optimization is performed on the vectors between human fingers and robotic fingers to obtain the joint angles of the robotic hand, enabling the robotic hand to complete action redirection.

[0154] In the embodiment of the present invention, by constructing a task-weight dynamic coupling mechanism and an incremental semantic knowledge base, the problem of over-reliance on specific tasks and models in traditional solutions is solved; in the mixed task scenarios of grasping, lifting, and precision assembly in the present invention, there is no need for manual adjustment of weight parameters. Using the present invention can improve the generalization ability across tasks and heterogeneous models, and support zero-shot adaptation from 2-degree-of-freedom grippers to 21-degree-of-freedom bionic hands.

[0155] Figure 3 It is a block diagram of a device for redirecting human hand movements to a heterogeneous robotic hand in a multi-task scenario shown according to an exemplary embodiment. This device is used for the method of redirecting human hand movements to a heterogeneous robotic hand in a multi-task scenario. Refer to Figure 3 This device includes a task type allocation unit 310, a dynamic weight correction unit 320, a human hand key point recognition unit 330, a calculation and acquisition unit 340, and an optimization unit 350. Among them:

[0156] The task type allocation unit 310 is used to obtain the visual data and tactile data of the robotic hand; process the visual data to obtain a visual embedding vector; process the tactile data to obtain a tactile embedding vector; perform embedding splicing on the visual embedding vector and the tactile embedding vector and input them into a pre-trained Perceiver IO model for task type classification to obtain a task type code;

[0157] The dynamic weight correction unit 320 is used to obtain the current URDF / XML file of the robotic hand, query the task-weight knowledge base according to the task type code to obtain a basic weight template; according to the template, use a physical constraint dynamic correction mechanism to correct the joint weights of the robotic fingers, and through normalization processing, obtain the normalized finger weights;

[0158] The human hand key point recognition unit 330 is used to detect the collected human hand video using the hand detection module of the MediaPipe framework to obtain the human hand area, and according to the human hand area; use the key point positioning module of the MediaPipe framework to extract the key points of the human hand, and calculate the vectors between each joint of the human hand by the geometric vector method;

[0159] A calculation and acquisition unit 340, configured to perform parsing and initialization according to the URDF / XML file and pre-configured joint parameters, establish a joint mapping matrix, and calculate an adapted Jacobian matrix through driven joint constraint processing; perform forward kinematic mapping and inverse kinematic mapping according to the Jacobian matrix to obtain an adapted and standardized robotic arm kinematic model;

[0160] An optimization unit 350, configured to perform non-linear optimization on the vectors between human fingers and robotic fingers according to the kinematic model, normalized finger weights, and vectors between each joint of the human hand, to obtain the joint angles of the robotic arm, so that the robotic arm completes action redirection.

[0161] Optionally, the task type allocation unit 310 is configured to:

[0162] Perform embedding splicing on the visual embedding vector and the tactile embedding vector and input them into a pre-trained Perceiver IO model, perform feature fusion through a self-attention mechanism and a cross-modal query mechanism to obtain a fused multi-modal feature;

[0163] The fused multi-modal feature is input into the Transformer encoder layer of the Perceiver IO model for deep semantic representation to obtain a deep feature;

[0164] The deep feature is input into a fully connected classification layer to be mapped to a task category, and a task type encoding is obtained through calculation using a softmax activation function.

[0165] Optionally, the process of correcting the joint weights of the robotic fingers is represented by the following formula (1):

[0166] (1)

[0167] Where, represents dynamic weight correction; represents the basic weight of the i-th finger; represents the maximum torque of the i-th finger joint; represents the system global safety torque threshold; represents the maximum limit of the i-th finger joint angle; represents the minimum limit of the i-th finger joint angle; represents the natural movement range of the i-th finger of the human hand; represents the torque constraint correction term; represents the motion space constraint correction term.

[0168] Optionally, the calculation and acquisition unit 340 is configured to:

[0169] Use the yourdfpy toolkit to parse URDF / XML files and extract the robot structure and parameters;

[0170] Calibrate and supplement information for the robot structure and parameters according to the pre-configured joint parameters to obtain the processed robot structure and parameters;

[0171] Establish a joint mapping matrix based on the processed robot structure and parameters;

[0172] Calculate the driven joint position based on the joint mapping matrix to complete the driven joint constraint processing; where the calculation formula for the driven joint position is represented by the following formula (2):

[0173] (2)

[0174] where, represents the driven joint position vector; represents the active joint position vector; represents the transmission ratio vector; represents the position offset vector; represents element-wise multiplication of vectors.

[0175] Optionally, the process of calculating the adapted Jacobian matrix is represented by the following formula (3):

[0176] (3)

[0177] where, represents the adapted Jacobian matrix; represents the target joint Jacobian matrix; represents the Jacobian matrix of the i-th driven joint; represents the transmission ratio of the i-th driven joint.

[0178] Optionally, the calculation formula for the forward kinematic mapping is represented by the following formula (4):

[0179] (4)

[0180] where x represents the finger coordinates in Cartesian coordinates; f represents the forward kinematic function;

[0181] where, the calculation formula for the inverse kinematic mapping is represented by the following formula (5):

[0182] (5)

[0183] where, represents the inverse kinematic mapping function.

[0184] Optionally, the problem of non-linearly optimizing the vector between the human fingers and the robotic fingers is represented by the following formula (6):

[0185] (6)

[0186]

[0187] Wherein, represents the base weight of the i-th finger; represents the regularization term coefficient; represents the finger joint angle; represents the forward kinematic function of the i-th finger of the robotic hand, outputting the end position coordinates; represents the 3D coordinates of the end of the i-th finger of the human hand; represents the safe default joint of the robot, in the semi-open state; represents the minimum joint of the finger joint; represents the maximum joint of the finger joint; represents the joint torque vector, obtained by calculating through the Jacobian matrix and the contact force; Boolean function, detecting whether the pose of the robotic hand collides with the environment.

[0188] In the embodiment of the present invention, first, the visual data and tactile data of the robotic hand are obtained; the visual data is processed to obtain a visual embedding vector; the tactile data is processed to obtain a tactile embedding vector; the visual embedding vector and the tactile embedding vector are embedded and spliced and input into a pre-trained Perceiver IO model for task type classification to obtain a task type encoding; the current URDF / XML file of the robotic hand is obtained, and according to the task type encoding, the task-weight knowledge base is queried to obtain a base weight template; secondly, according to the template, a physical constraint dynamic correction mechanism is adopted to correct the joint weights of the robotic fingers, and through normalization processing, the normalized finger weights are obtained; the hand detection module of the MediaPipe framework is used to detect the collected human hand video to obtain the human hand area, according to the human hand area; the key point positioning module of the MediaPipe framework is used to extract the key points of the human hand, and the vector between each joint of the human hand is calculated by the geometric vector method; according to the URDF / XML file and the pre-configured joint parameters, parsing and initialization are performed to establish a joint mapping matrix, and through the processing of the driven joint constraint, the adapted Jacobian matrix is calculated; according to the Jacobian matrix, forward kinematic mapping and inverse kinematic mapping are performed to obtain the adapted and standardized kinematic model of the robotic hand; finally, according to the kinematic model, the normalized finger weights, and the vector between each joint of the human hand, the vector between the human fingers and the robotic fingers is non-linearly optimized to obtain the joint angles of the robotic hand, so that the robotic hand completes action redirection.

[0189] In the embodiments of the present invention, by constructing a task-weight dynamic coupling mechanism and an incremental semantic knowledge base, the problem of over-reliance on specific tasks and models in traditional solutions is solved; in the hybrid task scenario of grasping, lifting, and precision assembly, the present invention does not require manual adjustment of weight parameters. The present invention can improve the generalization ability across tasks and heterogeneous models, and support zero-shot adaptation from 2-degree-of-freedom grippers to 21-degree-of-freedom bionic hands.

[0190] Figure 4 FIG. 4 is a schematic structural diagram of a device for redirecting human hand movements to a heterogeneous robotic hand in a multi-task scenario provided by an embodiment of the present invention, as Figure 4 shown. The device for redirecting human hand movements to a heterogeneous robotic hand in a multi-task scenario may include the above Figure 3 shown device for redirecting human hand movements to a heterogeneous robotic hand in a multi-task scenario. Optionally, the device 410 for redirecting human hand movements to a heterogeneous robotic hand in a multi-task scenario may include a first processor 2001.

[0191] Optionally, the device 410 for redirecting human hand movements to a heterogeneous robotic hand in a multi-task scenario may further include a memory 2002 and a transceiver 2003.

[0192] Wherein, the first processor 2001, the memory 2002, and the transceiver 2003 may be connected through a communication bus, for example.

[0193] Next, in conjunction with Figure 4 each component of the device 410 for redirecting human hand movements to a heterogeneous robotic hand in a multi-task scenario will be specifically introduced:

[0194] Among them, the first processor 2001 is the control center of the device 410 for redirecting human hand movements to a heterogeneous robotic hand in a multi-task scenario, and may be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as: one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0195] Optionally, the first processor 2001 can execute various functions of the device 410 that redirects the human hand movement to the heterogeneous robotic hand in a multi-task scenario by running or executing software programs stored in the memory 2002 and invoking data stored in the memory 2002.

[0196] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 shown in

[0197] In a specific implementation, as an embodiment, the device 410 that redirects the human hand movement to the heterogeneous robotic hand in a multi-task scenario may also include multiple processors, such as Figure 4 the first processor 2001 and the second processor 2004 shown in

[0198] Among them, the memory 2002 is used to store the software program for implementing the solution of the present invention and is controlled by the first processor 2001 for execution. The specific implementation manner can refer to the above method embodiments and will not be elaborated here.

[0199] Optionally, the memory 2002 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but not limited thereto. The memory 2002 can be integrated with the first processor 2001 or exist independently and is coupled to the first processor 2001 through an interface circuit ( Figure 4 not shown in

[0200] A transceiver 2003 is used to communicate with a network device or with a terminal device.

[0201] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 not shown separately). Among them, the receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0202] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and is coupled to the first processor 2001 through an interface circuit ( Figure 4 not shown) of a device 410 that redirects a human hand movement to a heterogeneous robotic arm in a multi-task scenario. The embodiments of the present invention do not make specific limitations in this regard.

[0203] It should be noted that Figure 4 the structure of the device 410 that redirects a human hand movement to a heterogeneous robotic arm in the multi-task scenario shown does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0204] In addition, the technical effects of the device 410 that redirects a human hand movement to a heterogeneous robotic arm in the multi-task scenario may refer to the technical effects of the method of redirecting a human hand movement to a heterogeneous robotic arm in the multi-task scenario described in the above method embodiments, and will not be elaborated here.

[0205] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), and this processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0206] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0207] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more collections of available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0208] It should be understood that the term "and / or" in this document is merely a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B can be singular or plural. Additionally, the character " / " in this document generally represents an "or" relationship between the associated objects before and after, but it may also represent an "and / or" relationship, which can be understood specifically by referring to the context.

[0209] In the present invention, "at least one" means one or more, and "a plurality" means two or more. "At least one of the following" or its similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0210] It should be understood that in various embodiments of the present invention, the magnitudes of the sequence numbers of the above processes do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0211] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0212] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the devices, apparatuses, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0213] In several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be electrical, mechanical, or other forms.

[0214] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0215] In addition, the functional units in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0216] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0217] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for redirecting human hand movements to heterogeneous robotic hands in a multi-task scenario, characterized in that: The method comprises: S1. Obtain visual data and tactile data of the robot; process the visual data to obtain a visual embedding vector; process the tactile data to obtain a tactile embedding vector; embed and concatenate the visual embedding vector and the tactile embedding vector and input them into a pre-trained Perceiver IO model to classify the task type and obtain the task type code; S2. Obtain the current robot hand URDF / XML file, query the task-weight knowledge base according to the task type code, and obtain the basic weight template; according to the template, use the physical constraint dynamic correction mechanism to correct the joint weights of the robot fingers, and obtain the normalized finger weights through normalization processing; S3, using the hand detection module of the MediaPipe framework to detect the collected human hand video, obtain the human hand area, and based on the human hand area; using the key point positioning module of the MediaPipe framework to extract the key points of the human hand, and calculate the vectors between the joints of the human hand through the geometric vector method; S4, parsing and initializing according to the URDF / XML file and pre-configured joint parameters, establishing a joint mapping matrix, calculating the adapted Jacobian matrix through slave joint constraint processing; performing forward kinematics mapping and inverse kinematics mapping according to the Jacobian matrix to obtain an adapted standardized robot hand kinematics model; S5. According to the kinematic model, the normalized finger weights and the vectors between the joints of the human hand, nonlinear optimization is performed on the vectors between the human fingers and the robot fingers to obtain the joint angles of the robot hand so that the robot hand can complete the action redirection.

2. The method for redirecting human hand movements to heterogeneous robotic hands in a multi-tasking scenario according to claim 1, characterized in that: The S1 embeds and concatenates the visual embedding vector and the tactile embedding vector and inputs them into the pre-trained PerceiverIO model to classify the task type and obtain the task type code, including: S11, embed the visual embedding vector and the tactile embedding vector and concatenate them into the pre-trained Perceiver IO model, perform feature fusion through the self-attention mechanism and cross-modal query mechanism, and obtain the fused multi-modal features; S12, the fused multimodal features are input into the Transformer encoder layer of the Perceiver IO model for deep semantic representation to obtain deep features; S13. The deep features are input into the fully connected classification layer and mapped to the task category. The task type encoding is obtained by calculation through the softmax activation function.

3. The method for redirecting human hand movements to heterogeneous robotic hands in a multi-tasking scenario according to claim 1, characterized in that: The process of correcting the joint weights of the robot finger is expressed by the following formula (1): （1） in, Indicates the dynamic correction weight; represents the basic weight of the i-th finger; represents the maximum torque of the i-th finger joint; Indicates the system global safety torque threshold; Indicates the maximum limit of the angle of the i-th finger joint; Indicates the minimum limit of the angle of the i-th finger joint; Indicates the natural range of motion of the i-th finger of a human hand; represents the torque constraint correction term; Represents the motion space constraint correction term.

4. The method for redirecting human hand movements to heterogeneous robotic hands in a multi-tasking scenario according to claim 1, characterized in that: The S4 parses and initializes the URDF / XML file and pre-configured joint parameters, establishes a joint mapping matrix, and processes the driven joint constraints, including: S41. Use yourdfpy toolkit to parse URDF / XML files and extract robot structure and parameters; S42, correcting and supplementing information of the robot structure and parameters according to the pre-configured joint parameters to obtain processed robot structure and parameters; S43, establishing a joint mapping matrix according to the processed robot structure and parameters; S44, according to the joint mapping matrix, the driven joint position is calculated to complete the driven joint constraint processing; wherein the calculation formula of the driven joint position is expressed by the following formula (2): （2） in, represents the position vector of the slave joint; represents the active joint position vector; represents the transmission ratio vector; represents the position bias vector; Represents vector element-wise multiplication.

5. The method for redirecting human hand movements to heterogeneous robotic hands in a multi-tasking scenario according to claim 1, characterized in that: The process of calculating the adapted Jacobian matrix is ​​expressed by the following formula (3): （3） in, represents the adapted Jacobian matrix; represents the target joint Jacobian matrix; represents the Jacobian matrix of the i-th driven joint; represents the transmission ratio of the i-th driven joint.

6. The method for redirecting human hand movements to heterogeneous robotic hands in a multi-tasking scenario according to claim 1, characterized in that: The calculation formula of the forward kinematics mapping is expressed by the following formula (4): （4） Where x represents the finger coordinates in Cartesian coordinates; f represents the positive kinematic function; The calculation formula of the inverse kinematics mapping is expressed by the following formula (5): （5） in, Represents the inverse kinematics mapping function.

7. The method for redirecting human hand movements to heterogeneous robotic hands in a multi-tasking scenario according to claim 1, characterized in that: The problem of nonlinear optimization of the vectors between human fingers and machine fingers in S5 is expressed by the following formula (6): （6） ； in, represents the basic weight of the i-th finger; represents the regularization term coefficient; Indicates the finger joint angle; Represents the positive kinematics function of the i-th finger of the robot hand, outputting the end position coordinates; represents the 3D coordinates of the end of the i-th finger of a human hand; Indicates the robot's safe default joint, half-open state; Indicates the smallest joint of the finger joint; Indicates the largest joint of the finger joint; represents the joint torque vector, which is obtained by calculating the Jacobian matrix and contact force; Boolean function that detects whether the robot's posture has no collision with the environment.

8. A device for redirecting human hand movements to heterogeneous robotic arms in a multi-task scenario, wherein the device for redirecting human hand movements to heterogeneous robotic arms in a multi-task scenario is used to implement the method for redirecting human hand movements to heterogeneous robotic arms in a multi-task scenario as claimed in any one of claims 1 to 7, characterized in that: The device comprises: The task type assignment unit is used to obtain the visual data and tactile data of the robot hand; process the visual data to obtain the visual embedding vector; process the tactile data to obtain the tactile embedding vector; embed the visual embedding vector and the tactile embedding vector and concatenate them into the pre-trained Perceiver IO model to classify the task type and obtain the task type code; The dynamic weight correction unit is used to obtain the current robot hand URDF / XML file, query the task-weight knowledge base according to the task type code, and obtain the basic weight template; according to the template, the physical constraint dynamic correction mechanism is adopted to correct the joint weights of the robot fingers, and the normalized finger weights are obtained through normalization processing; The hand key point recognition unit is used to detect the collected hand video using the hand detection module of the MediaPipe framework to obtain the hand area. Based on the hand area, the key point positioning module of the MediaPipe framework is used to extract the key points of the hand, and the vectors between the joints of the hand are calculated by the geometric vector method. A calculation and acquisition unit, used to parse and initialize according to the URDF / XML file and pre-configured joint parameters, establish a joint mapping matrix, calculate the adapted Jacobian matrix through slave joint constraint processing; perform forward kinematics mapping and inverse kinematics mapping according to the Jacobian matrix to obtain an adapted standardized robot hand kinematics model; The optimization unit is used to perform nonlinear optimization on the vectors between human fingers and robot fingers according to the kinematic model, the normalized finger weights and the vectors between the joints of the human hand, obtain the joint angles of the robot hand, and enable the robot hand to complete the action redirection.

9. A device for redirecting human hand movements to heterogeneous robotic arms in a multi-tasking scenario, characterized in that: The device for redirecting human hand movements to heterogeneous robotic arms in the multi-task scenario includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images