Method and device for redirecting human hand action to heterogeneous robot arm in multi-task scene

Abstract

The invention provides a method and a device for redirecting human hand actions to a heterogeneous manipulator in a multi-task scene, and relates to the technical field of robots and intelligent control. The method comprises the following steps: acquiring visual data and tactile data; performing task type classification by adopting a pre-trained Perciver IO model to obtain a task type code; according to a physical constraint dynamic correction mechanism, correcting the joint weight of the machine finger to obtain a normalized finger weight; key point extraction is conducted on the human hand, and vectors between all joints of the human hand are calculated through a geometric vector method; performing driven joint constraint processing according to the established joint mapping matrix, calculating an adapted Jacobian matrix, and obtaining an adapted standardized robot arm kinematics model; and carrying out nonlinear optimization on the vectors between the human fingers and between the robot fingers to obtain each joint angle of the robot arm, so that the robot arm completes action redirection. By adopting the method, the generalization ability of cross-task and heterogeneous models can be improved.

Description

Technical Field

[0001] The present invention relates to the field of robotics and intelligent control technology, and in particular to a method and device for redirecting human hand movements to heterogeneous robotic hands in a multi-task scenario. Background Art

[0002] Motion redirection technology can not only map the complex movements of human hands to robot hands to achieve efficient and accurate motion transfer, but also show unique advantages in data collection and remote control. Using motion redirection technology, large-scale training data sets can be built from real data collected from natural movements, while also enabling remote operators to perform real-time operations with the help of hand movements. As an important component of the robot system, the high degree of freedom and flexibility of dexterous hands play a vital role in fine operations, complex environment interactions, and multi-task scenarios, providing key support for practical applications in various fields.

[0003] However, there are obvious physical and kinematic differences between human hands and robotic hands. The human hand has 27 degrees of freedom, while different robotic hands have huge differences in their degrees of freedom, joint restrictions and physical dimensions when they are designed. This mismatch in structure and capabilities makes it difficult to directly map human hand movements, and may cause motion distortion or even fail to achieve the expected effect even in diverse task environments. Many traditional methods rely on manually set rules or fixed mapping schemes, which show significant limitations when dealing with heterogeneous robotic systems. They often lack the necessary adaptive capabilities and find it difficult to balance the requirements for fine and coarse-grained operations in different task scenarios. In order to make up for the above shortcomings, unsupervised neural networks are currently used to map the function vectors of the key points of the human hand to the joint angles of the robot hand through energy functions, thereby trying to capture the implicit nonlinear relationship in the human hand movements. On the other hand, through rule-based kinematic mapping methods, the intermediate joint rotations are used to reconstruct the posture of the human hand, thereby achieving the mapping of finger angles to a certain extent. In addition, there is a method that uses fingertip vector optimization technology to adjust the optimization strategy to balance functionality, constraints and motion smoothness. However, the above method cannot efficiently allocate the optimization weights between fingers when facing different complex scenarios, and cannot meet the requirements of real-time dynamic adaptability in complex task scenarios. Summary of the invention

[0004] In order to solve the technical problem that the existing redirection technology in the prior art cannot efficiently allocate the optimization weights between fingers in different complex scenarios, resulting in the inability to meet the requirements of real-time dynamic applicability in task scenarios, the embodiment of the present invention provides a method and device for redirecting human hand movements to heterogeneous robotic hands in multi-task scenarios. The technical solution is as follows:

[0005] On the one hand, a method for redirecting human hand movements to heterogeneous robotic arms in a multi-tasking scenario is provided, the method being implemented by a device for redirecting human hand movements to heterogeneous robotic arms in a multi-tasking scenario, the method comprising:

[0006] S1. Obtain visual data and tactile data of the robot; process the visual data to obtain a visual embedding vector; process the tactile data to obtain a tactile embedding vector; embed and concatenate the visual embedding vector and the tactile embedding vector and input them into a pre-trained Perceiver IO model to classify the task type and obtain the task type code;

[0007] S2. Obtain the current robot hand URDF / XML file, query the task-weight knowledge base according to the task type code, and obtain the basic weight template; according to the template, use the physical constraint dynamic correction mechanism to correct the joint weights of the robot fingers, and obtain the normalized finger weights through normalization processing;

[0008] S3, using the hand detection module of the MediaPipe framework to detect the collected human hand video, obtain the human hand area, and based on the human hand area; using the key point positioning module of the MediaPipe framework to extract the key points of the human hand, and calculate the vectors between the joints of the human hand through the geometric vector method;

[0009] S4, parsing and initializing according to the URDF / XML file and pre-configured joint parameters, establishing a joint mapping matrix, calculating the adapted Jacobian matrix through slave joint constraint processing; performing forward kinematics mapping and inverse kinematics mapping according to the Jacobian matrix to obtain an adapted standardized robot hand kinematics model;

[0010] S5. According to the kinematic model, the normalized finger weights and the vectors between the joints of the human hand, nonlinear optimization is performed on the vectors between the human fingers and the robot fingers to obtain the joint angles of the robot hand so that the robot hand can complete the action redirection.

[0011] Optionally, the step of embedding and concatenating the visual embedding vector and the tactile embedding vector in S1 and inputting the embedded vector into a pre-trained Perceiver IO model to classify the task type and obtain the task type code includes:

[0012] S11. Embed the visual embedding vector and the tactile embedding vector and concatenate them into the pre-trained PerceiverIO model, perform feature fusion through the self-attention mechanism and cross-modal query mechanism, and obtain the fused multi-modal features;

[0013] S12, the fused multimodal features are input into the Transformer encoder layer of the Perceiver IO model for deep semantic representation to obtain deep features;

[0014] S13. The deep features are input into the fully connected classification layer and mapped to the task category. The task type encoding is obtained by calculation through the softmax activation function.

[0015] Optionally, the process of correcting the joint weights of the robot finger is expressed by the following formula (1):

[0016] (1)

[0017] in, Indicates the dynamic correction weight; represents the basic weight of the i-th finger; represents the maximum torque of the i-th finger joint; Indicates the system global safety torque threshold; Indicates the maximum limit of the angle of the i-th finger joint; Indicates the minimum limit of the angle of the i-th finger joint; Indicates the natural range of motion of the i-th finger of a human hand; represents the torque constraint correction term; Represents the motion space constraint correction term.

[0018] Optionally, the S4 parses and initializes the URDF / XML file and pre-configured joint parameters, establishes a joint mapping matrix, and processes the driven joint constraints, including:

[0019] S41. Use yourdfpy toolkit to parse URDF / XML files and extract robot structure and parameters;

[0020] S42, correcting and supplementing information of the robot structure and parameters according to the pre-configured joint parameters to obtain processed robot structure and parameters;

[0021] S43, establishing a joint mapping matrix according to the processed robot structure and parameters;

[0022] S44, according to the joint mapping matrix, the driven joint position is calculated to complete the driven joint constraint processing; wherein the calculation formula of the driven joint position is expressed by the following formula (2):

[0023] (2)

[0024] in, represents the position vector of the slave joint; represents the active joint position vector; represents the transmission ratio vector; represents the position bias vector; Represents vector element-wise multiplication.

[0025] Optionally, the process of calculating the adapted Jacobian matrix is ​​expressed by the following formula (3):

[0026] (3)

[0027] in, represents the adapted Jacobian matrix; represents the target joint Jacobian matrix; represents the Jacobian matrix of the i-th driven joint; represents the transmission ratio of the i-th driven joint.

[0028] Optionally, the calculation formula of the forward kinematics mapping is expressed by the following formula (4):

[0029] (4)

[0030] Where x represents the finger coordinates in Cartesian coordinates; f represents the positive kinematic function;

[0031] The calculation formula of the inverse kinematics mapping is expressed by the following formula (5):

[0032] (5)

[0033] in, Represents the inverse kinematics mapping function.

[0034] Optionally, the problem of performing nonlinear optimization on the vectors between human fingers and machine fingers in S5 is expressed by the following formula (6):

[0035] (6)

[0036]

[0037] in, represents the basic weight of the i-th finger; represents the regularization term coefficient; Indicates the finger joint angle; Represents the positive kinematics function of the i-th finger of the robot hand, outputting the end position coordinates; represents the 3D coordinates of the end of the i-th finger of a human hand; Indicates the robot's safe default joint, half-open state; Indicates the smallest joint of the finger joint; Indicates the largest joint of the finger joint; represents the joint torque vector, which is obtained by calculating the Jacobian matrix and contact force; Boolean function that detects whether the robot's posture has no collision with the environment.

[0038] On the other hand, a device for redirecting human hand movements to a heterogeneous robotic arm in a multi-task scenario is provided. The device is applied to a method for redirecting human hand movements to a heterogeneous robotic arm in a multi-task scenario. The device includes:

[0039] The task type assignment unit is used to obtain the visual data and tactile data of the robot hand; process the visual data to obtain the visual embedding vector; process the tactile data to obtain the tactile embedding vector; embed the visual embedding vector and the tactile embedding vector and concatenate them into the pre-trained Perceiver IO model to classify the task type and obtain the task type code;

[0040] The dynamic weight correction unit is used to obtain the current robot hand URDF / XML file, query the task-weight knowledge base according to the task type code, and obtain the basic weight template; according to the template, the physical constraint dynamic correction mechanism is adopted to correct the joint weights of the robot fingers, and the normalized finger weights are obtained through normalization processing;

[0041] The hand key point recognition unit is used to detect the collected hand video using the hand detection module of the MediaPipe framework to obtain the hand area. Based on the hand area, the key point positioning module of the MediaPipe framework is used to extract the key points of the hand, and the vectors between the joints of the hand are calculated by the geometric vector method.

[0042] A calculation and acquisition unit, used to parse and initialize according to the URDF / XML file and pre-configured joint parameters, establish a joint mapping matrix, calculate the adapted Jacobian matrix through slave joint constraint processing; perform forward kinematics mapping and inverse kinematics mapping according to the Jacobian matrix to obtain an adapted standardized robot hand kinematics model;

[0043] The optimization unit is used to perform nonlinear optimization on the vectors between human fingers and robot fingers according to the kinematic model, the normalized finger weights and the vectors between the joints of the human hand, obtain the joint angles of the robot hand, and enable the robot hand to complete the action redirection.

[0044] Optionally, the task type allocation unit is used to:

[0045] The visual embedding vector and the tactile embedding vector are embedded and concatenated and input into the pre-trained Perceiver IO model. The self-attention mechanism and cross-modal query mechanism are used to perform feature fusion to obtain the fused multi-modal features.

[0046] The fused multimodal features are input into the Transformer encoder layer of the Perceiver IO model for deep semantic representation to obtain deep features;

[0047] The deep features are input into the fully connected classification layer and mapped to the task category, and the task type encoding is obtained by calculation through the softmax activation function.

[0048] Optionally, the process of correcting the joint weights of the robot finger is expressed by the following formula (1):

[0049] (1)

[0050] in, Indicates the dynamic correction weight; represents the basic weight of the i-th finger; represents the maximum torque of the i-th finger joint; Indicates the system global safety torque threshold; Indicates the maximum limit of the angle of the i-th finger joint; Indicates the minimum limit of the angle of the i-th finger joint; Indicates the natural range of motion of the i-th finger of a human hand; represents the torque constraint correction term; Represents the motion space constraint correction term.

[0051] Optionally, the calculation and acquisition unit is used to:

[0052] Use yourdfpy toolkit to parse URDF / XML files and extract robot structure and parameters;

[0053] Correcting and supplementing information of the robot structure and parameters according to pre-configured joint parameters to obtain processed robot structure and parameters;

[0054] According to the processed robot structure and parameters, a joint mapping matrix is ​​established;

[0055] According to the joint mapping matrix, the position of the slave joint is calculated to complete the slave joint constraint processing; wherein, the calculation formula of the slave joint position is expressed by the following formula (2):

[0056] (2)

[0057] in, represents the position vector of the slave joint; represents the active joint position vector; represents the transmission ratio vector; represents the position bias vector; Represents vector element-wise multiplication.

[0058] Optionally, the process of calculating the adapted Jacobian matrix is ​​expressed by the following formula (3):

[0059] (3)

[0060] in, represents the adapted Jacobian matrix; represents the target joint Jacobian matrix; represents the Jacobian matrix of the i-th driven joint; represents the transmission ratio of the i-th driven joint.

[0061] Optionally, the calculation formula of the forward kinematics mapping is expressed by the following formula (4):

[0062] (4)

[0063] Where x represents the finger coordinates in Cartesian coordinates; f represents the positive kinematic function;

[0064] The calculation formula of the inverse kinematics mapping is expressed by the following formula (5):

[0065] (5)

[0066] in, Represents the inverse kinematics mapping function.

[0067] Optionally, the problem of nonlinear optimization of the vectors between human fingers and machine fingers is expressed by the following formula (6):

[0068] (6)

[0069]

[0070] in, Represents the basic weight of the i-th finger; represents the regularization term coefficient; Indicates the finger joint angle; Represents the positive kinematics function of the i-th finger of the robot hand, outputting the end position coordinates; represents the 3D coordinates of the end of the i-th finger of a human hand; Indicates the robot's safe default joint, half-open state; Indicates the smallest joint of the finger joint; Indicates the largest joint of the finger joint; represents the joint torque vector, which is obtained by calculating the Jacobian matrix and contact force; Boolean function that detects whether the robot's posture has no collision with the environment.

[0071] On the other hand, a device for redirecting human hand movements to a heterogeneous robotic arm in a multi-tasking scenario is provided, and the device for redirecting human hand movements to a heterogeneous robotic arm in a multi-tasking scenario includes: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, any one of the methods for redirecting human hand movements to a heterogeneous robotic arm in a multi-tasking scenario is implemented.

[0072] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the methods for redirecting human hand movements to heterogeneous robotic hands in the above-mentioned multi-tasking scenario.

[0073] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0074] The embodiment of the present invention first obtains visual data and tactile data of the robot hand; processes the visual data to obtain a visual embedding vector; processes the tactile data to obtain a tactile embedding vector; embeds and splices the visual embedding vector and the tactile embedding vector into a pre-trained Perceiver IO model to classify the task type and obtain the task type code; obtains the current robot hand URDF / XML file, queries the task-weight knowledge base according to the task type code, and obtains the basic weight template; secondly, according to the template, adopts the physical constraint dynamic correction mechanism to correct the joint weights of the robot fingers, and obtains the normalized finger weights through normalization processing; adopts the hand detection module of the MediaPipe framework to detect the collected human hand video to obtain the human hand area, and according to the human hand area; adopts the key point positioning module of the MediaPipe framework to extract the key points of the human hand, and calculates the vectors between the joints of the human hand through the geometric vector method;

[0075] According to the URDF / XML file and the pre-configured joint parameters, parsing and initialization are performed to establish the joint mapping matrix. The adapted Jacobian matrix is ​​calculated through the slave joint constraint processing. According to the Jacobian matrix, forward kinematics mapping and inverse kinematics mapping are performed to obtain the adapted standardized robot hand kinematic model. Finally, according to the kinematic model, the normalized finger weights and the vectors between the joints of the human hand, the vectors between the human fingers and the robot fingers are nonlinearly optimized to obtain the joint angles of the robot hand, so that the robot hand can complete the motion redirection.

[0076] The embodiment of the present invention solves the problem of over-reliance on specific tasks and models in traditional solutions by constructing a task-weight dynamic coupling mechanism and an incremental semantic knowledge base; the present invention does not require manual adjustment of weight parameters in mixed task scenarios such as grasping, lifting, and precision assembly. The use of the present invention can improve the generalization ability across tasks and heterogeneous models, and can support zero-sample adaptation from a 2-DOF gripper to a 21-DOF bionic hand. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0078] Figure 1 It is a flow chart of a method for redirecting human hand movements to heterogeneous robotic hands in a multi-task scenario provided by an embodiment of the present invention;

[0079] Figure 2 It is a structural schematic diagram of redirecting human hand movements to heterogeneous robotic arms in a multi-task scenario provided by an embodiment of the present invention;

[0080] Figure 3 It is a block diagram of a device for redirecting human hand movements to heterogeneous robotic arms in a multi-task scenario provided by an embodiment of the present invention;

[0081] Figure 4 It is a structural schematic diagram of a device for redirecting human hand movements to heterogeneous robotic arms in a multi-tasking scenario provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0082] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0083] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0084] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0085] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0086] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0087] The embodiment of the present invention provides a method for redirecting human hand movements to heterogeneous robotic arms in a multi-tasking scenario. The method can be implemented by a device for redirecting human hand movements to heterogeneous robotic arms in a multi-tasking scenario. The device for redirecting human hand movements to heterogeneous robotic arms in a multi-tasking scenario can be a terminal or a server. Figure 1 The flowchart of the method for redirecting human hand movements to heterogeneous robotic hands in a multi-task scenario is shown. The processing flow of the method may include the following steps:

[0088] S1. Obtain visual data and tactile data of the robot arm; process the visual data to obtain a visual embedding vector; process the tactile data to obtain a tactile embedding vector; embed and concatenate the visual embedding vector and the tactile embedding vector and input them into the pre-trained Perceiver IO model to classify the task type and obtain the task type code.

[0089] Among them, an RGB-D camera is used to collect visual point cloud data of objects and robot arms; an array tactile sensor is used to collect tactile time series signal data.

[0090] In a feasible implementation, the visual point cloud data is downsampled and divided into local patches, and processed data is obtained by adding position encoding; the processed data is converted into a visual embedding vector of fixed dimension through point cloud embedding technology.

[0091] In a feasible implementation, the tactile time series signal data is normalized, one-dimensional convolution is used to embed each signal channel, and sequence position information is added to obtain a tactile embedding vector.

[0092] Among them, the Perceiver IO model architecture is a machine learning architecture that can process input and output data of any structure.

[0093] Optionally, the specific implementation process of S1 includes S11-S13:

[0094] S11. Embed the visual embedding vector and the tactile embedding vector and concatenate them into the pre-trained PerceiverIO model, perform feature fusion through the self-attention mechanism and cross-modal query mechanism, and obtain the fused multi-modal features;

[0095] In a feasible implementation, the visual embedding vector and the tactile embedding vector are embedded and spliced ​​to obtain a spliced ​​input vector; after linear mapping and position encoding the spliced ​​input vector, the input data of the Perceiver IO model is obtained.

[0096] Among them, the self-attention mechanism and cross-modal query mechanism effectively capture the correlation between different modalities and obtain the fused feature fusion.

[0097] S12, the fused multimodal features are input into the Transformer encoder layer of the Perceiver IO model for deep semantic representation to obtain deep features;

[0098] S13. The deep features are input into the fully connected classification layer and mapped to the task category. The task type encoding is obtained by calculation through the softmax activation function.

[0099] Among them, the task categories include grasping tasks, pinching tasks and lifting tasks; the softmax activation function is used for calculation to obtain the task type encoding and corresponding semantic labels, including the judgment results and confidence of grasping, pinching and lifting tasks.

[0100] S2. Obtain the current robot hand URDF / XML file, query the task-weight knowledge base according to the task type code, and obtain the basic weight template; according to the template, use the physical constraint dynamic correction mechanism to correct the joint weights of the robot fingers, and obtain the normalized finger weights through normalization processing.

[0101] In a feasible implementation, the process of obtaining the basic weight template, for example, using inspire_hand to grasp a cup with a handle task, the weight vector of the five fingers corresponding to the task number kneading task is [0.8, 0.8, 0.1, 0.1,0.1], which indicates that the thumb and index finger have greater weights in the kneading task.

[0102] Among them, the physical constraint dynamic correction mechanism can adaptively adjust the weight of each joint to ensure the safe operation of the mechanism, avoid reaching the physical limit while maintaining the task completion capability, and achieve a dynamic balance between task requirements and hardware constraints. According to the physical constraints of the robot, the weights are dynamically corrected, including joint limits and maximum clamping force.

[0103] Optionally, the process of correcting the joint weights of the robot finger is expressed by the following formula (1):

[0104] (1)

[0105] in, represents the dynamic correction weight of the i-th finger joint; represents the basic weight of the i-th finger; represents the maximum torque of the i-th finger joint; Indicates the system global safety torque threshold; Indicates the maximum limit of the angle of the i-th finger joint; Indicates the minimum limit of the angle of the i-th finger joint; Indicates the natural range of motion of the i-th finger of a human hand; represents the torque constraint correction term; Represents the motion space constraint correction term.

[0106] Among them, the torque constraint correction term reflects the ratio of the actual available torque of each joint to the system safety threshold; when the joint torque approaches the limit, the weight of the joint is reduced to ensure safe operation; the torque constraint correction term is used to prevent joint overload and avoid mechanical damage.

[0107] Among them, the motion space constraint correction term represents the ratio of the actual available motion range of the joint to the standard motion range of the human hand; when the joint motion space is limited, the weight of the joint is reduced accordingly; the feasibility of motion planning is guaranteed by the motion space constraint correction term.

[0108] In a feasible implementation, the process of obtaining the normalized finger weights is expressed by the following formula (2):

[0109] (2)

[0110] in, Represents the normalized finger weight.

[0111] S3. Use the hand detection module of the MediaPipe framework to detect the collected human hand video and obtain the human hand area. Based on the human hand area; use the key point positioning module of the MediaPipe framework to extract the key points of the human hand, and calculate the vectors between the joints of the human hand through the geometric vector method.

[0112] In a feasible implementation, a hand detection module of the MediaPipe framework is used to locate the human hand region from the human hand video, and 21 key points of the hand are extracted through a key point localization module of the MediaPipe framework.

[0113] S4. According to the URDF / XML file and the pre-configured joint parameters, parse and initialize, establish the joint mapping matrix, calculate the adapted Jacobian matrix through the slave joint constraint processing; perform forward kinematics mapping and inverse kinematics mapping according to the Jacobian matrix to obtain the adapted standardized robot hand kinematic model.

[0114] The pre-configured joint parameters include active joint parameters and passive joint parameters.

[0115] Among them, the adapted standardized robot hand kinematic model includes: a unified forward kinematics interface, a unified inverse kinematics interface, a slave joint constraint processing mechanism and a Jacobian matrix adaptation mechanism.

[0116] Optionally, the specific implementation process of S4 includes S41-S44:

[0117] S41. Use yourdfpy toolkit to parse URDF / XML files and extract robot structure and parameters;

[0118] S42, correcting and supplementing information of the robot structure and parameters according to the pre-configured joint parameters to obtain processed robot structure and parameters;

[0119] S43, establishing a joint mapping matrix according to the processed robot structure and parameters;

[0120] Among them, by establishing the joint mapping matrix, the legality of the joint configuration can be verified to ensure that the mechanism constraints meet the requirements.

[0121] S44, according to the joint mapping matrix, the driven joint position is calculated to complete the driven joint constraint processing; wherein, the calculation formula of the driven joint position is expressed by the following formula (3):

[0122] (3)

[0123] in, represents the position vector of the slave joint; represents the active joint position vector; represents the transmission ratio vector; represents the position bias vector; Represents vector element-wise multiplication.

[0124] Among them, the driven joint refers to the joint whose movement is determined by other joints, and its kinematic characteristics need special treatment.

[0125] Optionally, the process of calculating the adapted Jacobian matrix is ​​expressed by the following formula (4):

[0126] (4)

[0127] in, represents the adapted Jacobian matrix; represents the target joint Jacobian matrix; represents the Jacobian matrix of the i-th driven joint; represents the transmission ratio of the i-th driven joint.

[0128] Optionally, the calculation formula of the forward kinematics mapping is expressed by the following formula (5):

[0129] (5)

[0130] Where x represents the finger coordinates in Cartesian coordinates; f represents the positive kinematic function;

[0131] In a feasible implementation, the specific implementation process of forward kinematics mapping includes:

[0132] (1) Calculate the position of the driven joint based on the active joint position and transmission ratio;

[0133] (2) Combine the active joint positions and the driven joint positions to construct a complete joint model;

[0134] (3) Based on the complete joint model, the forward kinematics algorithm is used to calculate the position and posture of each end fingertip.

[0135] Among them, the forward kinematics algorithm is a method mastered by technical personnel in this field, which calculates the position and posture of the robot end execution through given robot joint variables.

[0136] The calculation formula of the inverse kinematics mapping is expressed by the following formula (6):

[0137] (6)

[0138] in, Represents the inverse kinematics mapping function.

[0139] In a feasible implementation manner, the specific implementation process of inverse kinematics mapping includes:

[0140] (1) Calculate the adapted Jacobian matrix;

[0141] (2) Based on the adapted Jacobian matrix, solve the problem of minimizing the position error;

[0142] (3) Based on minimizing the position error, joint limitation processing is performed to keep the joints within the physical limitation range.

[0143] S5. According to the kinematic model, the normalized finger weights and the vectors between the joints of the human hand, the vectors between the human fingers and the robot fingers are nonlinearly optimized to obtain the angles of the joints of the robot hand, so that the robot hand can complete the movement redirection.

[0144] Optionally, the problem of nonlinear optimization of the vectors between human fingers and machine fingers in S5 is expressed by the following formula (7):

[0145] (7)

[0146]

[0147] in, Represents the basic weight of the i-th finger; represents the regularization term coefficient; Indicates the finger joint angle; Represents the positive kinematics function of the i-th finger of the robot hand, outputting the end position coordinates; represents the 3D coordinates of the end of the i-th finger of a human hand; Indicates the robot's safe default joint, half-open state; Indicates the smallest joint of the finger joint; Indicates the largest joint of the finger joint; represents the joint torque vector, which is obtained by calculating the Jacobian matrix and contact force; Boolean function that detects whether the robot's posture has no collision with the environment.

[0148] Among them, the vectors between human fingers and robot fingers are nonlinearly optimized to obtain the angles of each joint of the robot hand, so that the robot can realize the functions of human hands and the robot hand can complete motion redirection.

[0149] The present invention adopts a dynamic weight allocation mechanism driven by task semantics to identify task types such as pinching, grasping, and lifting within 0.1 seconds; builds a task-weight knowledge base to dynamically set the optimization weight of each finger, solves the problem of deformation caused by excessive squeezing of the back three fingers when grasping thin-walled objects in traditional methods, and improves the success rate of the operation. After identifying the task type and determining the finger importance weight, real-time redirection can be performed according to the robot URDF / XML file of the heterogeneous robot, so that different settings do not need to be made for different robots, which improves the scene generalization and hardware adaptability.

[0150] Among them, the Realsense D435i RGBD camera used in the present invention can collect 1280×720 resolution point cloud, depth accuracy ±1mm, and transmit it to the graphics workstation via USB3.0. An array tactile sensor is used to collect tactile data, which is integrated in the fingertips of the dexterous hand, with 5-channel pressure detection (0-10N, ±0.1N), and a 100Hz sampling rate output via the SPI interface; wherein, the graphics workstation is an NVIDIA RTX 4090 GPU; wherein, the real-time controller is an Intel i9 processor to process kinematic adaptation and optimization solution; the hardware used is the dexterous hand of the time, which is a 6-degree-of-freedom bionic hand, and the joints use MaxonEC-i40 motors with a control accuracy of 0.1°. The hardware seven-degree-of-freedom robotic arm used is a load of 5kg, synchronously controlled by the EtherCAT bus, and the end repeat positioning accuracy is ±0.05mm. The shadow hand, allogro and ability robotic hands are simulated by the method proposed in the present invention.

[0151] Among them, Figure 2 The invention is a structural diagram of redirecting a human hand action to a heterogeneous robot hand in a multi-task scenario provided by an embodiment of the present invention; obtaining a human hand video and a robot hand URDF / XML file; obtaining the human hand key points and joint angles from the human hand video through a gesture recognition detection module; unifying the redirection class of the robot hand URDF / XML file through a motion adapter and a customized configuration file according to different hands and redirection types; obtaining a specific task scenario, and obtaining the normalized finger weights through a multimodal classifier and dynamic allocation of task weights; performing nonlinear optimization on the vectors between human fingers and between robot fingers according to the normalized finger weights and the human hand key points and joint angles, obtaining the joint angles of the robot hand, and enabling the robot hand to complete the action redirection.

[0152] The embodiment of the present invention first obtains visual data and tactile data of the robot hand; processes the visual data to obtain a visual embedding vector; processes the tactile data to obtain a tactile embedding vector; embeds and splices the visual embedding vector and the tactile embedding vector into a pre-trained Perceiver IO model to classify the task type and obtain the task type code; obtains the current robot hand URDF / XML file, queries the task-weight knowledge base according to the task type code, and obtains the basic weight template; secondly, according to the template, adopts the physical constraint dynamic correction mechanism to correct the joint weights of the robot fingers, and obtains the normalized finger weights through normalization processing; adopts the hand detection module of the MediaPipe framework to detect the collected human hand video to obtain the human hand area, and according to the human hand area; adopts the key point positioning module of the MediaPipe framework to extract the key points of the human hand, and calculates the vectors between the joints of the human hand through the geometric vector method;

[0153] According to the URDF / XML file and the pre-configured joint parameters, parsing and initialization are performed to establish the joint mapping matrix. The adapted Jacobian matrix is ​​calculated through the slave joint constraint processing. According to the Jacobian matrix, forward kinematics mapping and inverse kinematics mapping are performed to obtain the adapted standardized robot hand kinematic model. Finally, according to the kinematic model, the normalized finger weights and the vectors between the joints of the human hand, the vectors between the human fingers and the robot fingers are nonlinearly optimized to obtain the joint angles of the robot hand, so that the robot hand can complete the motion redirection.

[0154] The embodiment of the present invention solves the problem of over-reliance on specific tasks and models in traditional solutions by constructing a task-weight dynamic coupling mechanism and an incremental semantic knowledge base; the present invention does not require manual adjustment of weight parameters in mixed task scenarios such as grasping, lifting, and precision assembly. The use of the present invention can improve the generalization ability across tasks and heterogeneous models, and can support zero-sample adaptation from a 2-DOF gripper to a 21-DOF bionic hand.

[0155] Figure 3 The invention is a block diagram of a device for redirecting human hand movements to heterogeneous robotic arms in a multi-task scenario according to an exemplary embodiment. The device is used in a method for redirecting human hand movements to heterogeneous robotic arms in a multi-task scenario. Figure 3 The device includes a task type assignment unit 310, a dynamic weight correction unit 320, a hand key point recognition unit 330, a calculation and acquisition unit 340 and an optimization unit 350. Among them:

[0156] The task type assignment unit 310 is used to obtain visual data and tactile data of the robot hand; process the visual data to obtain a visual embedding vector; process the tactile data to obtain a tactile embedding vector; embed and concatenate the visual embedding vector and the tactile embedding vector and input them into a pre-trained Perceiver IO model to perform task type classification and obtain a task type code;

[0157] The dynamic weight correction unit 320 is used to obtain the current robot hand URDF / XML file, query the task-weight knowledge base according to the task type code, and obtain the basic weight template; according to the template, the physical constraint dynamic correction mechanism is adopted to correct the joint weights of the robot fingers, and the normalized finger weights are obtained through normalization processing;

[0158] The hand key point recognition unit 330 is used to detect the collected hand video using the hand detection module of the MediaPipe framework to obtain the hand area, and extract the hand key points based on the hand area; and calculate the vectors between the joints of the hand by the geometric vector method;

[0159] The calculation and acquisition unit 340 is used to parse and initialize according to the URDF / XML file and the pre-configured joint parameters, establish a joint mapping matrix, calculate the adapted Jacobian matrix through slave joint constraint processing; perform forward kinematics mapping and inverse kinematics mapping according to the Jacobian matrix to obtain an adapted standardized robot hand kinematics model;

[0160] The optimization unit 350 is used to perform nonlinear optimization on the vectors between human fingers and robot fingers according to the kinematic model, the normalized finger weights and the vectors between the joints of the human hand, obtain the joint angles of the robot hand, and enable the robot hand to complete the action redirection.

[0161] Optionally, the task type assignment unit 310 is used to:

[0162] The visual embedding vector and the tactile embedding vector are embedded and concatenated and input into the pre-trained Perceiver IO model. The self-attention mechanism and cross-modal query mechanism are used to perform feature fusion to obtain the fused multi-modal features.

[0163] The fused multimodal features are input into the Transformer encoder layer of the Perceiver IO model for deep semantic representation to obtain deep features;

[0164] The deep features are input into the fully connected classification layer and mapped to the task category, and the task type encoding is obtained by calculation through the softmax activation function.

[0165] Optionally, the process of correcting the joint weights of the robot finger is expressed by the following formula (1):

[0166] (1)

[0167] in, Indicates the dynamic correction weight; Represents the basic weight of the i-th finger; represents the maximum torque of the i-th finger joint; Indicates the system global safety torque threshold; Indicates the maximum limit of the angle of the i-th finger joint; Indicates the minimum limit of the angle of the i-th finger joint; Indicates the natural range of motion of the i-th finger of a human hand; represents the torque constraint correction term; Represents the motion space constraint correction term.

[0168] Optionally, the calculation and acquisition unit 340 is used to:

[0169] Use yourdfpy toolkit to parse URDF / XML files and extract robot structure and parameters;

[0170] Correcting and supplementing information of the robot structure and parameters according to pre-configured joint parameters to obtain processed robot structure and parameters;

[0171] According to the processed robot structure and parameters, a joint mapping matrix is ​​established;

[0172] According to the joint mapping matrix, the position of the slave joint is calculated to complete the slave joint constraint processing; wherein, the calculation formula of the slave joint position is expressed by the following formula (2):

[0173] (2)

[0174] in, represents the position vector of the slave joint; represents the active joint position vector; represents the transmission ratio vector; represents the position bias vector; Represents vector element-wise multiplication.

[0175] Optionally, the process of calculating the adapted Jacobian matrix is ​​expressed by the following formula (3):

[0176] (3)

[0177] in, represents the adapted Jacobian matrix; represents the target joint Jacobian matrix; represents the Jacobian matrix of the i-th driven joint; represents the transmission ratio of the i-th driven joint.

[0178] Optionally, the calculation formula of the forward kinematics mapping is expressed by the following formula (4):

[0179] (4)

[0180] Where x represents the finger coordinates in Cartesian coordinates; f represents the positive kinematic function;

[0181] The calculation formula of the inverse kinematics mapping is expressed by the following formula (5):

[0182] (5)

[0183] in, Represents the inverse kinematics mapping function.

[0184] Optionally, the problem of nonlinear optimization of the vectors between human fingers and machine fingers is expressed by the following formula (6):

[0185] (6)

[0186]

[0187] in, represents the basic weight of the i-th finger; represents the regularization term coefficient; Indicates the finger joint angle; Represents the positive kinematics function of the i-th finger of the robot hand, outputting the end position coordinates; represents the 3D coordinates of the end of the i-th finger of a human hand; Indicates the robot's safe default joint, half-open state; Indicates the smallest joint of the finger joint; Indicates the largest joint of the finger joint; represents the joint torque vector, which is obtained by calculating the Jacobian matrix and contact force; Boolean function that detects whether the robot's posture has no collision with the environment.

[0188] The embodiment of the present invention first obtains visual data and tactile data of the robot hand; processes the visual data to obtain a visual embedding vector; processes the tactile data to obtain a tactile embedding vector; embeds and splices the visual embedding vector and the tactile embedding vector into a pre-trained Perceiver IO model to classify the task type and obtain the task type code; obtains the current robot hand URDF / XML file, queries the task-weight knowledge base according to the task type code, and obtains the basic weight template; secondly, according to the template, adopts the physical constraint dynamic correction mechanism to correct the joint weights of the robot fingers, and obtains the normalized finger weights through normalization processing; adopts the hand detection module of the MediaPipe framework to detect the collected human hand video to obtain the human hand area, and according to the human hand area; adopts MediaPipe The key point positioning module of the framework extracts the key points of the human hand and calculates the vectors between the joints of the human hand through the geometric vector method; it parses and initializes the URDF / XML file and the pre-configured joint parameters, establishes the joint mapping matrix, and calculates the adapted Jacobian matrix through the slave joint constraint processing; it performs forward kinematic mapping and inverse kinematic mapping based on the Jacobian matrix to obtain the adapted standardized robot hand kinematic model; finally, based on the kinematic model, the normalized finger weights and the vectors between the joints of the human hand, it performs nonlinear optimization on the vectors between the human fingers and the robot fingers to obtain the joint angles of the robot hand, so that the robot hand can complete the motion redirection.

[0189] The embodiment of the present invention solves the problem of over-reliance on specific tasks and models in traditional solutions by constructing a task-weight dynamic coupling mechanism and an incremental semantic knowledge base; the present invention does not require manual adjustment of weight parameters in mixed task scenarios such as grasping, lifting, and precision assembly. The use of the present invention can improve the generalization ability across tasks and heterogeneous models, and can support zero-sample adaptation from a 2-DOF gripper to a 21-DOF bionic hand.

[0190] Figure 4 is a schematic diagram of a structure of a device for redirecting human hand movements to heterogeneous robotic arms in a multi-task scenario provided by an embodiment of the present invention, such as Figure 4 As shown, the device for redirecting human hand movements to heterogeneous robotic hands in a multi-task scenario may include the above Figure 3 The device for redirecting human hand movements to heterogeneous robotic arms in a multi-tasking scenario is shown. Optionally, the device 410 for redirecting human hand movements to heterogeneous robotic arms in a multi-tasking scenario may include a first processor 2001 .

[0191] Optionally, the device 410 for redirecting human hand movements to a heterogeneous robotic hand in a multi-tasking scenario may further include a memory 2002 and a transceiver 2003 .

[0192] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0193] Combine the following Figure 4 The components of the device 410 for redirecting human hand movements to heterogeneous robotic arms in a multi-task scenario are specifically introduced:

[0194] The first processor 2001 is the control center of the device 410 for redirecting the human hand action to the heterogeneous robot hand in the multi-task scenario, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0195] Optionally, the first processor 2001 can execute various functions of the device 410 for redirecting human hand movements to heterogeneous robotic arms in a multi-tasking scenario by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0196] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0197] In a specific implementation, as an embodiment, the device 410 for redirecting human hand movements to heterogeneous robotic arms in a multi-tasking scenario may also include multiple processors, such as Figure 4 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0198] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0199] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and redirect the human hand action to the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0200] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0201] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0202] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and redirect the human hand action in the multi-tasking scenario to the interface circuit of the device 410 of the heterogeneous robot hand ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0203] It should be noted that Figure 4 The structure of the device 410 for redirecting human hand actions to heterogeneous robotic arms in a multi-tasking scenario shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0204] In addition, the technical effect of the device 410 for redirecting human hand movements to heterogeneous robotic arms in multi-tasking scenarios can refer to the technical effect of the method for redirecting human hand movements to heterogeneous robotic arms in multi-tasking scenarios described in the above method embodiment, and will not be repeated here.

[0205] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0206] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0207] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0208] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0209] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0210] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0211] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0212] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0213] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0214] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0215] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0216] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0217] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for redirecting human hand movements to heterogeneous robotic hands in a multi-task scenario, characterized in that: The method comprises: S1. Obtain visual data and tactile data of the robot; process the visual data to obtain a visual embedding vector; process the tactile data to obtain a tactile embedding vector; embed and concatenate the visual embedding vector and the tactile embedding vector and input them into a pre-trained Perceiver IO model to classify the task type and obtain the task type code; S2. Obtain the current robot hand URDF / XML file, query the task-weight knowledge base according to the task type code, and obtain the basic weight template; according to the template, use the physical constraint dynamic correction mechanism to correct the joint weights of the robot fingers, and obtain the normalized finger weights through normalization processing; S3, using the hand detection module of the MediaPipe framework to detect the collected human hand video, obtain the human hand area, and based on the human hand area; using the key point positioning module of the MediaPipe framework to extract the key points of the human hand, and calculate the vectors between the joints of the human hand through the geometric vector method; S4, parsing and initializing according to the URDF / XML file and pre-configured joint parameters, establishing a joint mapping matrix, calculating the adapted Jacobian matrix through slave joint constraint processing; performing forward kinematics mapping and inverse kinematics mapping according to the Jacobian matrix to obtain an adapted standardized robot hand kinematics model; S5. According to the kinematic model, the normalized finger weights and the vectors between the joints of the human hand, nonlinear optimization is performed on the vectors between the human fingers and the robot fingers to obtain the joint angles of the robot hand so that the robot hand can complete the action redirection.

2. The method for redirecting human hand movements to heterogeneous robotic hands in a multi-tasking scenario according to claim 1, characterized in that: The S1 embeds and concatenates the visual embedding vector and the tactile embedding vector and inputs them into the pre-trained PerceiverIO model to classify the task type and obtain the task type code, including: S11, embed the visual embedding vector and the tactile embedding vector and concatenate them into the pre-trained Perceiver IO model, perform feature fusion through the self-attention mechanism and cross-modal query mechanism, and obtain the fused multi-modal features; S12, the fused multimodal features are input into the Transformer encoder layer of the Perceiver IO model for deep semantic representation to obtain deep features; S13. The deep features are input into the fully connected classification layer and mapped to the task category. The task type encoding is obtained by calculation through the softmax activation function.

3. The method for redirecting human hand movements to heterogeneous robotic hands in a multi-tasking scenario according to claim 1, characterized in that: The process of correcting the joint weights of the robot finger is expressed by the following formula (1): （1） in, Indicates the dynamic correction weight; represents the basic weight of the i-th finger; represents the maximum torque of the i-th finger joint; Indicates the system global safety torque threshold; Indicates the maximum limit of the angle of the i-th finger joint; Indicates the minimum limit of the angle of the i-th finger joint; Indicates the natural range of motion of the i-th finger of a human hand; represents the torque constraint correction term; Represents the motion space constraint correction term.

4. The method for redirecting human hand movements to heterogeneous robotic hands in a multi-tasking scenario according to claim 1, characterized in that: The S4 parses and initializes the URDF / XML file and pre-configured joint parameters, establishes a joint mapping matrix, and processes the driven joint constraints, including: S41. Use yourdfpy toolkit to parse URDF / XML files and extract robot structure and parameters; S42, correcting and supplementing information of the robot structure and parameters according to the pre-configured joint parameters to obtain processed robot structure and parameters; S43, establishing a joint mapping matrix according to the processed robot structure and parameters; S44, according to the joint mapping matrix, the driven joint position is calculated to complete the driven joint constraint processing; wherein the calculation formula of the driven joint position is expressed by the following formula (2): （2） in, represents the position vector of the slave joint; represents the active joint position vector; represents the transmission ratio vector; represents the position bias vector; Represents vector element-wise multiplication.

5. The method for redirecting human hand movements to heterogeneous robotic hands in a multi-tasking scenario according to claim 1, characterized in that: The process of calculating the adapted Jacobian matrix is ​​expressed by the following formula (3): （3） in, represents the adapted Jacobian matrix; represents the target joint Jacobian matrix; represents the Jacobian matrix of the i-th driven joint; represents the transmission ratio of the i-th driven joint.

6. The method for redirecting human hand movements to heterogeneous robotic hands in a multi-tasking scenario according to claim 1, characterized in that: The calculation formula of the forward kinematics mapping is expressed by the following formula (4): （4） Where x represents the finger coordinates in Cartesian coordinates; f represents the positive kinematic function; The calculation formula of the inverse kinematics mapping is expressed by the following formula (5): （5） in, Represents the inverse kinematics mapping function.

7. The method for redirecting human hand movements to heterogeneous robotic hands in a multi-tasking scenario according to claim 1, characterized in that: The problem of nonlinear optimization of the vectors between human fingers and machine fingers in S5 is expressed by the following formula (6): （6） ； in, represents the basic weight of the i-th finger; represents the regularization term coefficient; Indicates the finger joint angle; Represents the positive kinematics function of the i-th finger of the robot hand, outputting the end position coordinates; represents the 3D coordinates of the end of the i-th finger of a human hand; Indicates the robot's safe default joint, half-open state; Indicates the smallest joint of the finger joint; Indicates the largest joint of the finger joint; represents the joint torque vector, which is obtained by calculating the Jacobian matrix and contact force; Boolean function that detects whether the robot's posture has no collision with the environment.

8. A device for redirecting human hand movements to heterogeneous robotic arms in a multi-task scenario, wherein the device for redirecting human hand movements to heterogeneous robotic arms in a multi-task scenario is used to implement the method for redirecting human hand movements to heterogeneous robotic arms in a multi-task scenario as claimed in any one of claims 1 to 7, characterized in that: The device comprises: The task type assignment unit is used to obtain the visual data and tactile data of the robot hand; process the visual data to obtain the visual embedding vector; process the tactile data to obtain the tactile embedding vector; embed the visual embedding vector and the tactile embedding vector and concatenate them into the pre-trained Perceiver IO model to classify the task type and obtain the task type code; The dynamic weight correction unit is used to obtain the current robot hand URDF / XML file, query the task-weight knowledge base according to the task type code, and obtain the basic weight template; according to the template, the physical constraint dynamic correction mechanism is adopted to correct the joint weights of the robot fingers, and the normalized finger weights are obtained through normalization processing; The hand key point recognition unit is used to detect the collected hand video using the hand detection module of the MediaPipe framework to obtain the hand area. Based on the hand area, the key point positioning module of the MediaPipe framework is used to extract the key points of the hand, and the vectors between the joints of the hand are calculated by the geometric vector method. A calculation and acquisition unit, used to parse and initialize according to the URDF / XML file and pre-configured joint parameters, establish a joint mapping matrix, calculate the adapted Jacobian matrix through slave joint constraint processing; perform forward kinematics mapping and inverse kinematics mapping according to the Jacobian matrix to obtain an adapted standardized robot hand kinematics model; The optimization unit is used to perform nonlinear optimization on the vectors between human fingers and robot fingers according to the kinematic model, the normalized finger weights and the vectors between the joints of the human hand, obtain the joint angles of the robot hand, and enable the robot hand to complete the action redirection.

9. A device for redirecting human hand movements to heterogeneous robotic arms in a multi-tasking scenario, characterized in that: The device for redirecting human hand movements to heterogeneous robotic arms in the multi-task scenario includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images