Dexterous arm hand isomorphic teleoperation motion mapping method and system

Abstract

The application provides a dexterous arm hand isomeric teleoperation action mapping method and system, and relates to the technical field of robot teleoperation. The method comprises the following steps: collecting three-dimensional space position data of human hand key points; constructing a mapping function from the three-dimensional space position data of the human hand key points to robot joint angles; wherein the robot comprises a mechanical arm and a dexterous hand; constructing a comprehensive weighted optimization objective function, which is a weighted sum of multiple optimization sub-target items; optimizing the mapping function according to the three-dimensional space position data of the human hand key points with the comprehensive weighted optimization objective function as the optimization criterion to obtain an optimized mapping function; in real-time teleoperation, inputting the real-time collected three-dimensional space position data of the human hand key points into the optimized mapping function to obtain target joint angles of the robot, so as to drive the robot to perform corresponding actions.

Description

Technical Field

[0001] This application relates to the field of robot teleoperation technology, specifically to a method and system for mapping heterogeneous teleoperation motions of a dexterous arm. Background Technology

[0002] Teleoperation technology refers to the real-time mapping of human operator movements to robots, enabling them to remotely perform complex tasks. It is a crucial foundation for fields such as embodied intelligence, telemedicine, deep-sea exploration, and operations in hazardous environments. As a humanoid end effector with high degrees of freedom, the dexterous hand offers immense potential for dexterous operation in teleoperation systems. The key to achieving dexterous teleoperation lies in accurately mapping the natural movements of the human hand to the robot's dexterous hand.

[0003] However, dexterous hand teleoperation technology still faces multiple challenges. First, due to the significant heterogeneity between the mechanical structure and joint configuration of the robotic dexterous hand and the human hand, existing mapping methods have limitations in reproducing complex human hand movements, causing the robot's gestures to deviate from the operator's true intentions. Second, in existing teleoperation systems, strict alignment of the wrist position often affects the fingertip pose accuracy, which is crucial for fine manipulation, making it difficult to achieve fine tasks such as grasping. Third, motion sensing devices inevitably have observation errors when collecting data. When the fingertips actually make contact but the sensing results show a gap, the system struggles to complete accurate physical contact operations, resulting in poor robustness. Furthermore, the high degrees of freedom of the dexterous hand means that motion mapping involves high-dimensional nonlinear optimization problems, resulting in high online computational overhead, high system response latency, and limited control frequency, making it difficult to meet the requirements of high-quality real-time teleoperation.

[0004] Therefore, there is an urgent need for a dexterous arm telemanipulation motion mapping method that can take into account fingertip precision, morphological naturalness, system robustness, and low latency response in heterogeneous structures. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for mapping movements of heterogeneous teleoperations of a dexterous arm and hand, which can achieve dexterous arm and hand teleoperation control that takes into account fingertip accuracy, movement naturalness, system robustness and low latency response under heterogeneous structure.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows: A first aspect of this application discloses a method for mapping heterogeneous teleoperation movements of a dexterous arm, the method comprising: Collect three-dimensional spatial position data of key points on the human hand; Construct a mapping function from the three-dimensional spatial position data of the key points of the human hand to the joint angles of the robot; wherein the robot includes a robotic arm and a dexterous hand; A comprehensive weighted optimization objective function is constructed, which is a weighted sum of multiple optimization sub-objectives. The multiple optimization sub-objectives include at least: a first sub-objective for constraining the positioning accuracy of the robot's fingertips, a second sub-objective for constraining the naturalness of the robot's motion shape, and a third sub-objective for compensating for human hand perception errors. Using the comprehensive weighted optimization objective function as the optimization criterion, the mapping function is optimized based on the three-dimensional spatial position data of the key points of the human hand to obtain the optimized mapping function; In real-time teleoperation, the three-dimensional spatial position data of the key points of the human hand collected in real time are input into the optimized mapping function to obtain the target joint angle of the robot, so as to drive the robot to perform corresponding actions.

[0007] Optionally, the first sub-target item is constructed according to the following steps: The position of the target fingertip of the human hand is determined as the global position reference, and the desired wrist orientation of the robot is determined. Based on the mapping function, determine the position of the robot's target fingertip corresponding to the position of the human hand's target fingertip, as well as the robot's actual wrist orientation; Calculate the positional deviation between the target fingertip position of the human hand and the target fingertip position of the robot, and calculate the orientation deviation between the desired wrist orientation of the robot and the actual wrist orientation of the robot; The position deviation and the orientation deviation are weighted and summed to obtain the first sub-target item; wherein the weight of the position deviation is greater than the weight of the orientation deviation, so as to allow the robot wrist to make local adaptive adjustments while prioritizing the positioning accuracy of the target fingertip position.

[0008] Optionally, the target fingertip position of the human hand is the fingertip position of the human thumb; the position deviation is the deviation between the fingertip position of the human thumb and the fingertip position of the robot thumb; the orientation deviation is the angular deviation between the robot's desired wrist orientation and the robot's actual wrist orientation.

[0009] Optionally, the second sub-target item is constructed according to the following steps: Based on the three-dimensional spatial position data of the key points of the human hand, the current relative position of the fingertips relative to the palm of the human hand is determined, and at the current relative position, at least one random small relative displacement vector is generated, which represents the local movement trend direction of the fingertips relative to the palm of the human hand. Based on the mapping function and the robot's forward kinematics model, calculate the predicted relative displacement vector of the robot's fingertip relative to the robot's palm, corresponding to the tiny relative displacement vector; The minute relative displacement vector and the predicted relative displacement vector are normalized to unit direction vectors, and the difference between the two normalized unit direction vectors is calculated as a difference measure. The generated minute relative displacement vectors and the difference measures corresponding to each finger are accumulated or averaged to obtain the second sub-target item.

[0010] Optionally, the third sub-target item is constructed according to the following steps: Based on the three-dimensional spatial position data of the key points of the human hand, the relative position vector between the tip of the thumb and the tips of the other fingers is determined, and the magnitude of each relative position vector is calculated as the fingertip distance perceived by the human hand. Based on the fingertip distance perceived by the human hand, the compensated target distance is calculated using a preset distance compensation function; wherein, the distance compensation function is used to map the fingertip distance perceived by the human hand from the original interval containing non-zero perception error to the target interval containing zero contact point. Based on the direction of the relative position vector and the compensated target distance, the remapped expected relative position vector is obtained; Based on the current mapping function and the robot's forward kinematics model, calculate the actual relative position vectors of the robot side corresponding to the tip of the thumb and the tips of the other fingers; Calculate the deviation between the desired relative position vector after remapping and the actual relative position vector on the robot side; A weighting coefficient is determined based on the fingertip distance perceived by the human hand, and the weighting coefficient takes a larger value when the fingertip distance is small; The product of the deviation corresponding to each finger and the weighting coefficient is accumulated or averaged to obtain the third sub-target item.

[0011] Optionally, based on the fingertip distance perceived by the human hand, the compensated target distance is calculated using a preset distance compensation function, including: If the fingertip distance perceived by the human hand is less than the minimum error threshold, the compensated target distance is mapped to zero. When the fingertip distance perceived by the human hand is between the minimum error threshold and the fine operation threshold, the compensated target distance is calculated using a smooth nonlinear mapping function; If the fingertip distance perceived by the human hand is greater than the fine operation threshold, the compensated target distance is set to be equal to the fingertip distance perceived by the human hand.

[0012] Optionally, the plurality of optimization sub-objectives further includes a fourth sub-objective, used to guide or anchor the mapping results of a specific hand configuration during the optimization process; the fourth sub-objective is constructed according to the following steps: Pre-set at least one special gesture, and save the corresponding human hand key point configuration data for each special gesture as the preset human hand configuration, and save the expected robot joint angle data as the preset robot configuration; Calculate the current output robot joint angle corresponding to the preset human hand configuration based on the mapping function; Calculate the configuration deviation between the current output robot joint angle and the preset robot configuration; A confidence weight is assigned to each special gesture, and the confidence weight is used to adjust the constraint strength of the special gesture on the mapping function; The fourth sub-target item is obtained by summing or averaging the product of the configuration deviation and the confidence weight corresponding to each special gesture.

[0013] Optionally, the plurality of optimization sub-objectives further includes a fifth sub-objective, used to constrain the robot joint angles obtained by the mapping to satisfy joint constraint conditions and motion smoothness conditions; the fifth sub-objective is constructed according to the following steps: Based on the mapping function, obtain the current output joint angle of each joint of the robot; For each joint, calculate the joint limit exceedance based on whether the current output joint angle exceeds the preset joint physical limit; Based on the mapping function outputs of the current and historical moments, the second derivative of the mapping function with respect to time is calculated as a temporal variation feature of the joint motion, which is used to characterize the smoothness of the joint motion. The fifth sub-target item is obtained by weighted summation of the joint limit excess and the temporal change characteristic.

[0014] Optionally, the mapping function is a deep neural network; using the comprehensive weighted optimization objective function as the optimization criterion, the mapping function is optimized based on the three-dimensional spatial position data of the key points of the hand, including: Using the three-dimensional spatial position data of the key points of the human hand as training samples, the comprehensive weighted optimization objective function as the loss function, and the robot's differentiable forward kinematics model, the deep neural network is trained through the error backpropagation algorithm to obtain the trained deep neural network. The real-time acquired 3D spatial position data of the key points of the human hand are input into the optimized mapping function to obtain the target joint angles of the robot, including: The three-dimensional spatial position data of the key points of the human hand acquired in real time are input into the trained deep neural network, and the target joint angle of the robot is obtained through a forward calculation.

[0015] A second aspect of this application discloses a dexterous arm heterogeneous teleoperation motion mapping system for implementing the dexterous arm heterogeneous teleoperation motion mapping method described in the first aspect of this application. The system includes: The data acquisition module is used to collect three-dimensional spatial position data of key points on the human hand; The storage module stores a mapping function constructed and optimized by the dexterous arm heterogeneous teleoperation motion mapping method according to the first aspect of the embodiments of this application; The control module is used to input the three-dimensional spatial position data of the key points of the human hand acquired in real time by the data acquisition module into the mapping function stored in the storage module to obtain the target joint angle of the robot and drive the robot to perform corresponding actions.

[0016] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the dexterous arm heterogeneous teleoperation motion mapping method described in the first aspect of this application.

[0017] A fourth aspect of this application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the dexterous arm heterogeneous teleoperation motion mapping method described in the first aspect of this application.

[0018] A fifth aspect of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the dexterous arm-hand heterogeneous teleoperation motion mapping method described in the first aspect of this application.

[0019] The embodiments of this application have the following advantages: First, by constructing a comprehensive weighted optimization objective function containing multiple optimization sub-objectives, and then optimizing the mapping function based on this objective function, the quality of motion mapping is comprehensively constrained from multiple dimensions. Compared with the single-dimensional optimization methods in related technologies, this method can simultaneously take into account multiple key performance indicators such as fingertip positioning accuracy, motion morphology naturalness, and perception error compensation, thus solving the problem of difficult collaborative optimization of multiple objectives in teleoperation.

[0020] Secondly, the constructed comprehensive weighted optimization objective function includes at least a first sub-objective term for constraining fingertip positioning accuracy, which focuses on constraining key parts performing fine operations to ensure that the mapped robot fingertip pose accurately responds to the operator's intentions; at least a second sub-objective term for constraining the naturalness of motion, which makes the mapped robot movements closer to the operator's hand shape; and at least a third sub-objective term for compensating for human hand perception errors, which actively compensates for the impact of sensing device observation errors on the mapping results, achieving accurate physical contact even in the presence of gaps in perception. Through the synergistic effect of the above sub-objective terms, this method achieves significant improvements in fingertip accuracy, motion naturalness, and system robustness.

[0021] Finally, the comprehensive weighted optimization objective function is used as the optimization criterion to optimize the mapping function. After obtaining the optimized mapping function, in real-time teleoperation, only the key points of the human hand collected in real time need to be input into this mapping function to obtain the robot joint angles. Since the optimization process is completed in advance, no complex iterative optimization calculations are required during online execution, which significantly reduces the online computational overhead, realizes low-latency real-time control, and meets the high-frequency response speed requirements of high-quality teleoperation. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the steps of a dexterous arm heterogeneous teleoperation motion mapping method provided in an embodiment of this application; Figure 2 This is an overall architecture diagram of a dexterous arm heterogeneous teleoperation motion mapping method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a dexterous arm heterogeneous teleoperation motion mapping system provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] This application provides a method and system for mapping heterogeneous teleoperation motions in a dexterous arm. The technical concept is to construct the motion mapping problem in teleoperation as a multi-objective weighted optimization problem. By designing a comprehensive weighted optimization objective function containing multiple optimization sub-objectives, the mapping quality is comprehensively constrained from multiple dimensions such as fingertip positioning accuracy, naturalness of motion form, and compensation for human hand perception errors. Based on this objective function, the mapping function is optimized so that the optimized mapping function can simultaneously consider the above-mentioned multiple performance indicators under heterogeneous structures. Furthermore, the optimization process is completed in advance; during online teleoperation, only the forward calculation of the mapping function needs to be performed to achieve low-latency real-time control. The technical solution of this application is described in detail below through specific embodiments.

[0026] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of a heterogeneous teleoperation motion mapping method for a dexterous arm provided in an embodiment of this application. Figure 1 As shown, the method for mapping heterogeneous teleoperation movements of a dexterous arm provided in this application embodiment may include steps S110 to S150: Step S110: Collect three-dimensional spatial position data of key points of the human hand.

[0027] In this step, three-dimensional spatial position data of key points on the human hand are collected using motion sensing devices. These devices may include, but are not limited to, visual sensors (such as depth cameras), data gloves, and VR controllers. The collected key points include at least the three-dimensional coordinate information of the fingertips, knuckles, and wrist, which constitute the input vector describing the hand's posture. To facilitate subsequent processing, the collected three-dimensional spatial position data of the key points can be preprocessed, such as filtered and interpolated, to eliminate noise and missing values.

[0028] Step S120: Construct a mapping function from the three-dimensional spatial position data of the key points of the human hand to the joint angles of the robot; wherein the robot includes a robotic arm and a dexterous hand.

[0029] In this step, a mapping function is constructed to convert the key points of the human hand collected in step S110 into robot joint angles. The robot consists of a robotic arm and a dexterous hand, and its joint angle vectors encompass the angle values ​​of each joint of the robotic arm and each finger joint of the dexterous hand. This mapping function can be a function model to be optimized or a deep neural network.

[0030] Step S130: Construct a comprehensive weighted optimization objective function, which is a weighted sum of multiple optimization sub-objectives. The multiple optimization sub-objectives include at least: a first sub-objective for constraining the positioning accuracy of the robot's fingertips, a second sub-objective for constraining the naturalness of the robot's motion shape, and a third sub-objective for compensating for human hand perception errors.

[0031] In this step, a comprehensive weighted optimization objective function is constructed as the optimization criterion for subsequent optimization of the mapping function. This comprehensive weighted optimization objective function is formed by the weighted sum of multiple optimization sub-objectives, each of which constrains different dimensions of the mapping quality.

[0032] Specifically, the first sub-objective (i.e., the global pose objective) constrains the positioning accuracy of the robot's fingertips, ensuring that key parts performing fine operations (such as the thumb tip) accurately reproduce the operator's intentions. The second sub-objective (i.e., the humanoid form preservation objective) constrains the naturalness of the robot's movements, making the mapped robot movements visually and kinematically closer to the natural form of a human hand. The third sub-objective (i.e., the fingertip relative position constraint objective) compensates for observation errors in human hand sensing devices, ensuring accurate physical contact can still be achieved even when perception deviations exist.

[0033] It should be noted that the above three sub-objectives form the basis for constructing the comprehensive weighted optimization objective function. To achieve better mapping results, other sub-objectives can be introduced on this basis, such as guiding terms to enhance the accuracy of key gestures and constraint terms to ensure the safety of joint movements.

[0034] Step S140: Using the comprehensive weighted optimization objective function as the optimization criterion, optimize the mapping function based on the three-dimensional spatial position data of the key points of the human hand to obtain the optimized mapping function.

[0035] In this step, the comprehensive weighted optimization objective function constructed in step S130 is used as the optimization criterion. The mapping function constructed in step S120 is optimized using the collected three-dimensional spatial position data of the key hand points. The goal of the optimization is to achieve the optimal solution (i.e., minimum value) of the comprehensive weighted optimization objective function given the key hand points. Through optimization, the parameters of the mapping function are adjusted to the best state that simultaneously considers fingertip precision, movement naturalness, and perceptual error compensation.

[0036] Understandably, this optimization process can be carried out offline, using pre-collected large-scale 3D spatial location data of key points on the human hand to fully train the mapping function, so that the optimized mapping function can be applied to various operating scenarios.

[0037] Step S150: In real-time teleoperation, the three-dimensional spatial position data of the key points of the human hand collected in real time is input into the optimized mapping function to obtain the target joint angle of the robot, so as to drive the robot to perform the corresponding action.

[0038] In this step, the mapping function optimized in step S140 is applied to the real-time teleoperation scenario. Specifically, the three-dimensional spatial position data of the operator's hand key points are collected in real time and input into the optimized mapping function. The corresponding target joint angle of the robot can be obtained through a single calculation. Subsequently, the target joint angle is sent to the robot controller to drive the robot's robotic arm and dexterous hand to perform actions corresponding to the operator's hand movements.

[0039] Since the optimization process has been completed offline in step S140, the real-time teleoperation stage does not require complex iterative optimization calculations. Only the forward calculation of the mapping function needs to be performed to obtain the control command, thereby reducing the online calculation delay and realizing high-frequency real-time control.

[0040] The technical solution of this embodiment constructs a comprehensive weighted optimization objective function containing multiple optimization sub-objectives, and optimizes the mapping function based on this objective function, thereby achieving comprehensive constraint on motion mapping quality from multiple dimensions. Compared with the single-dimensional optimization methods in related technologies, this method can simultaneously consider multiple key performance indicators such as fingertip positioning accuracy, motion morphology naturalness, and perception error compensation, solving the problem of difficult collaborative optimization of multiple objectives in teleoperation. Furthermore, by completing the optimization process in advance, the real-time teleoperation stage only needs to perform the forward calculation of the mapping function, reducing online computational overhead and achieving low-latency real-time control, meeting the high-frequency response speed requirements of high-quality teleoperation.

[0041] In an optional embodiment, the specific construction method of the aforementioned first sub-objective item is further described in detail. The first sub-objective item is used to constrain the positioning accuracy of the robot's fingertip and is an important component of the comprehensive weighted optimization objective function. Referring to step S130 above, the first sub-objective item constructed in this embodiment is constructed according to the following steps A1 to A4: Step A1: Determine the position of the fingertip of the target finger of the human hand as the global position reference, and determine the desired wrist orientation of the robot.

[0042] In this step, considering that the wrist does not directly contact the object during actual operation, and that the fingertips, which perform the finer operations, may lose necessary positioning accuracy due to the heterogeneity between the human hand and the robot, requiring strict wrist matching, a global position reference is established by defining the target fingertip position as the global position reference. This target fingertip position can be the tip of any finger. Simultaneously, based on the requirements of the operation task and the operator's posture, the desired orientation of the robot's wrist is determined. This desired orientation can be based on the human hand's wrist orientation or set according to task requirements.

[0043] Step A2: Based on the mapping function, determine the position of the robot's target fingertip corresponding to the position of the human hand's target fingertip, as well as the robot's actual wrist orientation.

[0044] In this step, the collected three-dimensional spatial position data of key points of the human hand are mapped to the robot's joint angles using the current mapping function. Then, the actual spatial position of the robot's target fingertip and the robot's actual wrist orientation are calculated using the robot's forward kinematics model. The forward kinematics model is used to calculate the spatial pose of each link and end effector of the robot based on the joint angles.

[0045] Step A3: Calculate the positional deviation between the target fingertip position of the human hand and the target fingertip position of the robot, and calculate the orientation deviation between the robot's desired wrist orientation and the robot's actual wrist orientation.

[0046] In this step, the positional deviation reflects the positioning error between the target fingertip position of the human hand and the corresponding fingertip position of the robot, and is usually calculated using Euclidean distance, which is the straight-line distance between two points in three-dimensional space. The orientation deviation reflects the angular deviation between the robot's desired wrist orientation and the actual wrist orientation, and is usually measured using angular deviation.

[0047] Step A4: The position deviation and the orientation deviation are weighted and summed to obtain the first sub-target item; wherein the weight of the position deviation is greater than the weight of the orientation deviation, so as to allow the robot wrist to locally adapt and adjust while prioritizing the positioning accuracy of the target fingertip position.

[0048] In this step, the two aforementioned deviations are weighted and summed to obtain the first sub-target item. The key is to set the weight of the position deviation greater than that of the orientation deviation. This weighting prioritizes reducing positional deviations during the optimization process, ensuring that the robot's target fingertip accurately replicates the position of the human hand's target finger. Correspondingly, the wrist orientation deviation has a smaller weight, allowing the wrist to adaptively adjust within a local range to achieve higher fingertip accuracy.

[0049] In a preferred embodiment, the target fingertip position of the human hand is the fingertip position of the human thumb; the position deviation is the deviation between the fingertip position of the human thumb and the fingertip position of the robot thumb; and the orientation deviation is the angular deviation between the desired wrist orientation of the robot and the actual wrist orientation of the robot.

[0050] In this embodiment, the thumb tip is chosen as the global position reference because the thumb plays a crucial role in human hand movements, performing key functions such as palm opposition and pinching, and is the core execution part for fine manipulation. Using the thumb tip as the reference ensures the most accurate reproduction of the intended operation on the robot. Simultaneously, wrist orientation deviation is measured using angular deviation, which directly reflects differences in wrist posture, facilitating gradient calculation and parameter adjustment during optimization.

[0051] For example, the first sub-target item constructed in this way It can be represented as:

[0052] In the formula, the first term is the positional deviation, and the second term is the orientation deviation. The location of the thumb tip was collected. The position of the robot's thumb tip is calculated using a mapping function and forward kinematics. For the robot's desired wrist orientation, The actual wrist orientation of the robot is calculated using a mapping function and forward kinematics. This is the weighting coefficient for the orientation deviation term, and its value is less than the coefficient for the position deviation term (i.e., the weight of the position deviation term is 1 by default). (A positive number less than 1).

[0053] As can be seen from the formula above, the weight of the positional deviation term is much greater than that of the orientation deviation term. This means that during the optimization process, priority is given to ensuring the positional accuracy of the thumb tip. Once the positional deviation is satisfied, the wrist orientation is then aligned with the desired orientation as much as possible. This design breaks through the limitation of traditional teleoperation, which prioritizes strict wrist alignment, and fully considers the practical need for fingertip accuracy to be more critical than wrist posture in delicate operation tasks.

[0054] The technical solution of this embodiment constructs a first sub-target item with the fingertip (preferably the thumb tip) as its core, and sets the position deviation weight to be greater than the orientation deviation weight. This breaks through the traditional wrist-centered alignment method in teleoperation, shifting the optimization focus from the wrist to the key part (i.e., the fingertip) for performing fine operations, making the mapped robot movements more in line with the essential requirements of the operation task. Secondly, by allowing local adaptive adjustment of the wrist, the loss of fingertip accuracy caused by the heterogeneity of the mechanical structure is avoided.

[0055] In an optional embodiment, the specific construction method of the aforementioned second sub-objective item is further described in detail. The second sub-objective item is used to constrain the naturalness of the robot's motion form, enabling the robot to maintain local motion characteristics consistent with those of a human hand when performing actions. It is an important component of the comprehensive weighted optimization objective function. Referring to step S130 above, the second sub-objective item constructed in this embodiment is constructed according to the following steps B1 to B4: Step B1: Based on the three-dimensional spatial position data of the key points of the human hand, determine the current relative position of the fingertips relative to the palm of the human hand, and generate at least one random small relative displacement vector at the current relative position, wherein the small relative displacement vector represents the local movement trend direction of the fingertips relative to the palm of the human hand.

[0056] In this step, based on the collected 3D spatial position data of key points on the hand, the current relative position of each fingertip relative to the palm is calculated. This current relative position describes the spatial relationship between the fingertip and the palm in a static state. Subsequently, at the current relative position of each finger, one or more random, minute relative displacement vectors are generated. These minute displacement vectors represent a virtual local motion trend, assuming that the fingertip makes a small movement in a certain direction based on the current pose. This direction of movement reflects the local motion pattern of the hand when performing an action. Generating multiple minute displacement vectors through random sampling can cover all possible motion directions of the hand in local space, thereby enhancing the generalization ability of the constraints.

[0057] Step B2: Based on the mapping function and the robot's forward kinematics model, calculate the predicted relative displacement vector of the robot's fingertip relative to the robot's palm, corresponding to the tiny relative displacement vector.

[0058] In this step, the robot's response displacement is calculated using a mapping function and the robot's forward kinematics model when the human fingertip generates the small relative displacement vector described in step B1. This process involves the differential properties of the composite mapping function. By using the Jacobian matrix of the mapping function and the robot's forward kinematics model, the mapping relationship from the small displacement input by the human hand to the relative displacement of the robot's fingertip can be calculated.

[0059] For example, let the mapping function be... , where x is the 3D spatial position data input of the human hand keypoints, and q is the robot joint angle output. The current relative position of the robot's fingertip with respect to the robot's palm is . ,in The position of the robot's fingertip is calculated using a mapping function and forward kinematics. The position of the robot's palm is calculated using a mapping function and forward kinematics. This represents the predicted change in the relative position of the robot's fingertips after a small displacement *d* between the human fingertips. It can be calculated using the Jacobian matrix of the composite mapping function, i.e.: Then the predicted relative displacement vector is the difference between the relative position of the robot's fingertip after a small displacement and the current relative position.

[0060] Step B3: Normalize the small relative displacement vector and the predicted relative displacement vector into unit direction vectors, and calculate the difference between the two normalized unit direction vectors as a difference measure.

[0061] In this step, the minute relative displacement vector generated in step B1 and the predicted relative displacement vector calculated in step B2 are normalized to unit direction vectors. Then, the difference between these two unit direction vectors is calculated as a measure of the finger's directional consistency under that minute displacement. This difference can be measured using Euclidean distance, cosine distance, or other distance metrics.

[0062] This difference metric reflects whether a robot's fingertip can produce a similar geometrically topological direction of movement when a human fingertip moves in a certain direction. The smaller the difference, the more consistent the robot's fingertip's movement direction is with that of a human fingertip, and the more natural the movement.

[0063] Step B4: Accumulate or average the generated minute relative displacement vectors and the difference measures corresponding to each finger to obtain the second sub-target item.

[0064] In this step, the difference metric calculated in step B3 is accumulated or averaged for all generated one or more small relative displacement vectors and all fingers to obtain the expression for the second sub-objective. For example, suppose there are N fingers involved in the constraint, and for the i-th finger, M random small relative displacement vectors are generated, then the expression for the second sub-objective is... It can be represented as:

[0065] in, This represents the expectation of a randomly generated infinitesimal relative displacement vector d; The position of the robot's fingertip is calculated using a mapping function and forward kinematics. Let be the position of the robot's palm calculated using the mapping function and forward kinematics. The formula implies that for every possible local motion direction d of the human hand, the relative motion direction of the robot's fingertips should be aligned with it as much as possible, thus constraining the Jacobian matrix of the mapping function to maintain directional consistency within the local space.

[0066] The technical solution of this embodiment ensures that the local motion directions of the human hand and the robot are geometrically and topologically consistent, enabling the robot to exhibit motion characteristics similar to those of a human hand when performing continuous actions, thus improving the naturalness of robot movements during teleoperation. Secondly, by employing a tangent space constraint method based on the Jacobian matrix, which directly acts on the differential space of the mapping function, this method better handles the geometric heterogeneity between the human hand and the robot compared to traditional position-level constraints, giving the mapping function better smoothness and generalization ability across the entire space. Thirdly, by randomly sampling multiple small displacement vectors, this embodiment can comprehensively constrain the behavior of the mapping function in its local neighborhood, avoiding the problem of distortion in other directions that may result from constraining only a single direction, thus enhancing the completeness and robustness of the constraints.

[0067] In an optional embodiment, the specific construction method of the aforementioned third sub-objective item is described in detail. The third sub-objective item is used to compensate for human hand perception errors, ensuring accurate physical contact of the robot's fingertips even in the presence of observation errors, and is an important component of the comprehensive weighted optimization objective function. Referring to step S130 above, the third sub-objective item constructed in this embodiment is constructed according to the following steps C1 to C7: Step C1: Based on the three-dimensional spatial position data of the key points of the human hand, determine the relative position vector between the tip of the thumb and the tips of the other fingers, and calculate the magnitude of each relative position vector as the fingertip distance perceived by the human hand.

[0068] In this step, based on the collected three-dimensional spatial position data of key points on the hand, the relative position vector between the tip of the thumb and the tips of the other fingers (such as the index, middle, ring, and little fingers) is calculated. Let the position of the thumb tip be... The position of the tip of the i-th finger is Then the relative position vector can be expressed as Then, the magnitude of each relative position vector is calculated as the fingertip distance perceived by the human hand, i.e.: This distance reflects the degree of opening and closing between the operator's fingers during teleoperation. In actual operation, due to observation errors in sensing devices (such as vision sensors and data gloves), the perceived fingertip distance may vary even when the operator's fingertips are actually in contact. It may still be displayed as a non-zero positive number, indicating the presence of gap error.

[0069] Step C2: Based on the fingertip distance perceived by the human hand, calculate the compensated target distance using a preset distance compensation function; wherein, the distance compensation function is used to map the fingertip distance perceived by the human hand from the original interval containing non-zero perception error to the target interval containing zero contact point.

[0070] In this step, a distance compensation function is introduced. The perceived fingertip distance is remapped. This compensation function aims to address the gap error problem in sensing devices: when an operator intends for their fingertip to touch (i.e., the actual distance is zero), the detected distance may still be non-zero due to sensing errors, preventing the robot from achieving precise closure. The distance compensation function maps the fingertip distance perceived by the human hand from the original sensing range to the target range, ensuring that when the perceived distance is less than a certain threshold, the compensated target distance is zero, thus driving the robot to achieve precise finger alignment.

[0071] Specifically, based on the fingertip distance perceived by the human hand, the compensated target distance is calculated using a preset distance compensation function, including steps C2-1 to C2-3: Step C2-1: If the fingertip distance perceived by the human hand is less than the minimum error threshold, the compensated target distance is mapped to zero.

[0072] Specifically, when the distance between the fingertips perceived by the human hand Less than the minimum error threshold At that time, the operator's intention was determined to be fingertip contact (i.e., the actual distance was zero), while the currently perceived positive value was due to equipment observation error. Therefore, the compensated target distance... The mapping is zero, that is:

[0073] This ensures that the robot can achieve precise finger-to-finger closure when the operator's finger is actually in contact with or very close to the contact.

[0074] Step C2-2: When the fingertip distance perceived by the human hand is between the minimum error threshold and the fine operation threshold, the compensated target distance is calculated using a smooth nonlinear mapping function.

[0075] Specifically, when the distance between the fingertips perceived by the human hand Between the minimum error threshold With fine operation threshold During this process, a smooth, nonlinear mapping function is used to calculate the compensated target distance. This nonlinear mapping function can take various forms. In some feasible implementations, a mapping function containing a second-order smoothing term is used, i.e.:

[0076] in, This is a smoothing coefficient used to control the smoothness of the mapping curve at the threshold critical point. This results in better dynamic characteristics of the mapped trajectory at the threshold critical point, avoiding robot motion jitter caused by abrupt mapping changes.

[0077] Step C2-3: If the fingertip distance perceived by the human hand is greater than the fine operation threshold, the compensated target distance is set to be equal to the fingertip distance perceived by the human hand.

[0078] Specifically, when the distance between the fingertips perceived by the human hand Greater than the fine operation threshold At that time, it is assumed that the operator's fingers are in an open state, indicating no intention to make contact, and the impact of perception error is relatively small. Therefore, the compensated target distance is directly equal to the perceived distance, i.e.:

[0079] Step C3: Based on the direction of the relative position vector and the compensated target distance, obtain the remapped desired relative position vector.

[0080] In this step, the compensated target distance calculated in step C2 is combined with the direction unit vector of the relative position vector in step C1 to obtain the remapped desired relative position vector. Let the direction unit vector of the relative position vector be... (when (At this time, the direction vector can be determined based on historical data or the default direction), and the compensated target distance is... Then the expected relative position vector after remapping is This expected relative position vector represents the expected relative spatial relationship between the robot's thumb and its corresponding finger after compensating for perception errors.

[0081] Step C4: Based on the current mapping function and the robot's forward kinematics model, calculate the actual relative position vectors of the robot side corresponding to the tip of the thumb and the tips of the other fingers.

[0082] In this step, the robot's actual relative position vector reflects the relative spatial relationship between the fingers during actual robot execution under the current mapping function. Specifically, let's assume the robot's thumb tip... The position of the tip of the i-th finger is Then the robot's actual relative position vector for: .

[0083] Step C5: Calculate the deviation between the desired relative position vector after remapping and the actual relative position vector on the robot side.

[0084] For example, the deviation between the remapped desired relative position vector and the actual relative position vector on the robot side can be expressed as: This deviation reflects the difference between the expected relative position of the robot's fingertips and the actual relative position of the fingertips after compensating for perception errors. During optimization, this deviation will be minimized as much as possible so that the robot's fingertips can perform actions according to the compensated target position.

[0085] Step C6: Determine the weighting coefficient based on the fingertip distance perceived by the human hand. The weighting coefficient takes a larger value when the fingertip distance is small.

[0086] In this step, a weighting function based on perceived distance is introduced to adjust the strength of the constraint at different distances. This weight is larger when the fingertip distance is small and smaller when the fingertip distance is large. The core of the weighting coefficient design is that when the fingertip distance is small, the operator may be performing a fine pinching or contact operation; while when the fingertip distance is large, the operator may be performing a large-amplitude opening motion, where the requirement for relative fingertip position accuracy is relatively low, and a smaller weight can be assigned.

[0087] Step C7: Accumulate or average the product of the deviation amount corresponding to each finger and the weight coefficient to obtain the third sub-target item.

[0088] For example, the third sub-target item It can be represented as:

[0089] Where N is the total number of fingers involved in the constraint (including the thumb), and i represents the number of fingers traversed except for the thumb. These are the weighting coefficients. For distance compensation function, The direction unit vector of the relative position vector. This is the actual relative position vector on the robot side.

[0090] The technical solution adopted in this embodiment effectively solves the problem of "contact present but no contact detected" caused by the observation error of the sensing device by introducing a distance compensation function. Firstly, a piecewise nonlinear mapping function is used to introduce a smooth transition between the minimum error threshold and the fine operation threshold, avoiding robot motion jitter caused by abrupt mapping changes and ensuring the smoothness and stability of the contact process. Secondly, by introducing a weighting function based on the sensing distance, the constraint strength is enhanced during the fine operation stage and weakened during the large-amplitude movement stage, achieving adaptive adjustment for different operation scenarios and improving the system's flexibility and robustness.

[0091] In an optional embodiment, the plurality of optimization sub-objectives further includes a fourth sub-objective (i.e., a predefined special case gesture guidance item), used to guide or anchor the mapping results of specific human hand configurations during the optimization process. This embodiment provides a detailed description of the specific construction method of the fourth sub-objective. This fourth sub-objective, by introducing expert knowledge or calibration data, enhances the accuracy of the mapping function on key gestures, enabling the robot to present the expected form when performing gestures with strong semantic features, and is an important component of the comprehensive weighted optimization objective function.

[0092] Referring to the aforementioned step S130, the fourth sub-target item constructed in this embodiment is constructed according to the following steps D1 to D5: Step D1: Preset at least one special gesture, and save the corresponding human hand key point configuration data for each special gesture as the preset human hand configuration, and save the expected robot joint angle data as the preset robot configuration.

[0093] In this step, one or more key gestures with strong semantic features are pre-defined as special gestures based on actual application requirements. Examples include "five fingers spread," "standard fist," "two-finger pinch," and "three-finger grasp." For each special gesture, the corresponding preset human hand configuration and preset robot configuration are saved. These preset special gestures constitute an expert knowledge base, used to guide the mapping function to exhibit high accuracy when regressing to these key poses.

[0094] Among them, the pre-set human hand configuration This represents the key point configuration data of the operator's hand under this gesture, which is usually collected by motion capture equipment or obtained through manual setting. Preset robot configuration. This indicates that the joint angle configuration expected to be presented by the robot's dexterous hand under this gesture can be obtained through methods such as manual teaching, kinematic solving, or manual setting.

[0095] Step D2: Calculate the current output robot joint angle corresponding to the preset human hand configuration based on the mapping function.

[0096] In this step, the current mapping function is used. For the preset hand configuration saved in step D1 By mapping, the corresponding current output robot joint angle is obtained, i.e. Because the mapping function is continuously updated during the optimization process, the current output joint angle will also change with each optimization iteration.

[0097] Step D3: Calculate the configuration deviation between the current output robot joint angle and the preset robot configuration.

[0098] In this step, Euclidean distance, weighted Euclidean distance, or other distance metrics can be used to calculate the configuration deviation between the current output robot joint angle obtained in step D2 and the preset robot configuration saved in step D1. This configuration deviation reflects the difference between the mapping result of the special gesture and the expected result under the current mapping function; the smaller the deviation, the more accurate the mapping function is in performing that gesture.

[0099] Step D4: Set a confidence weight for each special gesture, the confidence weight being used to adjust the constraint strength of the special gesture on the mapping function.

[0100] In this step, the confidence weight can be set according to the importance, reliability, or application scenario of the special gesture. For example, for gestures that are frequently used in teleoperation tasks or are crucial to operational safety (such as "clenching a fist"), a higher confidence weight can be set to make them more anchored to the mapping function; while for minor gestures, a lower confidence weight can be set to avoid excessive constraints affecting the mapping quality of other gestures.

[0101] Step D5: Accumulate or average the product of the configuration deviation and the confidence weight corresponding to each special gesture to obtain the fourth sub-target item.

[0102] In this step, the products of the configurational deviations and confidence weights corresponding to all special gestures are summed or averaged to obtain the expression for the fourth sub-objective, namely:

[0103] Where K is the total number of special gestures, Let be the confidence weight of the k-th special case gesture.

[0104] The technical solution of this embodiment incorporates expert knowledge or calibration data into the optimization process as constraints, thereby explicitly guiding the mapping function and enabling it to exhibit extremely high accuracy and morphological naturalness on key gestures with strong semantic features. Secondly, by setting independent confidence weights for each specific gesture, the constraint strength of different gestures can be flexibly adjusted according to actual application needs, ensuring the accuracy of key gestures while maintaining mapping flexibility. Thirdly, the fourth sub-objective item in this embodiment works synergistically with the aforementioned sub-objective items, ensuring both the flexibility and naturalness of the mapping function in continuous motion and its accuracy when reverting to the key pose.

[0105] In an optional embodiment, the plurality of optimization sub-objectives further include a fifth sub-objective (i.e., a joint space motion constraint term), used to constrain the mapped robot joint angles to meet joint limitation conditions and motion smoothness conditions. This embodiment provides a detailed description of the specific construction method of the fifth sub-objective. This sub-objective, by introducing physical constraints and motion smoothness constraints in the joint space, ensures the physical feasibility and smoothness of the mapped robot motion, and is an important component of the comprehensive weighted optimization objective function.

[0106] Referring to the aforementioned step S130, the fifth sub-target item constructed in this embodiment is constructed according to the following steps E1 to E4: Step E1: Obtain the current output joint angle of each joint of the robot according to the mapping function.

[0107] In this step, the current mapping function to be optimized is used to process the 3D position data of the input hand key points. By mapping, we obtain the current output joint angles of each joint of the robot, i.e.:

[0108] Where M is the total number of degrees of freedom of the robot, including the joints of the robotic arm and the finger joints of the dexterous hand; These are the current output joint angles for the 1st to Mth joints, respectively. This joint angle vector directly determines the action posture that the robot will perform.

[0109] Step E2: For each joint, calculate the joint limit exceedance based on whether the current output joint angle exceeds the preset joint physical limit.

[0110] In this step, the physical range of motion for each joint is constrained. Each robot joint has its inherent physical rotation range, namely the lower limit and the upper limit. When the joint angle output by the mapping function exceeds this range, the actual robot will be unable to execute the angle command, and forced execution may lead to hardware damage or unstable control. For joints exceeding the limit, the excess amount is calculated as the limit excess amount; for joints within the limit, the limit excess amount is set to zero.

[0111] Step E3: Based on the mapping function outputs of the current time and historical time, calculate the second derivative of the mapping function with respect to time, which is used as a temporal variation feature of the joint motion. The temporal variation feature is used to characterize the smoothness of the joint motion.

[0112] In this step, the rate of change of the mapping function output in the time dimension is constrained to prevent drastic changes in joint position during continuous motion, ensuring the smoothness and compliance of the robot's movements. Specifically, the second derivative of the mapping function with respect to time, i.e., the joint acceleration, is calculated as a temporal characteristic of joint motion.

[0113] Step E4: Perform a weighted summation of the joint limit excess and the temporal change characteristic to obtain the fifth sub-target item.

[0114] In this step, the joint limit exceedance amount obtained in step E2 and the temporal change characteristic amount obtained in step E3 are weighted and summed to obtain the fifth sub-objective item. For example, the fifth sub-objective item... It can be represented as:

[0115] in, This is the weighting coefficient for joint limiting constraints, used to adjust the strength of the joint limiting constraints; These are the weighting coefficients for motion smoothing constraints, used to adjust the strength of the smoothness constraints; The upper limit of the joint, The lower limit of the joint; This is the current output joint angle of the k-th joint.

[0116] The technical solution adopted in this embodiment introduces joint limit constraint components to explicitly constrain the joint angles output by the mapping function, ensuring that the command angles of each joint of the robot are always within the physically feasible range, effectively avoiding hardware damage or control failure caused by commands exceeding the limits. Secondly, by introducing motion smoothing constraint components, the acceleration of joint motion is constrained, suppressing instantaneous jumps in joint position during continuous motion, making the robot's motion trajectory more continuous, smooth, and compliant. Thirdly, by weighted summing the two, the strength of different constraints can be flexibly adjusted according to the characteristics of the actual robot hardware, achieving a balance between constraint strength and motion performance.

[0117] In one optional embodiment, the mapping function is a deep neural network. This embodiment further specifies how to utilize a deep neural network to optimize the mapping function and perform real-time inference, thereby addressing the problem of high online computational overhead in related technologies.

[0118] In step S140 above, "using the comprehensive weighted optimization objective function as the optimization criterion, the mapping function is optimized according to the three-dimensional spatial position data of the key points of the human hand" specifically includes: using the three-dimensional spatial position data of the key points of the human hand as training samples, using the comprehensive weighted optimization objective function as the loss function, using the robot's differentiable forward kinematics model, and training the deep neural network through the error backpropagation algorithm to obtain the trained deep neural network.

[0119] In this step, the mapping function is implemented as a deep neural network. The input to this deep neural network is the three-dimensional spatial position data of the hand key points, and the output is the robot joint angles. The parameters of the deep neural network are the parameters of the mapping function to be optimized.

[0120] During training, a large amount of three-dimensional spatial location data of key points on the human hand is acquired as training samples, and the comprehensive weighted optimization objective function constructed in step S130 above is used as the loss function of the neural network. ,Right now To achieve end-to-end training, a forward kinematics model for the robot is introduced. This model maps the joint angles output by the neural network to the spatial positions of the robot's key points. By constructing the forward kinematics model in a differentiable mathematical form, the loss function can be backpropagated to the various parameters of the neural network using the chain rule. The training process is repeated until the loss function converges, resulting in a trained deep neural network.

[0121] Through the above training process, the deep neural network learns the optimal mapping relationship from the three-dimensional spatial position data of the key points of the human hand to the joint angle of the robot. This mapping relationship fully reflects the multi-dimensional performance requirements such as fingertip accuracy, movement naturalness and perception error compensation constrained by the comprehensive weighted optimization objective function.

[0122] Furthermore, in step S150 above, "inputting the real-time acquired three-dimensional spatial position data of the key points of the human hand into the optimized mapping function to obtain the target joint angle of the robot" specifically includes: inputting the real-time acquired three-dimensional spatial position data of the key points of the human hand into the trained deep neural network, and obtaining the target joint angle of the robot through a forward calculation.

[0123] In this step, the trained deep neural network is applied to a real-time teleoperation scenario. During real-time teleoperation, the 3D spatial position data of the key points of the human hand, collected in real time by the motion sensing device, is directly input into the trained deep neural network. The network performs a single forward propagation calculation to output the corresponding robot target joint angle. Since the forward calculation of the deep neural network only involves basic operations such as matrix multiplication and activation functions, the computational load is far less than that of solving high-dimensional nonlinear optimization problems in traditional methods. Therefore, it can complete single-frame mapping in milliseconds, meeting the requirements of high-frequency real-time control.

[0124] The technical solution adopted in this embodiment utilizes a differentiable forward kinematics model to directly use the comprehensive weighted optimization objective function as the loss function of the neural network, achieving end-to-end training. This enables the neural network to autonomously learn the optimal mapping relationship from human hand keypoints to robot joint angles, eliminating the need for manually designing complex mapping rules. Secondly, by moving the computational burden of high-dimensional nonlinear optimization to the offline training stage, only one forward computation of the neural network is required to obtain the optimal joint angles during real-time teleoperation, significantly reducing online computational overhead. Thirdly, since the parameters are fixed after neural network training, the online inference process has deterministic computation time and stable control frequency, avoiding the problem of uncertain solution time in traditional optimization methods and improving the real-time performance and stability of the teleoperation system.

[0125] like Figure 2 As shown, Figure 2 This is an overall architecture diagram of a heterogeneous teleoperation motion mapping method for a dexterous arm provided in an embodiment of this application. This embodiment adopts an "offline training, online inference" architecture, mainly including two parts: an offline optimization stage and an online teleoperation stage.

[0126] During the offline optimization phase, a large amount of 3D spatial position data of key points on the human hand is collected to form a training sample set. The training samples are then input into a constructed deep neural network. This deep neural network is used to convert the key points of the human hand into robot joint angles.

[0127] During training, a comprehensive weighted optimization objective function is constructed as the loss function of the neural network. This comprehensive weighted optimization objective function is a weighted sum of multiple optimization sub-objectives, including at least the following: First sub-objective (global pose objective): used to constrain the robot's fingertip positioning accuracy, using the position of the human thumb as the global position reference, allowing for local adaptive adjustment of the wrist, prioritizing fingertip accuracy. Second sub-objective (human-like form preservation): used to constrain the naturalness of the robot's motion form, ensuring that the local motion direction of the human hand and the local motion direction of the robot are geometrically consistent, making the robot's movements more natural. Third sub-objective (fingertip relative position constraint): used to compensate for human hand perception errors, mapping the distance of the human fingertip with perception errors to a target range containing zero-contact points through a distance-enhanced remapping mechanism, ensuring precise physical contact. Fourth sub-objective (predefined special case gesture guidance): used to guide or anchor the mapping results of specific human hand configurations, enhancing the execution accuracy of key gestures. The fifth sub-objective (joint space motion constraint): is used to constrain the robot joint angles obtained by the mapping to meet the joint limit conditions and motion smoothness conditions, ensuring the physical feasibility and smoothness of the robot's actions.

[0128] Using the robot's differentiable forward kinematics model, the joint angles output by the neural network are converted into the spatial positions of the robot's key points, and then the value of the aforementioned comprehensive weighted optimization objective function is calculated. The gradient of the loss function is backpropagated to the parameters of each layer of the neural network using an error backpropagation algorithm. This training process is repeated until the loss function converges, resulting in a trained deep neural network.

[0129] During the online teleoperation phase, the real-time collected 3D spatial position data of the key points of the human hand are directly input into the trained deep neural network. The network performs a single forward propagation calculation and outputs the corresponding target joint angle of the robot. Subsequently, this target joint angle is sent to the robot controller, driving the robot's robotic arm and dexterous hand to perform actions corresponding to the operator's hand movements. Since the forward computation of the deep neural network only involves basic operations such as matrix multiplication and activation functions, the computational load is far less than that of solving high-dimensional nonlinear optimization problems in traditional methods. Therefore, single-frame mapping can be completed in milliseconds, meeting the requirements of high-frequency real-time control.

[0130] This application also provides a dexterous arm heterogeneous teleoperation motion mapping system for implementing the dexterous arm heterogeneous teleoperation motion mapping method described in the above embodiments, referencing... Figure 3 As shown, Figure 3 This is a schematic diagram of a dexterous arm heterogeneous teleoperation motion mapping system provided in an embodiment of this application. The system includes: The data acquisition module 310 is used to acquire the three-dimensional spatial position data of key points of the human hand.

[0131] The storage module 330 stores a mapping function constructed and optimized according to the dexterous arm heterogeneous teleoperation motion mapping method described in the above embodiments.

[0132] The control module 330 is used to input the three-dimensional spatial position data of the key points of the human hand acquired in real time by the data acquisition module into the mapping function stored in the storage module to obtain the target joint angle of the robot and drive the robot to perform corresponding actions.

[0133] It is understood that the dexterous arm heterogeneous teleoperation motion mapping system in the embodiments of this application can realize the dexterous arm heterogeneous teleoperation motion mapping method in the above embodiments. The dexterous arm heterogeneous teleoperation motion mapping system has the same advantages as the above dexterous arm heterogeneous teleoperation motion mapping method compared with the prior art, and will not be repeated here.

[0134] This application also provides an electronic device, see embodiments thereof. Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 4 As shown, the electronic device 400 includes a memory 410 and a processor 420. The memory 410 and the processor 420 are connected via a bus for communication. The memory 410 stores a computer program that can run on the processor 420 to implement the steps of the dexterous arm heterogeneous teleoperation motion mapping method described in the embodiments of this application.

[0135] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the dexterous arm heterogeneous teleoperation motion mapping method described in this application.

[0136] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the dexterous arm heterogeneous teleoperation motion mapping method described in this application.

[0137] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0138] This application describes embodiments of methods and apparatus according to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0139] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0141] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0142] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0143] The above provides a detailed description of a method and system for mapping heterogeneous teleoperation movements of a dexterous arm and hand, as provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for mapping heterogeneous teleoperation movements of a dexterous arm, characterized in that, include: Collect three-dimensional spatial position data of key points on the human hand; Construct a mapping function from the three-dimensional spatial position data of the key points of the human hand to the joint angles of the robot; wherein the robot includes a robotic arm and a dexterous hand; A comprehensive weighted optimization objective function is constructed, which is a weighted sum of multiple optimization sub-objectives. The multiple optimization sub-objectives include at least: a first sub-objective for constraining the positioning accuracy of the robot's fingertips, a second sub-objective for constraining the naturalness of the robot's motion shape, and a third sub-objective for compensating for human hand perception errors. Using the comprehensive weighted optimization objective function as the optimization criterion, the mapping function is optimized based on the three-dimensional spatial position data of the key points of the human hand to obtain the optimized mapping function; In real-time teleoperation, the three-dimensional spatial position data of the key points of the human hand collected in real time are input into the optimized mapping function to obtain the target joint angle of the robot, so as to drive the robot to perform corresponding actions.

2. The method according to claim 1, characterized in that, The first sub-target item is constructed according to the following steps: The position of the target fingertip of the human hand is determined as the global position reference, and the desired wrist orientation of the robot is determined. Based on the mapping function, determine the position of the robot's target fingertip corresponding to the position of the human hand's target fingertip, as well as the robot's actual wrist orientation; Calculate the positional deviation between the target fingertip position of the human hand and the target fingertip position of the robot, and calculate the orientation deviation between the desired wrist orientation of the robot and the actual wrist orientation of the robot; The position deviation and the orientation deviation are weighted and summed to obtain the first sub-target item; wherein the weight of the position deviation is greater than the weight of the orientation deviation, so as to allow the robot wrist to make local adaptive adjustments while prioritizing the positioning accuracy of the target fingertip position.

3. The method according to claim 2, characterized in that, The target fingertip position of the human hand is the fingertip position of the human thumb; the position deviation is the deviation between the fingertip position of the human thumb and the fingertip position of the robot thumb; the orientation deviation is the angular deviation between the robot's desired wrist orientation and the robot's actual wrist orientation.

4. The method according to claim 1, characterized in that, The second sub-target item is constructed according to the following steps: Based on the three-dimensional spatial position data of the key points of the human hand, the current relative position of the fingertips relative to the palm of the human hand is determined, and at the current relative position, at least one random small relative displacement vector is generated, which represents the local movement trend direction of the fingertips relative to the palm of the human hand. Based on the mapping function and the robot's forward kinematics model, calculate the predicted relative displacement vector of the robot's fingertip relative to the robot's palm, corresponding to the tiny relative displacement vector; The minute relative displacement vector and the predicted relative displacement vector are normalized to unit direction vectors, and the difference between the two normalized unit direction vectors is calculated as a difference measure. The generated minute relative displacement vectors and the difference measures corresponding to each finger are accumulated or averaged to obtain the second sub-target item.

5. The method according to claim 1, characterized in that, The third sub-target item is constructed according to the following steps: Based on the three-dimensional spatial position data of the key points of the human hand, the relative position vector between the tip of the thumb and the tips of the other fingers is determined, and the magnitude of each relative position vector is calculated as the fingertip distance perceived by the human hand. Based on the fingertip distance perceived by the human hand, the compensated target distance is calculated using a preset distance compensation function; wherein, the distance compensation function is used to map the fingertip distance perceived by the human hand from the original interval containing non-zero perception error to the target interval containing zero contact point. Based on the direction of the relative position vector and the compensated target distance, the remapped expected relative position vector is obtained; Based on the current mapping function and the robot's forward kinematics model, calculate the actual relative position vectors of the robot side corresponding to the tip of the thumb and the tips of the other fingers; Calculate the deviation between the desired relative position vector after remapping and the actual relative position vector on the robot side; A weighting coefficient is determined based on the fingertip distance perceived by the human hand, and the weighting coefficient takes a larger value when the fingertip distance is small; The product of the deviation corresponding to each finger and the weighting coefficient is accumulated or averaged to obtain the third sub-target item.

6. The method according to claim 5, characterized in that, Based on the fingertip distance perceived by the human hand, the compensated target distance is calculated using a preset distance compensation function, including: If the fingertip distance perceived by the human hand is less than the minimum error threshold, the compensated target distance is mapped to zero. When the fingertip distance perceived by the human hand is between the minimum error threshold and the fine operation threshold, the compensated target distance is calculated using a smooth nonlinear mapping function; If the fingertip distance perceived by the human hand is greater than the fine operation threshold, the compensated target distance is set to be equal to the fingertip distance perceived by the human hand.

7. The method according to claim 1, characterized in that, The plurality of optimization sub-objectives also includes a fourth sub-objective, used to guide or anchor the mapping results of a specific hand configuration during the optimization process; the fourth sub-objective is constructed according to the following steps: Pre-set at least one special gesture, and save the corresponding human hand key point configuration data for each special gesture as the preset human hand configuration, and save the expected robot joint angle data as the preset robot configuration; Calculate the current output robot joint angle corresponding to the preset human hand configuration based on the mapping function; Calculate the configuration deviation between the current output robot joint angle and the preset robot configuration; A confidence weight is assigned to each special gesture, and the confidence weight is used to adjust the constraint strength of the special gesture on the mapping function; The fourth sub-target item is obtained by summing or averaging the product of the configuration deviation and the confidence weight corresponding to each special gesture.

8. The method according to claim 1, characterized in that, The plurality of optimization sub-objectives also includes a fifth sub-objective, used to constrain the robot joint angles obtained by the mapping to satisfy joint constraint conditions and motion smoothness conditions; the fifth sub-objective is constructed according to the following steps: Based on the mapping function, obtain the current output joint angle of each joint of the robot; For each joint, calculate the joint limit exceedance based on whether the current output joint angle exceeds the preset joint physical limit; Based on the mapping function outputs of the current and historical moments, the second derivative of the mapping function with respect to time is calculated as a temporal variation feature of the joint motion, which is used to characterize the smoothness of the joint motion. The fifth sub-target item is obtained by weighted summation of the joint limit excess and the temporal change characteristic.

9. The method according to any one of claims 1-8, characterized in that, The mapping function is a deep neural network; Using the aforementioned comprehensive weighted optimization objective function as the optimization criterion, the mapping function is optimized based on the three-dimensional spatial position data of the key points of the human hand, including: Using the three-dimensional spatial position data of the key points of the human hand as training samples, the comprehensive weighted optimization objective function as the loss function, and the robot's differentiable forward kinematics model, the deep neural network is trained through the error backpropagation algorithm to obtain the trained deep neural network. The real-time acquired 3D spatial position data of the key points of the human hand are input into the optimized mapping function to obtain the target joint angles of the robot, including: The three-dimensional spatial position data of the key points of the human hand acquired in real time are input into the trained deep neural network, and the target joint angle of the robot is obtained through a forward calculation.

10. A dexterous arm-hand heterogeneous teleoperation motion mapping system, characterized in that, For implementing the dexterous arm heterogeneous teleoperation motion mapping method as described in any one of claims 1 to 9, the system comprises: The data acquisition module is used to collect three-dimensional spatial position data of key points on the human hand; The storage module stores a mapping function constructed and optimized according to the dexterous arm heterogeneous teleoperation motion mapping method as described in any one of claims 1 to 9; The control module is used to input the three-dimensional spatial position data of the key points of the human hand acquired in real time by the data acquisition module into the mapping function stored in the storage module to obtain the target joint angle of the robot and drive the robot to perform corresponding actions.

Images