Control method and apparatus for humanoid robot

Abstract

The application discloses a control method and device of a humanoid robot, and relates to the technical field of robots, and comprises the following steps: acquiring three-dimensional position data of hand skeleton key points of a target operator and pose data of a palm root skeleton of the target operator; determining a skeletal direction vector of each finger joint of the target operator according to the three-dimensional position data, and determining each target finger joint angle of a dexterous hand of a humanoid robot to be controlled according to the skeletal direction vector; generating an upper limb joint angle reference track of each upper limb joint of the humanoid robot to be controlled according to the pose data, and generating a joint driving instruction for controlling the motion of the upper limb and the dexterous hand of the humanoid robot to be controlled according to the target finger joint angle and the upper limb joint angle reference track. The application improves the grasping posture reproduction accuracy of the humanoid robot, and enhances the coordination between the grasping action and the arm motion.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a control method and apparatus for a humanoid robot. Background Technology

[0002] Existing robot teleoperation solutions typically utilize interactive devices such as VR controllers or data gloves to acquire motion information from the operator's hands when controlling humanoid robots, and then use this information to drive the robot's upper limbs and dexterous hands to perform corresponding actions.

[0003] However, existing solutions still have the following shortcomings in practical applications: Firstly, in the process of mapping the operator's hand posture to a dexterous hand grasping configuration, existing methods mostly adopt direct proportional mapping based on joint angles or preset gesture template matching. This has limited ability to reproduce complex gestures and makes it difficult to accurately reproduce the multi-joint collaborative posture of the human hand during fine grasping, thus affecting operational accuracy and task adaptability.

[0004] Secondly, dexterous hand control and upper limb control are usually executed independently, making it difficult for the robot's grasping action and arm movement to coordinate. The movement performance differs significantly from the natural movement patterns of the human body, which restricts the immersive experience and execution efficiency of teleoperation. Summary of the Invention

[0005] This invention provides a control method and device for a humanoid robot to solve the problem that existing solutions cannot accurately reproduce the complex grasping posture of a human hand, as well as the lack of parallel coordination between dexterous hand control and upper limb control.

[0006] According to one aspect of the present invention, a control method for a humanoid robot is provided, the method comprising: Acquire the three-dimensional position data of key points of the hand bones of the target operator, as well as the pose data of the bones at the base of the palm of the target operator; The skeletal direction vectors of each finger joint of the target operator are determined based on the three-dimensional position data, and the angles of each target finger joint of the dexterous hand on the humanoid robot to be controlled are determined based on the skeletal direction vectors. Based on the pose data, reference trajectories of the upper limb joint angles of the humanoid robot to be controlled are generated. Based on the target finger joint angles and the upper limb joint angle reference trajectories, joint drive commands for controlling the movement of the upper limbs and the dexterous hand of the humanoid robot to be controlled are generated.

[0007] According to another aspect of the present invention, a control device for a humanoid robot is provided, the device comprising: The data acquisition module is used to acquire the three-dimensional position data of key points of the hand bones of the target operator, as well as the pose data of the bones at the base of the palm of the target operator; The target finger joint angle determination module is used to determine the skeletal direction vector of each finger joint of the target operator based on the three-dimensional position data, and to determine the angle of each target finger joint of the dexterous hand on the humanoid robot to be controlled based on the skeletal direction vector. The joint drive command generation module is used to generate upper limb joint angle reference trajectories for each upper limb joint of the humanoid robot to be controlled based on the pose data, and to generate joint drive commands for controlling the movement of the upper limbs and the dexterous hand of the humanoid robot to be controlled based on the target finger joint angles and the upper limb joint angle reference trajectories.

[0008] The beneficial effects of this invention are as follows: Firstly, by acquiring the three-dimensional position data of the key points of the hand skeleton of the target operator, and determining the skeletal direction vector of each finger joint based on the three-dimensional position data, the angles of each target finger joint of the dexterous hand can be calculated. Since the skeletal direction vector directly originates from the spatial geometric relationship between adjacent key points of the operator's hand skeleton, the target finger joint angles of the dexterous hand can be generated based on the relative posture of each finger joint of the operator's hand. Compared with the simplified proportional mapping or template matching methods in the prior art, this invention can more accurately reflect the true posture of each finger joint when the human hand is grasping, thereby improving the reproduction accuracy of the grasping configuration and the adaptability to different gestures.

[0009] Secondly, while calculating the angles of the target finger joints of the dexterous hand, the pose data of the bones at the base of the operator's hand are also acquired, and reference trajectories of the upper limb joint angles are generated accordingly. Finally, the target finger joint angles and the upper limb joint angle reference trajectories are used together as the basis for generating joint drive commands. Thus, the grasping configuration of the dexterous hand and the end-effector pose trajectory of the upper limb are generated in parallel and output collaboratively in the same control process, enabling the grasping action of the dexterous hand and the movement of the upper limb to be effectively coordinated in terms of temporal and spatial relationships. This makes the robot's overall movement more in line with the natural collaborative movement patterns of the human body, thereby improving the immersion of teleoperation and the task execution effect.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a control method for a humanoid robot provided in Embodiment 1 of the present invention; Figure 2 A flowchart of another control method for a humanoid robot provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a control device for a humanoid robot provided in Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the control method for a humanoid robot according to an embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," "third," "fourth," "fifth," "sixth," "seventh," "eighth," "ninth," and "tenth," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] Example 1 Figure 1This is a flowchart of a control method for a humanoid robot provided in Embodiment 1 of the present invention. This embodiment is applicable to scenarios involving remote operation control of humanoid robots, particularly scenarios requiring mapping of the operator's hand posture to a dexterous hand grasping configuration and coordinating upper limb movements with grasping actions. This method can be executed by a control device for the humanoid robot, which can be implemented in hardware and / or software. Figure 1 As shown, the method includes: S101. Obtain the three-dimensional position data of the key points of the hand bones of the target operator, as well as the pose data of the bones at the base of the palm of the target operator.

[0016] In this context, the target operator refers to a natural person user who controls the humanoid robot remotely. In this embodiment of the invention, the target operator's hand posture and arm movements serve as the control input source for the humanoid robot to be controlled, guiding it to perform corresponding upper limb movements and grasping operations.

[0017] Hand skeletal keypoints refer to a set of feature points collected from the target operator's hand using a hand tracking device. These points are used to represent the skeletal structure of the human hand and the positional distribution of its joints in three-dimensional space. Hand skeletal keypoints typically cover the palm area of ​​the target operator's hand, as well as the joint positions of the thumb, index finger, middle finger, ring finger, and little finger, providing a spatial basis for subsequent finger joint vector construction and joint angle calculation.

[0018] Taking a single hand as an example, key points of the hand bones may include, but are not limited to: key points of the wrist joint, key points of the base of the palm, key points of the thumb's carpometacarpophalangeal joint, key points of the thumb's metacarpophalangeal joint, key points of the thumb's interphalangeal joint, key points of the thumb's fingertip, key points of the index finger's metacarpophalangeal joint, key points of the index finger's proximal interphalangeal joint, key points of the index finger's distal interphalangeal joint, key points of the index fingertip, key points of the middle finger's metacarpophalangeal joint, key points of the middle finger's proximal interphalangeal joint, key points of the middle finger's distal interphalangeal joint, key points of the middle fingertip, key points of the ring finger's metacarpophalangeal joint, key points of the ring finger's proximal interphalangeal joint, key points of the ring finger's distal interphalangeal joint, key points of the ring fingertip, key points of the little finger's metacarpophalangeal joint, key points of the little finger's proximal interphalangeal joint, key points of the little finger's distal interphalangeal joint, and key points of the little fingertip. Each key point of the hand bones corresponds to a set of three-dimensional position data in a preset spatial coordinate system, and the direction of the line connecting adjacent key points can be used to characterize the skeletal orientation of the corresponding phalanx.

[0019] Three-dimensional position data refers to the three-dimensional coordinate values ​​of each key point of the hand skeleton in a preset spatial coordinate system. This data describes the specific orientation of each key point in space and is the direct basis for constructing finger joint direction vectors and analyzing the geometric relationships of hand posture.

[0020] The palm base bone refers to a reference bone point located in the center of the target operator's palm area or near the base of the wrist. This point can reflect the overall spatial position and orientation trend of the palm, and therefore is used as a reference for obtaining the target pose of the upper limb in this embodiment of the invention.

[0021] The pose data of the hand heel bones refers to the combined expression of the position and orientation information of the hand heel bones in a preset spatial coordinate system. The position information describes the specific coordinates of the hand heel in space, and the orientation information describes the orientation angle of the hand heel. Together, they constitute the target pose input used to drive the generation of upper limb joint trajectories.

[0022] In one implementation, the system can acquire 3D position data of key points in the hand bones and pose data of the bones at the base of the hand by communicating with a VR headset worn by the target operator. Specifically, the VR headset has multiple built-in front-facing cameras and a depth sensor, enabling bare-hand tracking. During operation, the VR headset uses the front-facing cameras to capture real-time image information of the target operator's hand and processes the image data using a built-in hand bone tracking algorithm to generate 3D position coordinates of each key point in the target operator's hand in the device's coordinate system, serving as the 3D position data of the key points in the hand bones. Simultaneously, based on the spatial distribution relationship of multiple key points in the palm area, the VR headset estimates the position and pose information of the bones at the base of the hand in real-time, serving as the pose data of the bones at the base of the hand. By receiving the above data transmitted in real-time from the VR headset, the system acquires the 3D position data of the key points in the hand bones and the pose data of the bones at the base of the hand. This method does not require additional physical devices to be worn on the target operator's hand, resulting in a high degree of integration.

[0023] In another implementation, the system can communicate with a combination of a spatial positioning tracker and a hand posture acquisition device to acquire 3D position data of key points of the hand bones and pose data of the palm root bones. Specifically, the target operator can wear a spatial positioning tracker near the back of the hand or wrist. The system receives the position and posture information calculated by the tracker as the pose data of the palm root bones. Simultaneously, the target operator wears a hand posture acquisition device such as a data glove. This device detects the posture changes of each finger joint using flexion sensors or inertial sensors distributed at each finger joint. The system receives the hand bone structure data generated by this device, processes it, and obtains the 3D position data of key points of each bone in the target operator's hand. When acquiring these two types of data, the system can ensure the temporal consistency between the 3D position data of the key points of the hand bones and the pose data of the palm root bones through a timestamp synchronization mechanism.

[0024] S102. Determine the skeletal direction vectors of each finger joint of the target operator based on the three-dimensional position data, and determine the angles of each target finger joint of the dexterous hand on the humanoid robot to be controlled based on the skeletal direction vectors.

[0025] In this context, a finger joint refers to a skeletal segment formed between two adjacent key points of the hand bones of the operator. Each finger consists of multiple sequentially connected joints, which are linked together by joints. The spatial orientation of a finger joint can be represented by a vector connecting its two ends to the key points of the hand bones.

[0026] Taking the index finger as an example, the key skeletal points of this finger include the metacarpophalangeal joint key points, the proximal interphalangeal joint key points, the distal interphalangeal joint key points, and the fingertip key points. Among them, the metacarpophalangeal joint key points and the proximal interphalangeal joint key points form the proximal phalanx of the index finger, the proximal interphalangeal joint key points and the distal interphalangeal joint key points form the middle phalanx of the index finger, and the distal interphalangeal joint key points and the fingertip key points form the distal phalanx of the index finger.

[0027] The segmentation of the other fingers is similar to that of the index finger. Although the thumb differs from the other four fingers in its joint naming, it is also composed of multiple sequentially connected segments. The segmentation of the thumb and the calculation of its skeletal direction vector follow the same principle of vector difference between key points of adjacent hand bones.

[0028] The skeletal orientation vector is a direction vector obtained by calculating the vector difference between the three-dimensional position coordinates of two adjacent key points of the same finger in the target operator's hand. It is used to represent the direction and length of the corresponding finger joint in space. The skeletal orientation vector is an intermediate geometric quantity for solving the angles of each finger joint in the target operator's hand and for subsequently mapping the angles of the target finger joints in a dexterous hand.

[0029] The humanoid robot to be controlled refers to the robot device that is the object of control in the embodiments of the present invention. The robot has a humanoid upper limb structure and dexterous hand, and can receive and execute joint drive commands to simulate the upper limb movement and hand grasping action of the target operator.

[0030] The dexterous fingers, attached to the end of the upper limb of the humanoid robot to be controlled, form a humanoid robotic hand with multiple independently actuated finger joints, used to perform fine manipulation tasks such as grasping and pinching. In this embodiment of the invention, the angles of each target finger joint of the dexterous hand are generated by mapping and solving based on the three-dimensional position data of key points of the hand bones of the target operator.

[0031] The target joint angle refers to the joint angle value used to drive the dexterous hand of the humanoid robot to move to the desired position. The target joint angle is obtained by parametric transformation of the joint angles of the target operator's hand, and serves as the direct basis for generating dexterous hand joint drive commands.

[0032] In one implementation, after acquiring the three-dimensional position data of the key points of the target operator's hand bones, the system performs a vector difference operation on the three-dimensional position coordinates of two adjacent key points of the hand bones for each finger of the target operator to obtain a skeletal direction vector representing the spatial orientation of each finger joint. For example, for the index finger, the system can use the vector from the metacarpophalangeal joint key point to the proximal interphalangeal joint key point as the skeletal direction vector of the proximal phalanx, and the vector from the proximal interphalangeal joint key point to the distal interphalangeal joint key point as the skeletal direction vector of the middle phalanx, and so on.

[0033] After obtaining the skeletal direction vectors of each finger joint, the system calculates the spatial angles between the skeletal direction vectors of adjacent finger joints to obtain the angles of each finger joint reflecting the target operator's hand. Based on this, the system applies parameter transformation to the original angle information and adapts it to the joint motion space of the dexterous hand on the humanoid robot to be controlled, thus obtaining the angles of each target finger joint of the dexterous hand.

[0034] In another implementation, the system can pre-construct a mapping model from the 3D position data of key points of the hand skeleton to the angles of the target finger joints of the dexterous hand. After acquiring the 3D position data of the key points of the target operator's hand skeleton, the system also first determines the skeletal orientation vector of each finger joint by calculating the vector difference between adjacent key points. Subsequently, the system inputs the skeletal orientation vector or the posture features extracted based on the skeletal orientation vector into the pre-constructed mapping model, which outputs the angles of each target finger joint of the dexterous hand.

[0035] The mapping model can be trained or calibrated using offline sample data, and it establishes a correspondence between the skeletal posture features of the human hand and the joint angle space of the dexterous hand. During actual operation, the system only needs to input the relevant features of the real-time acquired skeletal orientation vectors into the mapping model to quickly obtain the corresponding target finger joint angles of the dexterous hand. This method replaces real-time geometric calculation with a pre-established mapping relationship, reducing online computational complexity while ensuring mapping accuracy.

[0036] S103. Generate reference trajectories of upper limb joint angles for each upper limb joint of the humanoid robot to be controlled based on the pose data, and generate joint drive commands for controlling the movement of the upper limbs and dexterous hands of the humanoid robot to be controlled based on the target finger joint angles and upper limb joint angle reference trajectories.

[0037] In this context, "each upper limb joint of the humanoid robot to be controlled" refers to the collective term for all movable joints within the upper limb portion of the robot. These upper limb joints are located in the shoulder, elbow, and wrist areas of the robot, with each joint corresponding to one or more independently controllable rotational degrees of freedom. By driving each upper limb joint to move at a target angle, the position and posture of the robot's upper limb end-effector in space can be adjusted, thereby simulating the motion trajectory of a target operator's arm.

[0038] The upper limb joint angle reference trajectory refers to the sequence of target angles used to guide the movement of each upper limb joint of the humanoid robot to be controlled. The upper limb joint angle reference trajectory is generated by solving the pose data of the hand root bones and contains target angle values ​​of the upper limb joints at multiple consecutive time points. This reference trajectory describes the desired motion path of each joint of the upper limb of the humanoid robot as it changes over time, providing a kinematic reference for the generation of upper limb joint actuation commands.

[0039] Taking the elbow joint as an example, the upper limb joint angle reference trajectory can be represented as follows: the target angle is 0.50 radians at time t0, 0.52 radians at time t1, 0.54 radians at time t2, and so on. This sequence describes the expected motion path of the elbow joint angle changing over time. The upper limb joint angle reference trajectories of other upper limb joints (such as the shoulder joint, wrist joint, etc.) follow the same serialization representation. The target angle sequences of each joint together constitute the overall motion reference of the upper limb of the humanoid robot to be controlled.

[0040] Joint drive commands refer to the instruction information used to directly control the upper limb joints and dexterous finger joints of the humanoid robot to perform movements. Joint drive commands are generated by the system based on the target finger joint angles and upper limb joint angle reference trajectories, and may include control parameters such as the target position, movement speed, or driving torque of each joint. After receiving the joint drive commands, the humanoid robot to be controlled drives the corresponding joint motors or actuators to move, thereby achieving synchronous control of the upper limb posture and the dexterous hand grasping configuration.

[0041] In one implementation, after acquiring the pose data of the base of the hand bones of the target operator, the system uses this pose data as the target pose of the upper limb end effector of the humanoid robot to be controlled. Based on the target pose of the upper limb end effector, the system calculates and determines the target angles of each upper limb joint of the humanoid robot at multiple consecutive time points, forming an upper limb joint angle reference trajectory. This reference trajectory describes the desired motion path of each upper limb joint over time.

[0042] Meanwhile, the system has obtained the target finger joint angles of the dexterous hand through the aforementioned steps. Based on the target finger joint angles and the upper limb joint angle reference trajectories, the system jointly generates drive commands for controlling the upper limb joint movements of the humanoid robot under control, as well as drive commands for controlling the dexterous finger joint movements. After receiving the joint drive commands, the humanoid robot under control drives the corresponding joint actuators to move, thereby achieving synchronous execution of upper limb end-effector pose following and dexterous hand grasping configuration. This method, by generating upper limb reference trajectories and dexterous hand target angles separately, and then coordinating the output during the command generation stage, has a clear structure and is easy to implement.

[0043] In another implementation, after acquiring the pose data of the base of the hand bones of the target operator and the angles of each target finger joint of the dexterous hand, the system uses the pose data of the base of the hand bones as the basis for the desired pose of the upper limb end of the humanoid robot to be controlled, and uses the angles of each target finger joint of the dexterous hand as the basis for the desired position of each finger joint of the dexterous hand. Based on the above two types of basis, and considering the limitations of the range of motion of each joint of the humanoid robot to be controlled, the system uniformly determines the joint drive commands for each upper limb joint of the humanoid robot to be controlled and each finger joint of the dexterous hand. After receiving the joint drive commands, the humanoid robot to be controlled drives the corresponding joint actuators to move. This method integrates upper limb motion control and dexterous hand grasping control into the same processing flow for unified output, which enables the movements of the upper limb and dexterous hand to remain synchronized in time, helping to improve the overall coordination of movement.

[0044] The beneficial effects of the embodiments of the present invention are as follows: Firstly, by acquiring the three-dimensional position data of the key points of the hand skeleton of the target operator, and determining the skeletal direction vector of each finger joint based on the three-dimensional position data, the angles of each target finger joint of the dexterous hand can be calculated. Since the skeletal direction vector directly originates from the spatial geometric relationship between adjacent key points of the operator's hand skeleton, the target finger joint angles of the dexterous hand can be generated based on the relative posture of each finger joint of the operator's hand. Compared with the simplified proportional mapping or template matching methods in the prior art, this invention can more accurately reflect the true posture of each finger joint when the human hand is grasping, thereby improving the reproduction accuracy of the grasping configuration and the adaptability to different gestures.

[0045] Secondly, while calculating the angles of the target finger joints of the dexterous hand, the pose data of the bones at the base of the operator's hand are also acquired, and reference trajectories of the upper limb joint angles are generated accordingly. Finally, the target finger joint angles and the upper limb joint angle reference trajectories are used together as the basis for generating joint drive commands. Thus, the grasping configuration of the dexterous hand and the end-effector pose trajectory of the upper limb are generated in parallel and output collaboratively in the same control process, enabling the grasping action of the dexterous hand and the movement of the upper limb to be effectively coordinated in terms of temporal and spatial relationships. This makes the robot's overall movement more in line with the natural collaborative movement patterns of the human body, thereby improving the immersion of teleoperation and the task execution effect.

[0046] Example 2 Figure 2 This is a flowchart illustrating another control method for a humanoid robot provided in Embodiment 2 of the present invention. The above technical solution is further optimized and expanded, and can be combined with the various optional embodiments described above. For example... Figure 2 As shown, the method includes: S201. Obtain the three-dimensional position data of the key points of the hand bones of the target operator, as well as the pose data of the bones at the base of the palm of the target operator.

[0047] S202. For each finger of the target operator, perform vector difference operation on the three-dimensional position data of adjacent hand bone key points to obtain the bone direction vector of the corresponding finger joint.

[0048] Adjacent hand skeletal keypoints refer to two hand skeletal keypoints that are sequentially adjacent along the finger's extension direction within the hand skeletal keypoint sequence of the same finger of the target operator. Taking a single finger as an example, from the base of the finger to the fingertip, the preceding keypoint and the following keypoint are adjacent hand skeletal keypoints.

[0049] Vector difference operation refers to performing vector subtraction on the three-dimensional position coordinates of two key points of the hand bones in a preset spatial coordinate system to obtain a direction vector pointing from the previous key point to the next key point. This direction vector contains both the pointing information and length information of the knuckle.

[0050] A corresponding finger joint refers to a bone segment defined by the preceding and following keypoints of adjacent hand bones. Each corresponding finger joint corresponds one-to-one with an adjacent hand bone keypoint through vector difference calculation, and the bone direction vector obtained by the vector difference operation is the bone direction vector of that corresponding finger joint.

[0051] In one implementation, after acquiring the three-dimensional position data of the key points of the hand bones of the target operator, the system sequentially determines the arrangement order of the key points of the hand bones on each finger, from the base to the tip. For any two adjacent key points on that finger, the system subtracts the three-dimensional position coordinates of the preceding key point from the three-dimensional position coordinates of the latter key point, obtaining the vector difference result pointing from the preceding key point to the following key point. This vector difference result is the skeletal direction vector of the finger joint defined by the two adjacent key points, its direction representing the orientation of the joint in space, and its magnitude reflecting the length of the joint in space. By sequentially performing the above vector difference operation on each group of adjacent key points on each finger, the system can obtain the skeletal direction vectors of all the finger joints of the target operator.

[0052] Taking the index finger as an example, the key points of the hand bones of this finger, from the base of the finger to the fingertip, include: the key points of the metacarpophalangeal joint, the key points of the proximal interphalangeal joint, the key points of the distal interphalangeal joint, and the key points of the fingertip.

[0053] The system first treats the metacarpophalangeal joint key point and the proximal interphalangeal joint key point as a set of adjacent hand skeletal key points, and subtracts the three-dimensional position coordinates of the metacarpophalangeal joint key point from the three-dimensional position coordinates of the proximal interphalangeal joint key point to obtain the skeletal direction vector of the proximal phalanx of the index finger.

[0054] Subsequently, the system treats the proximal interphalangeal joint key point and the distal interphalangeal joint key point as a set of adjacent hand skeletal key points, and subtracts the three-dimensional position coordinates of the proximal interphalangeal joint key point from the three-dimensional position coordinates of the distal interphalangeal joint key point to obtain the skeletal direction vector of the middle phalanx of the index finger.

[0055] Finally, the system treats the distal interphalangeal joint key point and the fingertip key point as a set of adjacent hand skeletal key points, and subtracts the three-dimensional position coordinates of the distal interphalangeal joint key point from the three-dimensional position coordinates of the fingertip key point to obtain the skeletal direction vector of the distal phalanx of the index finger.

[0056] Through the vector difference operation described above, the system obtains three skeletal direction vectors for the index finger, corresponding to the proximal, middle, and distal phalanges, respectively. The determination of the skeletal direction vectors for the other fingers is similar, all following the principle of vector difference operation between key points of adjacent hand bones.

[0057] By performing vector difference calculations on the 3D position data of adjacent hand bone key points for each finger of the target operator, the skeletal direction vector of the corresponding finger joint is obtained. The beneficial effects are: This allows for the extraction of directional information reflecting the actual spatial orientation and length of each finger joint, based directly on the spatial geometric relationships between key points of the operator's hand bones. Compared to methods that only focus on the absolute position of joints or use overall gesture template matching, this vector difference operation can more precisely characterize the independent posture changes of each finger joint in three-dimensional space. This provides intermediate geometric quantities with clear physical correspondences for subsequent calculations of the angles of the target finger joints in dexterity hands, helping to improve the accuracy of mapping from human hand posture to dexterity hand grasping configuration.

[0058] S203. Determine the key points of the hand bones from the key points of the hand bones, and construct the hand reference vector and the hand plane normal vector based on the key points of the hand bones; determine the angles of each finger joint of the target operator's hand based on the spatial angle between the bone direction vectors and the spatial angle between the bone direction vector and the hand reference vector or the hand plane normal vector.

[0059] Among them, the skeletal key points of the palm region refer to the key points of the hand skeleton located in the palm region of the target operator. The skeletal key points of the palm region include, but are not limited to, wrist joint key points, palm root key points, and metacarpophalangeal joint key points of each finger. The skeletal key points of the palm region are used to construct geometric reference elements representing the spatial posture of the palm.

[0060] The palm reference vector is a direction vector obtained by calculating the vector difference between two specific key points in the skeletal structure of the palm region. It is used to characterize the reference pointing direction of the palm in space, serving as one of the reference bases for determining the angles of the finger joints of the target operator's hand.

[0061] The palm plane normal vector is the normal vector of a plane determined by at least three non-collinear key points among the key points of the hand bones. The palm plane normal vector is used to characterize the spatial orientation of the plane containing the palm and serves as one of the reference bases for determining the angles of the finger joints of the target operator's hand.

[0062] The angles of each finger joint in the target operator's hand refer to intermediate angles determined based on the spatial angles between skeletal direction vectors and the spatial angles between skeletal direction vectors and the palm reference vector or the palm plane normal vector. These angles reflect the degree of bending of each finger joint in the target operator's hand. The angles of each finger joint in the target operator's hand characterize the actual posture of each finger joint in space and serve as the direct basis for subsequently obtaining the angles of each target finger joint in the dexterous hand of the humanoid robot to be controlled through parameter transformation.

[0063] In one implementation, after acquiring the three-dimensional position data of the key points of the target operator's hand bones, the system first identifies a portion of the key points located in the palm region from the key points of the hand bones, which are then used as the key points of the palm region bones. The key points of the palm region bones may include wrist joint key points, key points at the base of the palm, and key points of the metacarpophalangeal joints of each finger, etc.

[0064] After determining the key points of the hand skeleton, the system selects two specific key points and uses the vector difference between the points as the reference vector for the hand. This reference vector represents the basic pointing direction of the hand in space.

[0065] Simultaneously, the system selects at least three non-collinear key points from the key points of the hand's skeleton. Using one of these key points as a reference, it constructs two vectors located in the hand's plane. By performing a cross product operation on these two vectors, it obtains the normal vector of the hand's plane. This normal vector of the hand's plane is used to characterize the spatial orientation of the plane containing the hand.

[0066] After obtaining the palm reference vector and the palm plane normal vector, the system calculates the corresponding spatial angle for each finger joint of the target operator's hand based on the bone direction vector of the finger and the palm reference vector or the palm plane normal vector, and uses the calculation result as the finger joint angle of the target operator's hand corresponding to that finger joint.

[0067] Specifically, for the joint between two adjacent phalanges of a finger, the system calculates the spatial angle between the skeletal direction vectors corresponding to the two phalanges, which is taken as the joint angle of that joint. For the joint connecting the base of the finger to the palm, the system calculates the spatial angle between the skeletal direction vector of the proximal phalanx of the finger and the palm reference vector, which is taken as the joint angle of that joint in the flexion direction; at the same time, the system projects the skeletal direction vector of the proximal phalanx onto a plane perpendicular to the normal vector of the palm plane, and calculates the spatial angle between it and the palm reference vector, which is taken as the joint angle of that joint in the lateral swing direction.

[0068] By sequentially performing the above spatial angle calculation on each finger joint of the target operator's hand, the system obtains the angles of each finger joint of the target operator's hand.

[0069] For example, taking the index finger as an example, after determining the palm reference vector and the palm plane normal vector, the system calculates the angles of each joint of the index finger as follows: For the proximal interphalangeal joint of the index finger, the system calculates the spatial angle between the skeletal direction vector of the proximal phalanx of the index finger and the skeletal direction vector of the middle phalanx of the index finger, and uses this angle as the joint angle of the proximal interphalangeal joint of the index finger.

[0070] For the metacarpophalangeal joint of the index finger, the system calculates the spatial angle between the skeletal direction vector of the proximal phalanx of the index finger and the palm reference vector, and uses this angle as the joint angle of the metacarpophalangeal joint of the index finger in the flexion direction; at the same time, the system projects the skeletal direction vector of the proximal phalanx of the index finger onto a plane perpendicular to the normal vector of the palm plane, and calculates the spatial angle between it and the palm reference vector, and uses this angle as the joint angle of the metacarpophalangeal joint of the index finger in the lateral swing direction.

[0071] S204. Based on the angles of each finger joint of the target operator's hand, determine the angles of each target finger joint of the dexterous hand on the humanoid robot to be controlled through parameter transformation.

[0072] Parameter transformation refers to the process of applying a preset mathematical transformation to the finger joint angles of the target operator's hand after obtaining them, so as to adapt them to the joint motion space of the dexterous hand on the humanoid robot to be controlled, thus obtaining the target finger joint angles of the dexterous hand. Parameter transformation is used to adjust the numerical correspondence between the finger joint angles of the target operator's hand and the target finger joint angles of the dexterous hand, so that the grasping configuration of the dexterous hand can accurately reflect the hand posture intention of the target operator, while adapting to the joint motion range and mechanical constraints of the dexterous hand itself.

[0073] In one implementation, after obtaining the angles of each finger joint of the target operator's hand, the system adjusts the angle value for each finger joint according to preset transformation parameters to obtain the target finger joint angles applicable to the corresponding finger joints of the dexterous hand on the humanoid robot to be controlled.

[0074] By determining the key points of the hand skeleton from the key points of the hand skeleton, and constructing a hand reference vector and a hand plane normal vector based on the key points of the hand skeleton; determining the angles of each finger joint of the target operator's hand based on the spatial angles between the bone direction vectors and the spatial angles between the bone direction vectors and the hand reference vector or the hand plane normal vector; and determining the angles of each target finger joint of the dexterous hand on the humanoid robot to be controlled through parameter transformation based on the angles of each finger joint of the target operator's hand, the beneficial effects are: Firstly, compared to the method of estimating angles based solely on key points of the fingers, incorporating reference information from the palm region can more accurately reflect the movement relationship of the fingers relative to the overall posture of the palm, making the calculated finger joint angles more consistent with the actual anatomical movement patterns of the human hand.

[0075] Secondly, this allows for angle calculation methods that match the movement characteristics of different types of finger joints. This categorized calculation method helps improve the rationality and accuracy of angle calculations for each finger joint.

[0076] Thirdly, by transforming the parameters of the finger joint angles of the target operator's hand, the angles are adapted to the joint motion space of the dexterous hand on the humanoid robot to be controlled. This ensures that the target finger joint angles of the dexterous hand can accurately reflect the hand posture intention of the target operator, while also meeting the joint motion range constraints and zero-position reference requirements of the dexterous hand itself. This improves the mapping accuracy and adaptability from human hand posture to dexterous hand grasping configuration.

[0077] Optionally, the parameter transformation includes applying at least one of coefficient scaling, orientation flipping, and offset correction to the angles of each finger joint of the target operator's hand.

[0078] Coefficient scaling refers to the transformation process of multiplying the angle values ​​of each finger joint of the target operator's hand by a preset scaling coefficient. Coefficient scaling is used to adjust the numerical ratio between the angles of the finger joints of the target operator's hand and the angles of the finger joints of the dexterous hand, so as to map the range of motion of the human hand joints to the allowable range of motion of the corresponding joints of the dexterous hand, so that the dexterous hand reaches its joint limit position when the target operator's hand reaches its maximum bending angle.

[0079] Direction reversal refers to the transformation process of inverting or maintaining the positive or negative signs of the angle values ​​of each finger joint in the target operator's hand. Direction reversal is used to unify the definition of the movement direction of the target operator's finger joints with the definition of the movement direction of the dexterous hand joints, so that the actual movement direction of the dexterous finger joints is consistent with the movement direction of the target operator's finger joints.

[0080] Offset correction refers to the transformation process of adding the angle values ​​of each finger joint of the target operator's hand to a preset offset. Offset correction is used to correct the reference deviation between the target operator's initial hand posture and the dexterity hand's zero posture, so that when the target operator's hand is in a naturally extended state, the dexterity hand is in its initial zero posture.

[0081] In one implementation, after obtaining the angles of each finger joint of the target operator's hand, the system performs one or more of the following transformation processes for each finger joint angle, based on the mechanism parameters and installation references of the corresponding dexterous finger joint: First, the system multiplies the angle value of the finger joint of the target operator's hand by a preset scaling factor to adjust the numerical range of the angle value. Since the range of motion of each joint of the human hand usually differs from the mechanical limit range of the corresponding joint of a dexterous hand, scaling by a factor can establish a correspondence between the two in terms of range of motion.

[0082] Secondly, the system retains or reverses the sign of the finger joint angle value based on preset direction parameters. Since the definition of the movement direction of the human hand joint and the definition of the motor rotation direction of the dexterous hand joint may be inconsistent, the direction reversal can unify the movement directions of the two.

[0083] Third, the system adds the angle values, after scaling and orientation processing, to a preset offset. Since there is usually a reference deviation between the angle values ​​of each finger joint corresponding to the natural extended posture of the target operator's hand and the joint angle values ​​corresponding to the mechanical zero-position posture of the dexterous hand, this reference deviation can be eliminated through offset correction.

[0084] By sequentially performing the above parameter transformation processing on the angles of each finger joint of the target operator's hand, the system obtains the target finger joint angles that correspond one-to-one with each finger joint of the dexterous hand.

[0085] For example, taking the proximal interphalangeal joint of the index finger as an example, assuming that the system calculates the joint angle of the target operator's hand to be 1.2 radians based on the spatial angle.

[0086] The system first applies a scaling factor to the angle value. If the scaling factor for this joint is 0.8, then the scaled angle value is 0.96 radians.

[0087] The system then applies a direction flip to this angle value. Assuming the joint's direction parameter remains positive, the sign of the angle value remains unchanged, still 0.96 radians.

[0088] Finally, the system applies an offset correction to the angle value. If the offset corresponding to the joint is -0.1 radians, then adding 0.96 radians to -0.1 radians yields 0.86 radians.

[0089] The system uses this 0.86 radians as the target interphalangeal joint angle of the proximal interphalangeal joint of the index finger of the humanoid robot to be controlled. The parameter transformation processing of the other finger joints is similar, and the scaling factor, direction parameter and offset of each joint can be preset according to the specific mechanism parameters of the humanoid hand.

[0090] Optionally, the angles of the finger joints of the target operator's hand include: The first flexion angle of the thumb's carpometacarpophalangeal joint, the first lateral swing angle of the thumb's carpometacarpophalangeal joint, and the second flexion angle of the thumb's metacarpophalangeal joint; the third flexion angle of the index finger's metacarpophalangeal joint, the second lateral swing angle of the index finger's metacarpophalangeal joint, and the fourth flexion angle of the index finger's proximal interphalangeal joint; the fifth flexion angle of the middle finger's metacarpophalangeal joint, and the sixth flexion angle of the middle finger's proximal interphalangeal joint; the seventh flexion angle of the ring finger's metacarpophalangeal joint, the third lateral swing angle of the ring finger's metacarpophalangeal joint, and the eighth flexion angle of the ring finger's proximal interphalangeal joint; the ninth flexion angle of the little finger's metacarpophalangeal joint, the fourth lateral swing angle of the little finger's metacarpophalangeal joint, and the tenth flexion angle of the little finger's proximal interphalangeal joint.

[0091] The flexion angle refers to the angle of movement of the finger joints of the target operator's hand in the direction of finger flexion. The flexion angle is used to characterize the degree to which the fingers retract or bend towards the palm. Among the finger joints of the target operator's hand, the thumb carpometacarpophalangeal joint, thumb metacarpophalangeal joint, index finger metacarpophalangeal joint, index finger proximal interphalangeal joint, middle finger metacarpophalangeal joint, middle finger proximal interphalangeal joint, ring finger metacarpophalangeal joint, ring finger proximal interphalangeal joint, little finger metacarpophalangeal joint, and little finger proximal interphalangeal joint all have the ability to move in the flexion direction, corresponding to the flexion angles from the first to the tenth flexion angle.

[0092] Lateral swing angle refers to the angle of lateral opening and closing movement of the finger joints of the target operator's hand, perpendicular to the direction of finger flexion. The lateral swing angle characterizes the degree to which the fingers move closer to or further away from adjacent fingers within the palm plane. Among the finger joints of the target operator's hand, the thumb carpometacarpophalangeal joint, index finger metacarpophalangeal joint, ring finger metacarpophalangeal joint, and little finger metacarpophalangeal joint have the ability to move in the lateral swing direction, corresponding to the lateral swing angles from the first to the fourth lateral swing angle, respectively.

[0093] The advantages of this optional embodiment are as follows: Firstly, by configuring three angles—a first flexion angle, a first lateral tilt angle, and a second flexion angle—for the thumb, and three angles—a ninth flexion angle, a fourth lateral tilt angle, and a tenth flexion angle—for the little finger, a symmetrical configuration of angles is achieved between the thumb and little finger. This symmetrical configuration allows the thumb and little finger sides to provide a relatively symmetrical distribution of grasping degrees of freedom during grasping operations. This helps the dexterous hand achieve a balanced distribution of contact force during envelope grasping or palm-opposition operations, avoiding grasping posture instability caused by asymmetrical configuration of degrees of freedom on both sides.

[0094] Secondly, by configuring the number and type of angles to match the anatomical motor capabilities of different fingers, the angle composition of each finger joint in the target operator's hand is made consistent with the actual distribution of the degrees of freedom of movement of each finger in a human hand. Specifically, fingers with flexion and lateral movement capabilities are configured with corresponding flexion and lateral movement angles, while fingers with only flexion movement capabilities are configured with only flexion angles. This configuration method ensures that the joint angle composition of each finger accurately reflects its actual motor capabilities, avoiding the incorrect configuration of corresponding angles for joints that do not have the ability to move in a certain direction.

[0095] Optionally, based on the spatial angles between skeletal direction vectors and the spatial angles between skeletal direction vectors and the palm reference vector or the palm plane normal vector, the angles of each finger joint of the target operator's hand are determined, including: 1) The spatial angle between the skeletal direction vector of the proximal phalanx of the thumb and the palm reference vector is determined as the first flexion angle.

[0096] For example, suppose the skeletal direction vector of the proximal phalanx of the thumb points to the front of the palm, and the palm reference vector points to the direction of finger extension. The system calculates that the spatial angle between the two vectors is 0.3 radians, and then 0.3 radians is determined as the first flexion angle.

[0097] 2) The spatial angle between the skeletal direction vector of the proximal phalanx of the thumb and the normal vector of the palm plane is determined as the first lateral swing angle.

[0098] For example, suppose the skeletal direction vector of the proximal phalanx of the thumb is tilted outward relative to the palm plane, and the normal vector of the palm plane is perpendicular to the palm plane and upward. The system calculates that the spatial angle between the two vectors is 0.2 radians, and then 0.2 radians is determined as the first lateral swing angle.

[0099] 3) The spatial angle between the skeletal direction vector of the middle phalanx of the thumb and the skeletal direction vector of the proximal phalanx of the thumb is determined as the second flexion angle.

[0100] For example, if the skeletal direction vector of the middle phalanx of the thumb is further bent relative to the skeletal direction vector of the proximal phalanx of the thumb, and the system calculates that the spatial angle between the two vectors is 0.4 radians, then 0.4 radians is determined as the second flexion angle.

[0101] 4) The spatial angle between the skeletal direction vector of the proximal phalanx of the index finger and the palm reference vector is determined as the third flexion angle.

[0102] For example, suppose the skeletal direction vector of the proximal phalanx of the index finger points towards the front of the palm, and the palm reference vector points towards the direction of finger extension. The system calculates that the spatial angle between the two vectors is 0.3 radians, and then 0.3 radians is determined as the third flexion angle.

[0103] 5) The spatial angle between the component of the skeletal direction vector of the proximal phalanx of the index finger that is perpendicular to the normal vector of the palm plane and the palm reference vector is determined as the second lateral swing angle.

[0104] For example, suppose the skeletal direction vector of the proximal phalanx of the index finger lies in a plane perpendicular to the normal vector of the palm plane, and deflects outward relative to the palm reference vector. The system calculates that the spatial angle between this projection component and the palm reference vector is 0.1 radians, and then determines 0.1 radians as the second lateral swing angle.

[0105] 6) The spatial angle between the skeletal direction vector of the proximal phalanx of the index finger and the skeletal direction vector of the middle phalanx of the index finger is determined as the fourth flexion angle.

[0106] For example, if the skeletal direction vector of the middle phalanx of the index finger is further bent relative to the skeletal direction vector of the proximal phalanx of the index finger, and the system calculates that the spatial angle between the two vectors is 0.5 radians, then 0.5 radians is determined as the fourth flexion angle.

[0107] 7) The spatial angle between the skeletal direction vector of the proximal phalanx of the middle finger and the palm reference vector is determined as the fifth flexion angle.

[0108] For example, if the skeletal direction vector of the proximal phalanx of the middle finger points towards the front of the palm, and the palm reference vector points towards the direction of finger extension, and the system calculates that the spatial angle between the two vectors is 0.3 radians, then 0.3 radians is determined as the fifth flexion angle.

[0109] 8) The spatial angle between the skeletal direction vector of the proximal phalanx of the middle finger and the skeletal direction vector of the middle phalanx of the middle finger is determined as the sixth flexion angle.

[0110] For example, if the skeletal direction vector of the middle phalanx of the middle finger is further bent relative to the skeletal direction vector of the proximal phalanx of the middle finger, and the system calculates that the spatial angle between the two vectors is 0.5 radians, then 0.5 radians is determined as the sixth flexion angle.

[0111] 9) The spatial angle between the skeletal direction vector of the proximal phalanx of the ring finger and the palm reference vector is determined as the seventh flexion angle.

[0112] For example, if the skeletal direction vector of the proximal phalanx of the ring finger points towards the front of the palm, and the palm reference vector points towards the direction of finger extension, and the system calculates that the spatial angle between the two vectors is 0.3 radians, then 0.3 radians is determined as the seventh flexion angle.

[0113] 10) The spatial angle between the component of the skeletal direction vector of the proximal phalanx of the ring finger that is perpendicular to the normal vector of the palm plane and the palm reference vector is determined as the third lateral swing angle.

[0114] For example, suppose the skeletal direction vector of the proximal phalanx of the ring finger lies in a plane perpendicular to the normal vector of the palm plane, and deflects outward from the palm reference vector. The system calculates that the spatial angle between this projection component and the palm reference vector is 0.1 radians, and then determines 0.1 radians as the third lateral swing angle.

[0115] 11) The spatial angle between the skeletal direction vector of the proximal phalanx of the ring finger and the skeletal direction vector of the middle phalanx of the ring finger is determined as the eighth flexion angle.

[0116] For example, if the skeletal direction vector of the middle phalanx of the ring finger is further bent relative to the skeletal direction vector of the proximal phalanx of the ring finger, and the system calculates that the spatial angle between the two vectors is 0.5 radians, then 0.5 radians is determined as the eighth flexion angle.

[0117] 12) The spatial angle between the skeletal direction vector of the proximal phalanx of the little finger and the palm reference vector is determined as the ninth flexion angle.

[0118] For example, if the skeletal direction vector of the proximal phalanx of the little finger points to the front of the palm, and the palm reference vector points to the direction of finger extension, and the system calculates that the spatial angle between the two vectors is 0.3 radians, then 0.3 radians is determined as the ninth flexion angle.

[0119] 13) The spatial angle between the component of the skeletal direction vector of the proximal phalanx of the little finger that is perpendicular to the normal vector of the palm plane and the palm reference vector is determined as the fourth lateral swing angle.

[0120] For example, suppose the skeletal direction vector of the proximal phalanx of the little finger lies in a plane perpendicular to the normal vector of the palm plane, and is deflected outward relative to the palm reference vector. The system calculates that the spatial angle between this projection component and the palm reference vector is 0.1 radians, and then determines 0.1 radians as the fourth lateral swing angle.

[0121] 14) The spatial angle between the skeletal direction vector of the proximal phalanx of the little finger and the skeletal direction vector of the middle phalanx of the little finger is determined as the tenth flexion angle.

[0122] For example, if the skeletal direction vector of the middle phalanx of the little finger is further bent relative to the skeletal direction vector of the proximal phalanx of the little finger, and the system calculates that the spatial angle between the two vectors is 0.5 radians, then 0.5 radians is determined as the tenth flexion angle.

[0123] The advantages of this optional embodiment are as follows: Firstly, by defining the spatial angle between the skeletal direction vector of the proximal phalanx of the thumb and the palm reference vector as the first flexion angle, the spatial angle between the skeletal direction vector of the proximal phalanx of the thumb and the palm plane normal vector as the first lateral angle, and the spatial angle between the skeletal direction vector of the middle phalanx of the thumb and the skeletal direction vector of the proximal phalanx of the thumb as the second flexion angle, the motion angles of the thumb in the three directions of carpometacarpal joint flexion, carpometacarpal joint lateral angle, and metacarpophalangeal joint flexion can all be determined based on clear vector spatial relationships. This method can effectively distinguish the motion components of the thumb root joint in the flexion and lateral angle directions, avoiding the posture distortion problem caused by simplifying the complex spatial movement of the thumb into a single angle.

[0124] Secondly, for the index, ring, and little fingers, the flexion angle of the metacarpophalangeal joint is determined by the spatial angle between the skeletal direction vector of the proximal phalanx and the palm reference vector; the lateral tilt angle of the metacarpophalangeal joint is determined by the spatial angle between the component of the skeletal direction vector of the proximal phalanx perpendicular to the normal vector of the palm plane and the palm reference vector; and the flexion angle of the proximal interphalangeal joint is determined by the spatial angle between the skeletal direction vector of the proximal phalanx and the skeletal direction vector of the middle phalanx. This ensures that the three types of angles—metacarpophalangeal joint flexion, metacarpophalangeal joint lateral tilt, and proximal interphalangeal joint flexion—of each finger have independent vector angle calculation bases. This classification method effectively distinguishes the differences in movement of the root joints of the fingers in the flexion and lateral tilt directions, while decoupling the bending angle of the middle joints of the fingers from the movement of the root joints, making the determination of the joint angles more consistent with actual anatomical movement laws.

[0125] Thirdly, for the middle finger, the flexion angle of the metacarpophalangeal joint is determined by the spatial angle between the skeletal direction vector of its proximal phalanx and the palm reference vector, and the flexion angle of the proximal interphalangeal joint is determined by the spatial angle between the skeletal direction vector of its proximal phalanx and the skeletal direction vector of the middle phalanx. This ensures that the method of determining the joint angle of the middle finger can match the actual movement ability of the middle finger when it does not have a degree of lateral swing freedom, thus avoiding the introduction of unnecessary lateral swing angle calculations.

[0126] S205. Use the pose data of the bones at the base of the hand as the target pose of the upper limb end; use the target pose of the upper limb end as input, call the inverse kinematics solver to solve for the target angle sequence of each upper limb joint that satisfies the joint limit constraint, and use it as the upper limb joint angle reference trajectory.

[0127] Among them, the target pose of the upper limb end effector refers to using the pose data of the bones at the base of the hand as the desired position and posture of the upper limb end effector of the humanoid robot to be controlled in space. The target pose of the upper limb end effector provides a spatial reference benchmark for the subsequent calculation of the target angles of each joint of the upper limb.

[0128] An inverse kinematics solver is a module used to solve for the target angles of each joint of the upper limb of a humanoid robot under control, based on the target pose of the upper limb end-effector. The inverse kinematics solver is based on the kinematic model of the upper limb of the humanoid robot under control, and outputs the combination of joint angles that enable the upper limb end-effector to reach that pose, given the end-effector pose. Optionally, the inverse kinematics solver can be the Drake inverse kinematics solver, the Drake inverse kinematics solver employing a two-stage solution strategy, or other inverse kinematics solvers based on numerical iteration.

[0129] Joint constraint refers to the limitation on the range of motion angles allowed for each upper limb joint of the humanoid robot to be controlled in terms of mechanical structure. The joint constraint specifies the minimum and maximum angle values ​​that each upper limb joint can reach. During the solution process, the inverse kinematics solver must ensure that the output combination of joint angles does not exceed the range limited by the joint constraint.

[0130] A target angle sequence refers to serialized data consisting of target angle values ​​for each upper limb joint at multiple consecutive time points. The target angle sequence describes the desired motion path of the humanoid robot under control as the angles of each upper limb joint change over time, serving as a specific representation of the upper limb joint angle reference trajectory.

[0131] In one implementation, after acquiring the pose data of the bones at the base of the hand of the target operator, the system directly uses this pose data as the target pose of the upper limb end effector of the humanoid robot to be controlled. This target pose of the upper limb end effector includes the target position coordinates and target orientation information of the upper limb end effector in space.

[0132] The system takes the target pose of the upper limb end-effector as input and calls the inverse kinematics solver for solution. Based on the kinematic model of the humanoid robot's upper limb, the inverse kinematics solver calculates the combinations of joint angles that allow the upper limb end-effector to reach the target pose, according to the current angles of each joint, joint connection relationships, and joint range of motion. During the solution process, the inverse kinematics solver applies joint constraint limits to the search range of each joint angle, ensuring that the output joint angle values ​​do not exceed the minimum and maximum angle range allowed by the mechanical structure of each upper limb joint.

[0133] The upper limb joint angle values ​​obtained by the inverse kinematics solver are arranged in chronological order to form a target angle sequence. This target angle sequence describes the expected joint angle values ​​of each upper limb joint at consecutive time points. The system uses this as a reference trajectory for upper limb joint angles for the generation of subsequent joint drive commands.

[0134] For example, suppose the pose data of the target operator's palm base bones are: position coordinates (0.4, 0.2, 0.3), in meters; the orientation is represented by quaternions (0.0, 0.0, 0.0, 1.0), corresponding to the palm plane facing forward.

[0135] The system uses this pose data as the target pose of the upper limb end effector of the humanoid robot to be controlled and inputs it into the inverse kinematics solver. Based on the joint configuration and kinematic model of the upper limb of the humanoid robot to be controlled, the inverse kinematics solver solves within the allowable range of motion of each upper limb joint, such as the shoulder, elbow, and wrist joints, to obtain a set of joint angle combinations that satisfy the target pose of the end effector. For example, the solution yields a shoulder flexion angle of 0.3 radians, a shoulder abduction angle of 0.1 radians, an elbow flexion angle of 0.8 radians, and a wrist flexion angle of 0.2 radians.

[0136] The above solution process is executed within each control cycle. As the pose data of the target operator's hand root bones changes, the inverse kinematics solver continuously outputs the corresponding upper limb joint angle combinations at each moment. Taking the elbow joint as an example, the target angles obtained in three consecutive control cycles are 0.80 radians, 0.82 radians, and 0.84 radians, respectively. This sequence is the target angle sequence of the elbow joint. The target angle sequences of each upper limb joint together constitute the upper limb joint angle reference trajectory.

[0137] The advantages of this optional embodiment are as follows: Firstly, by directly using the pose data of the bones at the base of the hand as the target pose of the upper limb, a direct mapping relationship is established between the spatial position and posture of the target operator's hand and the desired pose of the upper limb of the humanoid robot to be controlled. This method enables the target pose of the upper limb to follow the actual movement of the base of the target operator's hand in real time, without the need for complex intermediate coordinate transformations or pose calculations. This ensures the synchronization and consistency between the target pose of the upper limb and the posture of the target operator's hand, providing an accurate and real-time spatial reference benchmark for the subsequent generation of upper limb joint angle reference trajectories.

[0138] Secondly, by using the target pose of the upper limb end-effector as input, an inverse kinematics solver is invoked to solve the problem, transforming the tracking problem of the upper limb end-effector pose into the calculation problem of the angles of each joint of the upper limb. The inverse kinematics solver can automatically find the joint angle combination that meets the end-effector pose requirements within the upper limb joint space based on the kinematic model of the humanoid robot to be controlled, avoiding the insufficient adaptability problems caused by manual calibration or preset fixed trajectories. This solution method enables the generation of upper limb joint angle reference trajectories to adapt to the continuous changes in the hand posture of the target operator, improving the flexibility and real-time response capability of upper limb motion control.

[0139] Thirdly, by applying joint constraint during the inverse kinematics solution process, the obtained target angle sequence of each upper limb joint is always within the allowable range of motion of each joint. This constraint mechanism can effectively avoid joint over-limit commands caused by excessive movement of the target operator's palm or the joint angle combination output by the inverse kinematics solver exceeding the mechanical limit, thus ensuring the motion safety and structural integrity of the upper limb joints of the humanoid robot to be controlled.

[0140] S206. The target finger joint angle is used as the desired position constraint of the dexterous hand, and the upper limb joint angle reference trajectory is used as the desired trajectory constraint of the upper limb. The desired position constraint of the dexterous hand and the desired trajectory constraint of the upper limb are input into the whole body coordination controller. The whole body coordination controller performs a unified solution under the whole body dynamics constraint and outputs the upper limb joint drive command and the dexterous hand joint drive command in parallel.

[0141] Among them, the dexterous hand expected position constraint refers to using the target finger joint angle as the expected position target of each finger joint of the dexterous hand on the humanoid robot to be controlled during the control process. The dexterous hand expected position constraint specifies the position conditions that each finger joint of the dexterous hand should meet when solving for joint drive commands, and serves as one of the input constraints when the whole-body coordination controller performs a unified solution.

[0142] Upper limb desired trajectory constraint refers to using the reference trajectory of upper limb joint angles as the expected motion trajectory that each upper limb joint of the humanoid robot to be controlled is expected to follow during the control process. The upper limb desired trajectory constraint specifies the angle values ​​that each joint of the upper limb should reach at consecutive time points, and serves as one of the input constraints when the whole-body coordination controller performs a unified solution.

[0143] A full-body coordination controller is a control module used to coordinate the control of the upper limb joints and dexterous finger joints of the humanoid robot to be controlled. The full-body coordination controller receives the desired position constraints of the dexterous hand and the desired trajectory constraints of the upper limb as inputs, and, based on the overall kinematics and dynamics of the humanoid robot to be controlled, generates unified drive commands for the upper limb joints and the dexterous hand joints.

[0144] Whole-body dynamic constraints refer to the whole-body dynamic limitations imposed on the humanoid robot being controlled during its movement. These constraints may include limits on the driving torque of each joint, limits on joint movement speed, overall center of mass equilibrium conditions, and foot contact force constraints. When solving for joint drive commands, the whole-body coordination controller must ensure that the output commands meet the requirements of the whole-body dynamic constraints.

[0145] Upper limb joint drive commands refer to the instruction information generated by the whole-body coordinator to directly drive the upper limb joints of the humanoid robot under control to perform movements. Upper limb joint drive commands may include control parameters such as target position, target velocity, or target torque of each upper limb joint, and are used to control the upper limbs of the humanoid robot under control to move according to the desired trajectory constraints.

[0146] Dexterous hand joint drive commands refer to the instruction information generated by the whole-body coordinator to directly drive the movement of each finger joint of the dexterous hand on the humanoid robot being controlled. Dexterous hand joint drive commands may include control parameters such as target position, target velocity, or target torque of each finger joint of the dexterous hand, used to control the dexterous hand to perform grasping actions according to the desired position constraints of the dexterous hand.

[0147] In one implementation, after obtaining the target finger joint angles of the dexterous hand and the upper limb joint angle reference trajectories of each upper limb joint, the system uses the target finger joint angles as the desired position constraints of the dexterous hand, that is, it specifies the target angle values ​​that each finger joint of the dexterous hand should reach or approach within the control cycle; and uses the upper limb joint angle reference trajectories as the desired trajectory constraints of the upper limb, that is, it specifies the target angle sequence that each upper limb joint of the humanoid robot to be controlled should follow at continuous time nodes.

[0148] The system inputs the desired position constraints of the dexterous hand and the desired trajectory constraints of the upper limbs into the whole-body coordination controller. Based on the whole-body kinematic and dynamic models of the humanoid robot to be controlled, the whole-body coordination controller constructs a unified optimization problem. In this optimization problem, the optimization objectives include minimizing the deviation between the actual angles of each finger joint of the dexterous hand and the desired position constraints of the dexterous hand, and minimizing the deviation between the actual angles of each joint of the upper limb and the desired trajectory constraints of the upper limb. Simultaneously, the whole-body coordination controller applies whole-body dynamic constraints during the solution process. These constraints may include upper limits for the driving torque of each joint, upper limits for joint motion velocity, whole-body center-of-mass equilibrium conditions, and end-effector contact force constraints, to ensure the physical feasibility of the solution.

[0149] Under the combined influence of the aforementioned optimization objectives and whole-body dynamic constraints, the whole-body coordination controller performs a unified solution, simultaneously obtaining the drive commands for each joint of the upper limbs and each joint of the dexterous hand of the humanoid robot to be controlled. The drive commands for the upper limb joints and the dexterous hand joints coordinate with each other during the solution process. For example, when the upper limbs perform rapid movements, the whole-body coordination controller can adjust the drive commands for the dexterous hand joints accordingly to compensate for the influence of inertial forces on grasping stability; when the dexterous hand performs precise grasping, the whole-body coordination controller can constrain the movement speed of the upper limb extremities to ensure grasping accuracy.

[0150] Finally, the whole-body coordination controller outputs upper limb joint drive commands and dexterous hand joint drive commands in parallel, which are used to drive the upper limb joint actuators and dexterous finger joint actuators of the humanoid robot to be controlled, respectively, to achieve synchronous and coordinated movement of upper limb posture following and dexterous hand grasping configuration.

[0151] The advantages of this optional embodiment are as follows: Firstly, by using the target finger joint angles as constraints on the desired position of the dexterous hand and the reference trajectory of the upper limb joint angles as constraints on the desired trajectory of the upper limb, and inputting both into the whole-body coordination controller, the grasping configuration target of the dexterous hand and the motion trajectory target of the upper limb can be uniformly expressed within the same control framework. Compared to processing dexterous hand control and upper limb control as independent channels, this unified input mechanism allows the whole-body coordination controller to simultaneously perceive the desired states of both the dexterous hand and the upper limb, providing an information basis for their collaborative solution.

[0152] Secondly, by using a whole-body coordination controller to solve the problem uniformly under the constraints of whole-body dynamics, the generation process of upper limb joint drive commands and dexterous hand joint drive commands can take into account the overall dynamic characteristics of the humanoid robot to be controlled, and achieve mutual coordination at the dynamic level, rather than simply superimposing them at the kinematic level.

[0153] Thirdly, by outputting upper limb joint drive commands and dexterous hand joint drive commands in parallel, the upper limb joints and dexterous finger joints of the humanoid robot under control can synchronously receive and execute their respective control commands within the same control cycle. This parallel output mechanism ensures the temporal consistency of upper limb posture adjustment and grasping configuration changes, avoiding motion lag and incoherence caused by serial processing or time-division output, making the robot's overall movements more consistent with the natural coordinated movement patterns of the human upper limbs and hands.

[0154] Optionally, the method also includes: In response to the first control command, switch to fixed hand mode; in fixed hand mode, use each fixed finger joint angle in the preset set of fixed finger joint angles as the target finger joint angles of the dexterous hand.

[0155] The first control command refers to a control signal triggered by the target operator through a preset interaction method, used to instruct the system to switch from gesture following mode to fixed hand shape mode. The first control command may originate from interactive operations such as pressing a specific button, issuing a voice command, or performing a preset gesture by the target operator, and its function is to trigger the system to switch control modes.

[0156] Fixed hand mode refers to a specific control mode for the dexterous hand on a humanoid robot. In fixed hand mode, the angles of the target finger joints of the dexterous hand are no longer calculated in real time from the three-dimensional position data of the key points of the target operator's hand bones, but are directly taken from a preset set of fixed finger joint angles, so that the dexterous hand maintains a certain preset grasping configuration.

[0157] A fixed knuckle angle set refers to a pre-defined set of parameters containing the angle values ​​corresponding to each finger joint of a dexterous hand in a specific grasping posture. The fixed knuckle angle set stores fixed knuckle angle values ​​that correspond one-to-one with each finger joint of a dexterous hand.

[0158] Fixed knuckle angles refer to the preset knuckle angle values ​​for a specific finger joint in a dexterous hand, as defined in the set of fixed knuckle angles. These fixed knuckle angles are pre-calibrated values ​​that are directly output as the target knuckle angle for the corresponding finger joint in fixed hand mode.

[0159] In one implementation, the system defaults to gesture following mode. In this mode, the system calculates the angles of each target finger joint of the dexterous hand in real time based on the three-dimensional position data of the key points of the target operator's hand bones, so that the grasping configuration of the dexterous hand continuously changes with the hand posture of the target operator.

[0160] When the system receives the first control command, it switches from the current gesture-following mode to the fixed hand shape mode. The first control command can be triggered by the target operator by pressing a preset button, and the system obtains the command by listening to the target operator's key input events.

[0161] Upon entering the fixed hand mode, the system freezes the current real-time calculation process based on key points of the hand skeleton, and no longer updates the target finger joint angles of the dexterous hand based on changes in the operator's hand posture. Instead, the system reads pre-stored fixed finger joint angle values ​​from a preset set of fixed finger joint angles, and uses each read fixed finger joint angle value as the target finger joint angle for the corresponding finger joint of the dexterous hand. The set of fixed finger joint angles can be pre-stored in the system's configuration parameters or data file, containing the angle values ​​of each finger joint of the dexterous hand in a specific grasping posture.

[0162] The system outputs the aforementioned target knuckle angles to the subsequent instruction generation stage, which generates joint drive commands to control the movement of the dexterous hand. In this mode, regardless of how the operator's hand posture changes, the angles of each target knuckle of the dexterous hand remain at the values ​​defined by a fixed set of knuckle angles, thus enabling the dexterous hand to maintain a stable preset grasping configuration.

[0163] When the system receives an exit command triggered by the target operator again or meets the preset exit conditions, the system can switch from the fixed hand shape mode back to the gesture follow mode and resume real-time mapping control based on the target operator's hand posture.

[0164] The advantages of this optional embodiment are as follows: Firstly, by switching to a fixed hand form mode in response to a first control command, the system can flexibly switch between gesture following mode and fixed hand form mode. When the target operator needs the dexterous hand to maintain a specific grasping configuration without following changes in hand posture, the system can quickly enter the fixed hand form mode by triggering the first control command. This avoids instability in the dexterous hand's grasping configuration due to hand tremors or posture drift during gesture tracking, thus improving the dexterous hand's posture maintenance ability when performing stable grasping tasks.

[0165] Secondly, by using the fixed finger joint angles from a preset set of fixed finger joint angles as the target finger joint angles of the dexterous hand in the fixed hand form mode, the dexterous hand can directly reproduce the pre-calibrated grasping configuration. Since the angle values ​​in the fixed finger joint angle set are preset or obtained through offline calibration, the grasping configuration formed by them is deterministic and repeatable. The dexterous hand can output the same target finger joint angles every time it enters the fixed hand form mode, ensuring the consistency and reproducibility of the grasping posture.

[0166] Thirdly, by triggering mode switching through the first control command, the target operator can complete the control mode transition without interrupting the teleoperation process or performing complex parameter settings. The target operator only needs to perform simple interactive actions such as pressing buttons to switch from gesture following to fixed grasping configuration, which is convenient and responsive, and helps to improve the continuity and efficiency of teleoperation.

[0167] Optionally, each fixed finger joint angle in a preset set of fixed finger joint angles can be used as a target finger joint angle for the dexterous hand, including: 1) Based on the fixed finger joint angles of one dexterous hand in the set of fixed finger joint angles, determine the target finger joint angle of the corresponding finger joint of one dexterous hand.

[0168] One of the dexterous fingers is one of the two dexterous hands on the humanoid robot to be controlled. The dexterous hand on one side serves as the source of angle information, and its corresponding fixed finger joint angle is used to determine the target finger joint angle of that dexterous hand, while also serving as a reference for generating the fixed finger joint angle of the other dexterous hand.

[0169] The other dexterous hand refers to the other dexterous hand on the humanoid robot, distinct from the other dexterous hand on the other side. The fixed finger joint angles of the other dexterous hand are generated by mirroring the fixed finger joint angles of the dexterous hand on the other side, and serve as the target finger joint angles for the corresponding finger joints of the other dexterous hand.

[0170] In one implementation, after entering the fixed hand mode, the system reads the angle values ​​of each fixed finger joint from a preset set of fixed finger joint angles. The set of fixed finger joint angles may pre-store the angle parameters of each finger joint of the dexterous hand in a specific grasping posture. For the corresponding finger joints of the left and right dexterous hands, independent fixed finger joint angle values ​​may be stored separately, or only the fixed finger joint angle values ​​of one dexterous hand may be stored while the other side is obtained through symmetrical generation.

[0171] For each finger joint of a dexterous hand that is explicitly stored in the set of fixed finger joint angles, the system directly determines the read fixed finger joint angle value as the target finger joint angle of the corresponding finger joint of that dexterous hand. For example, if the set of fixed finger joint angles stores the fixed finger joint angle value of the metacarpophalangeal joint of the index finger of the right dexterous hand, the system directly outputs this angle value as the target finger joint angle of the metacarpophalangeal joint of the index finger of the right dexterous hand.

[0172] For the finger joint of the dexterous hand on the other side whose fixed angle value is not directly stored in the set of fixed finger joint angles, the system generates the fixed finger joint angle of the corresponding finger joint of the dexterous hand on the other side according to the mirror symmetry rule based on the fixed finger joint angle of the stored dexterous hand on the other side, and uses the generated angle value as the target finger joint angle of the corresponding finger joint of the dexterous hand on the other side.

[0173] 2) According to the mirror symmetry rule, generate the fixed finger joint angles of the corresponding finger joints of the dexterous hand on the other side based on the fixed finger joint angles of the dexterous hand on one side, and use the generated fixed finger joint angles as the target finger joint angles of the corresponding finger joints of the dexterous hand on the other side.

[0174] The mirror symmetry rule refers to a conversion rule that uses the midsagittal plane of the humanoid robot to be controlled as a symmetry reference plane, and converts the fixed joint angle values ​​of a certain finger joint of one dexterous hand into the corresponding fixed joint angle values ​​of the other dexterous hand according to a preset sign mapping relationship. The mirror symmetry rule specifies a mapping method in which the angle values ​​in the flexion direction remain unchanged, while the angle values ​​in the lateral swing direction take the opposite sign, so that the angles of each finger joint of the generated other dexterous hand are mirror-symmetric with the angles of each finger joint of the one dexterous hand in the direction of spatial motion.

[0175] In one implementation, the system generates the fixed joint angles of the corresponding joints of the dexterous hand on the other side according to the mirror symmetry rule, based on the fixed joint angles of each joint of the dexterous hand on one side that have been obtained.

[0176] Specifically, for the finger joint angles in the flexion direction, the system directly uses the fixed finger joint angle value of one dexterous hand as the fixed finger joint angle value of the corresponding finger joint of the other dexterous hand. For example, the flexion angle value of the metacarpophalangeal joint of the index finger of the right dexterous hand is directly assigned to the flexion angle of the metacarpophalangeal joint of the index finger of the left dexterous hand, so that the corresponding finger joints of the left and right hands are consistent in the degree of flexion.

[0177] For the lateral tilt angle of the finger joints, the system reverses the sign of the fixed finger joint angle value of one dexterous hand and uses it as the fixed finger joint angle value of the corresponding finger joint of the other dexterous hand. For example, if the lateral tilt angle value of the metacarpophalangeal joint of the index finger of the right dexterous hand is positive 0.1 radians (indicating a tilt towards the outside of the index finger), the system reverses this value to obtain a negative 0.1 radians, which is then used as the lateral tilt angle value of the metacarpophalangeal joint of the index finger of the left dexterous hand, causing the corresponding finger joint of the left dexterous hand to tilt in the opposite symmetrical direction. For the lateral tilt angle of the thumb metacarpophalangeal joint, the system uses the same reversal rule for mapping to ensure that the movement of the left and right thumbs in the direction of the opening and closing of the web of the thumb is mirror-symmetrical.

[0178] After generating the fixed joint angles of each corresponding finger joint in the dexterous hand on the other side, the system uses the generated angle values ​​as the target joint angles of the corresponding finger joints in the dexterous hand on the other side, which are then used to generate subsequent joint drive commands.

[0179] The advantages of this optional embodiment are as follows: Firstly, by determining the target finger joint angles of the corresponding finger joints of one dexterous hand based on the fixed finger joint angles of one side of the fixed finger joint angle set, the configuration of each target finger joint angle of that dexterous hand can be completed by only storing the fixed finger joint angle parameters of one side when setting the fixed finger joint angle set. This method reduces the amount of parameters required to store for the fixed finger joint angle set and reduces the workload of parameter configuration for the preset grasping configuration.

[0180] Secondly, through the automatic generation mechanism of mirror symmetry rules, the grasping configurations output by the left and right dexterous hands in fixed hand shape modes are spatially mirror-symmetrical. When the humanoid robot to be controlled performs a two-handed collaborative grasping task, the left and right dexterous hands can act together on the grasped object in a symmetrical posture, making the grasping force distribution more balanced and helping to improve the stability and reliability of two-handed collaborative grasping. At the same time, the target operator only needs to focus on the grasping configuration calibration of one side of the dexterous hand to obtain a bilaterally symmetrical grasping effect, reducing the complexity of operation.

[0181] Optionally, the method also includes: In the fixed hand mode, in response to the second control command, the fixed finger joint angles are cyclically rotated between at least two preset fixed finger joint angle sets, and each fixed finger joint angle in the rotated fixed finger joint angle set is used as the target finger joint angle of the dexterous hand.

[0182] The second control command refers to a control signal triggered by the target operator through a preset interaction method, used to instruct the system to switch from the current fixed hand shape mode to the next preset fixed knuckle angle set. The second control command can originate from interactive operations such as the target operator pressing a specific button, issuing a voice command, or performing a preset gesture, and its function is to trigger the system to switch between multiple fixed knuckle angle sets.

[0183] Cyclic switching refers to a system that, in a fixed hand position mode, responds to a second control command and sequentially switches from the current set of fixed knuckle angles to the next set of fixed knuckle angles according to a preset order, returning to the first set of fixed knuckle angles after traversing all preset sets of fixed knuckle angles. Cyclic switching allows the operator to quickly switch between multiple preset grasping configurations by continuously triggering the second control command, without needing to remember or specify the number of a particular configuration.

[0184] In one implementation, after entering the fixed hand mode, in addition to using the fixed finger joint angles from the currently preset set of fixed finger joint angles as the target finger joint angles of the dexterous hand, the system continuously monitors whether the target operator has triggered a second control command. The second control command can be triggered by the target operator by pressing a preset button, and the system obtains the command by monitoring the target operator's button input events.

[0185] The system pre-stores at least two sets of fixed knuckle angles, each corresponding to a preset grasping configuration of a dexterous hand. For example, the first set of fixed knuckle angles can be preset to correspond to a "strong grip" configuration, the second set to correspond to a "precise pinch" configuration, and the third set to correspond to a "side pinch" configuration, etc. The fixed knuckle angle sets form a rotation sequence according to a preset arrangement order.

[0186] When the system receives a second control command in fixed hand mode, it switches the currently effective set of fixed joint angles to the next set of fixed joint angles in the rotation sequence, and uses the fixed joint angle values ​​in the switched set as the target joint angles for the corresponding joints of the dexterous hand. If the currently effective set of fixed joint angles is the last set in the rotation sequence, the system switches to the first set of fixed joint angles in the rotation sequence upon receiving the second control command, thus achieving cyclic rotation.

[0187] The system outputs the updated target knuckle angles to the subsequent instruction generation stage, which generates joint drive commands to control the movement of the dexterous hand. The operator can quickly cycle through multiple preset grasping configurations by continuously triggering the second control command until the grasping configuration required for the current task is selected.

[0188] For example, suppose the system pre-stores three fixed knuckle angle sets, which are set one, set two, and set three in rotation order. The system is currently in a fixed hand shape mode, and set one is currently in effect, with the dexterous hand exhibiting the grasping configuration corresponding to this set. When the target operator triggers the second control command for the first time, the system switches to set two; when the second control command is triggered, the system switches to set three; when the second control command is triggered for the third time, the system switches back to set one, completing one full cycle.

[0189] The advantages of this optional embodiment are as follows: Firstly, by responding to a second control command in a fixed hand position mode and switching between at least two preset sets of fixed knuckle angles, the target operator can change the preset grasping configuration of the dexterous hand without exiting the fixed hand position mode. Compared to the method of needing to exit the current mode and reselect other fixed configurations, this solution simplifies the operation process and improves the convenience of grasping configuration switching.

[0190] Secondly, by using a cyclical switching method, the operator can sequentially traverse between multiple preset fixed knuckle angle sets by continuously triggering the same second control command. The operator does not need to memorize different control commands or numbers corresponding to different configurations; they only need to repeat the same operation to cycle through all preset configurations, reducing the burden of memorizing operations and improving the intuitiveness of the interaction.

[0191] Thirdly, by using the fixed finger joint angles in the rotated set of fixed finger joint angles as the target finger joint angles of the dexterous hand, the dexterous hand can immediately switch to the corresponding preset grasping configuration after each rotation. This instant response mechanism allows the target operator to quickly try different grasping configurations to adapt to the shape and size of the object being grasped, improving the efficiency and adaptability of grasping task execution.

[0192] Example 3 Figure 3 This is a schematic diagram of a control device for a humanoid robot provided in Embodiment 3 of the present invention. It is applicable to scenarios involving remote operation control of humanoid robots, particularly those requiring mapping the operator's hand posture to a dexterous hand grasping configuration and coordinating upper limb movements with grasping actions. Figure 3 As shown, the device includes: The data acquisition module 31 is used to acquire the three-dimensional position data of the key points of the hand bones of the target operator, as well as the pose data of the base of the palm bones of the target operator. The target finger joint angle determination module 32 is used to determine the skeletal direction vector of each finger joint of the target operator based on the three-dimensional position data, and to determine the angle of each target finger joint of the dexterous hand on the humanoid robot to be controlled based on the skeletal direction vector. The joint drive command generation module 33 is used to generate upper limb joint angle reference trajectories for each upper limb joint of the humanoid robot to be controlled based on the pose data, and to generate joint drive commands for controlling the movement of the upper limbs and the dexterous hand of the humanoid robot to be controlled based on the target finger joint angle and the upper limb joint angle reference trajectory.

[0193] Optionally, the target knuckle angle determination module 32 is specifically used for: For each finger of the target operator, the three-dimensional position data of adjacent key points of the hand bones are subjected to vector difference operation to obtain the bone direction vector of the corresponding finger joint.

[0194] Optionally, the target knuckle angle determination module 32 is further used for: Determine the key points of the hand bones from the key points of the hand bones, and construct the hand reference vector and the hand plane normal vector based on the key points of the hand bones. Based on the spatial angle between the bone direction vectors and the spatial angle between the bone direction vectors and the palm reference vector or the palm plane normal vector, the angles of each finger joint of the target operator's hand are determined. Based on the angles of each finger joint in the target operator's hand, the angles of each target finger joint in the dexterous hand of the humanoid robot to be controlled are determined through parameter transformation.

[0195] Optionally, the parameter transformation includes applying at least one of coefficient scaling, orientation flipping, and offset correction to the angles of each finger joint of the target operator's hand.

[0196] Optionally, the angles of each finger joint of the target operator's hand include: The first flexion angle of the thumb carpometacarpal joint, the first lateral displacement angle of the thumb carpometacarpal joint, and the second flexion angle of the thumb metacarpophalangeal joint; The third flexion angle of the metacarpophalangeal joint of the index finger, the second lateral swing angle of the metacarpophalangeal joint of the index finger, and the fourth flexion angle of the proximal interphalangeal joint of the index finger. The fifth flexion angle of the metacarpophalangeal joint of the middle finger and the sixth flexion angle of the proximal interphalangeal joint of the middle finger; The seventh flexion angle of the metacarpophalangeal joint of the ring finger, the third lateral swing angle of the metacarpophalangeal joint of the ring finger, and the eighth flexion angle of the proximal interphalangeal joint of the ring finger. The ninth flexion angle of the metacarpophalangeal joint of the little finger, the fourth lateral swing angle of the metacarpophalangeal joint of the little finger, and the tenth flexion angle of the proximal interphalangeal joint of the little finger.

[0197] Optionally, the target knuckle angle determination module 32 is further used for: The spatial angle between the skeletal direction vector of the proximal phalanx of the thumb and the palm reference vector is determined as the first flexion angle; The spatial angle between the skeletal direction vector of the proximal phalanx of the thumb and the normal vector of the palm plane is determined as the first lateral swing angle; The spatial angle between the skeletal direction vector of the middle phalanx of the thumb and the skeletal direction vector of the proximal phalanx of the thumb is determined as the second flexion angle; The spatial angle between the skeletal direction vector of the proximal phalanx of the index finger and the palm reference vector is determined as the third flexion angle; The spatial angle between the component of the skeletal direction vector of the proximal phalanx of the index finger that is perpendicular to the normal vector of the palm plane and the palm reference vector is determined as the second lateral swing angle. The spatial angle between the skeletal direction vector of the proximal phalanx of the index finger and the skeletal direction vector of the middle phalanx of the index finger is determined as the fourth flexion angle. The spatial angle between the skeletal direction vector of the proximal phalanx of the middle finger and the palm reference vector is determined as the fifth flexion angle; The spatial angle between the skeletal direction vector of the proximal phalanx of the middle finger and the skeletal direction vector of the middle phalanx of the middle finger is determined as the sixth flexion angle. The spatial angle between the skeletal direction vector of the proximal phalanx of the ring finger and the palm reference vector is determined as the seventh flexion angle; The spatial angle between the component of the skeletal direction vector of the proximal phalanx of the ring finger that is perpendicular to the normal vector of the palm plane and the palm reference vector is determined as the third lateral swing angle. The spatial angle between the skeletal direction vector of the proximal phalanx of the ring finger and the skeletal direction vector of the middle phalanx of the ring finger is defined as the eighth flexion angle. The spatial angle between the skeletal direction vector of the proximal phalanx of the little finger and the palm reference vector is determined as the ninth flexion angle; The spatial angle between the component of the skeletal direction vector of the proximal phalanx of the little finger that is perpendicular to the normal vector of the palm plane and the palm reference vector is determined as the fourth lateral swing angle. The spatial angle between the skeletal direction vector of the proximal phalanx of the little finger and the skeletal direction vector of the middle phalanx of the little finger is defined as the tenth flexion angle.

[0198] Optionally, the joint drive command generation module 33 is specifically used for: The pose data of the bones at the base of the palm are used as the target pose of the upper limb end. Using the target pose of the upper limb end as input, the inverse kinematics solver is called to solve the problem and obtain the target angle sequence of each upper limb joint that satisfies the joint limit constraint, which is used as the reference trajectory of the upper limb joint angle.

[0199] Optionally, the joint drive command generation module 33 is specifically used for: The target finger joint angle is used as the desired position constraint for the dexterous hand, and the upper limb joint angle reference trajectory is used as the desired trajectory constraint for the upper limb. The desired position constraint of the dexterous hand and the desired trajectory constraint of the upper limb are input together into the whole body coordination controller; The whole-body coordination controller performs a unified solution under the constraints of whole-body dynamics and outputs upper limb joint drive commands and dexterous hand joint drive commands in parallel.

[0200] Optionally, the device further includes a first control command response module, specifically used for: In response to the first control command, switch to fixed hand mode; In the fixed hand shape mode, each fixed finger joint angle in the preset set of fixed finger joint angles is used as the target finger joint angle of the dexterous hand.

[0201] Optionally, the device further includes a second control command response module, specifically used for: In the fixed hand mode, in response to the second control command, the fixed finger joint angles are cyclically rotated between at least two preset fixed finger joint angle sets, and each fixed finger joint angle in the rotated fixed finger joint angle set is taken as the target finger joint angle of the dexterous hand.

[0202] Optionally, the device further includes a knuckle angle mapping module, specifically used for: Based on the fixed finger joint angles of one dexterous hand in the set of fixed finger joint angles, determine the target finger joint angle of the corresponding finger joint of the dexterous hand on that side; According to the mirror symmetry rule, the fixed finger joint angles of the corresponding finger joints of the dexterous hand on the other side are generated based on the fixed finger joint angles of the dexterous hand on the other side, and the generated fixed finger joint angles are used as the target finger joint angles of the corresponding finger joints of the dexterous hand on the other side.

[0203] The control device for the humanoid robot provided in this embodiment of the invention can execute the control method for the humanoid robot provided in this embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0204] According to embodiments of this disclosure, embodiments of the present invention also provide an electronic device, a readable storage medium, and a computer program product.

[0205] Figure 4 A schematic diagram of an electronic device 40 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0206] like Figure 4 As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded from storage unit 48 into the RAM 43. The RAM 43 may also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0207] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0208] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as control methods for humanoid robots.

[0209] In some embodiments, the control method for the humanoid robot can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 58. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 50 via ROM 52 and / or communication unit 59. When the computer program is loaded into RAM 53 and executed by processor 51, one or more steps of the control method for the humanoid robot described above can be performed. Alternatively, in other embodiments, processor 51 can be configured to perform the control method for the humanoid robot by any other suitable means (e.g., by means of firmware).

[0210] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0211] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0212] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0213] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0214] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0215] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system to address the shortcomings of traditional physical hosts and virtual private servers, such as high management difficulty and weak business scalability.

[0216] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0217] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A control method for a humanoid robot, characterized in that, The method includes: Acquire the three-dimensional position data of key points of the hand bones of the target operator, as well as the pose data of the bones at the base of the palm of the target operator; The skeletal direction vectors of each finger joint of the target operator are determined based on the three-dimensional position data, and the angles of each target finger joint of the dexterous hand on the humanoid robot to be controlled are determined based on the skeletal direction vectors. Based on the pose data, reference trajectories of the upper limb joint angles of the humanoid robot to be controlled are generated. Based on the target finger joint angles and the upper limb joint angle reference trajectories, joint drive commands for controlling the movement of the upper limbs and the dexterous hand of the humanoid robot to be controlled are generated.

2. The method according to claim 1, characterized in that, Determining the skeletal orientation vectors of each finger joint of the target operator based on the three-dimensional position data includes: For each finger of the target operator, the three-dimensional position data of adjacent key points of the hand bones are subjected to vector difference operation to obtain the bone direction vector of the corresponding finger joint.

3. The method according to claim 1, characterized in that, The step of determining the angles of each target finger joint of the dexterous hand on the humanoid robot to be controlled based on the skeletal direction vector includes: Determine the key points of the hand bones from the key points of the hand bones, and construct the hand reference vector and the hand plane normal vector based on the key points of the hand bones. Based on the spatial angle between the bone direction vectors and the spatial angle between the bone direction vectors and the palm reference vector or the palm plane normal vector, the angles of each finger joint of the target operator's hand are determined. Based on the angles of each finger joint in the target operator's hand, the angles of each target finger joint in the dexterous hand of the humanoid robot to be controlled are determined through parameter transformation.

4. The method according to claim 3, characterized in that, The parameter transformation includes applying at least one of the following to the angles of each finger joint of the target operator's hand: coefficient scaling, direction flipping, and offset correction.

5. The method according to claim 3, characterized in that, The angles of each finger joint in the target operator's hand include: The first flexion angle of the thumb carpometacarpal joint, the first lateral displacement angle of the thumb carpometacarpal joint, and the second flexion angle of the thumb metacarpophalangeal joint; The third flexion angle of the metacarpophalangeal joint of the index finger, the second lateral swing angle of the metacarpophalangeal joint of the index finger, and the fourth flexion angle of the proximal interphalangeal joint of the index finger. The fifth flexion angle of the metacarpophalangeal joint of the middle finger and the sixth flexion angle of the proximal interphalangeal joint of the middle finger; The seventh flexion angle of the metacarpophalangeal joint of the ring finger, the third lateral swing angle of the metacarpophalangeal joint of the ring finger, and the eighth flexion angle of the proximal interphalangeal joint of the ring finger. The ninth flexion angle of the metacarpophalangeal joint of the little finger, the fourth lateral swing angle of the metacarpophalangeal joint of the little finger, and the tenth flexion angle of the proximal interphalangeal joint of the little finger.

6. The method according to claim 5, characterized in that, The step of determining the angles of each finger joint of the target operator's hand based on the spatial angles between the bone direction vectors and the spatial angles between the bone direction vectors and the palm reference vector or the palm plane normal vector includes: The spatial angle between the skeletal direction vector of the proximal phalanx of the thumb and the palm reference vector is determined as the first flexion angle; The spatial angle between the skeletal direction vector of the proximal phalanx of the thumb and the normal vector of the palm plane is determined as the first lateral swing angle; The spatial angle between the skeletal direction vector of the middle phalanx of the thumb and the skeletal direction vector of the proximal phalanx of the thumb is determined as the second flexion angle; The spatial angle between the skeletal direction vector of the proximal phalanx of the index finger and the palm reference vector is determined as the third flexion angle; The spatial angle between the component of the skeletal direction vector of the proximal phalanx of the index finger that is perpendicular to the normal vector of the palm plane and the palm reference vector is determined as the second lateral swing angle. The spatial angle between the skeletal direction vector of the proximal phalanx of the index finger and the skeletal direction vector of the middle phalanx of the index finger is determined as the fourth flexion angle. The spatial angle between the skeletal direction vector of the proximal phalanx of the middle finger and the palm reference vector is determined as the fifth flexion angle; The spatial angle between the skeletal direction vector of the proximal phalanx of the middle finger and the skeletal direction vector of the middle phalanx of the middle finger is determined as the sixth flexion angle. The spatial angle between the skeletal direction vector of the proximal phalanx of the ring finger and the palm reference vector is determined as the seventh flexion angle; The spatial angle between the component of the skeletal direction vector of the proximal phalanx of the ring finger that is perpendicular to the normal vector of the palm plane and the palm reference vector is determined as the third lateral swing angle. The spatial angle between the skeletal direction vector of the proximal phalanx of the ring finger and the skeletal direction vector of the middle phalanx of the ring finger is defined as the eighth flexion angle. The spatial angle between the skeletal direction vector of the proximal phalanx of the little finger and the palm reference vector is determined as the ninth flexion angle; The spatial angle between the component of the skeletal direction vector of the proximal phalanx of the little finger that is perpendicular to the normal vector of the palm plane and the palm reference vector is determined as the fourth lateral swing angle. The spatial angle between the skeletal direction vector of the proximal phalanx of the little finger and the skeletal direction vector of the middle phalanx of the little finger is defined as the tenth flexion angle.

7. The method according to claim 1, characterized in that, The step of generating upper limb joint angle reference trajectories for each upper limb joint of the humanoid robot to be controlled based on the pose data includes: The pose data of the bones at the base of the palm are used as the target pose of the upper limb end. Using the target pose of the upper limb end as input, the inverse kinematics solver is called to solve the problem and obtain the target angle sequence of each upper limb joint that satisfies the joint limit constraint, which is used as the reference trajectory of the upper limb joint angle.

8. The method according to claim 1, characterized in that, The step of generating joint drive commands for controlling the movement of the upper limb and dexterous hand of the humanoid robot under control, based on the target finger joint angle and the upper limb joint angle reference trajectory, includes: The target finger joint angle is used as the desired position constraint for the dexterous hand, and the upper limb joint angle reference trajectory is used as the desired trajectory constraint for the upper limb. The desired position constraint of the dexterous hand and the desired trajectory constraint of the upper limb are input together into the whole body coordination controller; The whole-body coordination controller performs a unified solution under the constraints of whole-body dynamics and outputs upper limb joint drive commands and dexterous hand joint drive commands in parallel.

9. The method according to claim 1, characterized in that, The method further includes: In response to the first control command, switch to fixed hand mode; In the fixed hand shape mode, each fixed finger joint angle in the preset set of fixed finger joint angles is used as the target finger joint angle of the dexterous hand.

10. The method according to claim 9, characterized in that, The method further includes: In the fixed hand mode, in response to the second control command, the fixed finger joint angles are cyclically rotated between at least two preset fixed finger joint angle sets, and each fixed finger joint angle in the rotated fixed finger joint angle set is taken as the target finger joint angle of the dexterous hand.

11. The method according to claim 9, characterized in that, The step of using each fixed finger joint angle from a preset set of fixed finger joint angles as the target finger joint angles of the dexterous hand includes: Based on the fixed finger joint angles of one dexterous hand in the set of fixed finger joint angles, determine the target finger joint angle of the corresponding finger joint of the dexterous hand on that side; According to the mirror symmetry rule, the fixed finger joint angles of the corresponding finger joints of the dexterous hand on the other side are generated based on the fixed finger joint angles of the dexterous hand on the other side, and the generated fixed finger joint angles are used as the target finger joint angles of the corresponding finger joints of the dexterous hand on the other side.

12. A control device for a humanoid robot, characterized in that, The device includes: The data acquisition module is used to acquire the three-dimensional position data of key points of the hand bones of the target operator, as well as the pose data of the bones at the base of the palm of the target operator; The target finger joint angle determination module is used to determine the skeletal direction vector of each finger joint of the target operator based on the three-dimensional position data, and to determine the angle of each target finger joint of the dexterous hand on the humanoid robot to be controlled based on the skeletal direction vector. The joint drive command generation module is used to generate upper limb joint angle reference trajectories for each upper limb joint of the humanoid robot to be controlled based on the pose data, and to generate joint drive commands for controlling the movement of the upper limbs and the dexterous hand of the humanoid robot to be controlled based on the target finger joint angles and the upper limb joint angle reference trajectories.

Images