A control method, system, device, storage medium and program product

Abstract

The application relates to the technical field of artificial intelligence, and discloses a control method, a control system, a control device, a storage medium and a program product. The method comprises the following steps: acquiring bone data of a user's hand collected by a data glove, and generating a hand key point set of the user based on a preset rule and the bone data. The pose information of each hand key point in a hand coordinate system is converted into target pose information of each hand key point in a dexterous hand coordinate system, and based on the joint angle limiting range of each joint in the dexterous hand, the weight information of each hand key point and the target pose information corresponding to each hand key point, the target joint angle of each joint is determined, so that the error between the predicted pose information of each key point in the dexterous hand in the dexterous hand coordinate system and the target pose information of the hand key point satisfies a preset error condition. Finally, based on the target joint angle of each joint, the dexterous hand is controlled to perform a hand action. In this way, the execution accuracy of the dexterous hand can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a control method, system, device, storage medium, and program product. Background Technology

[0002] Currently, in scenarios involving real-time control of remote devices (which can be called teleoperation), data gloves (such as optical data gloves) can collect human hand movement information in real time and map it onto a dexterous hand. This allows the dexterous hand to reproduce the fine operations of human hands, such as grasping, pinching, and manipulating, thus enabling it to better replace humans in performing high-precision tasks in special or dangerous environments such as medical and aviation fields.

[0003] Currently, data gloves can typically output the spatial pose information of the user's hand's multiple skeletons, thus providing a relatively complete depiction of human hand movements. However, due to the significant differences between the human hand skeleton model and the dexterous hand joint model, directly mapping the hand movements depicted by the data glove onto the dexterous hand joint model can result in noticeable distortions in scenarios involving multi-finger collaboration or fine manipulation. For example, a data glove can output the position and pose information of 25 bone nodes of the human hand relative to the wrist bones (e.g., 6D pose), while a dexterous hand typically only has 21 degrees of freedom. The two are clearly mismatched in terms of skeleton topology, number of degrees of freedom, and joint structure. Directly having the dexterous hand perform the user's hand movements perceived by the data glove (e.g., clenching a fist) may lead to decreased accuracy and insufficient consistency in movement. Summary of the Invention

[0004] This application provides a control method, system, device, storage medium, and program product to improve the execution accuracy of dexterous hands.

[0005] In a first aspect, this application provides a control method, comprising: acquiring skeletal data of a user's hand collected by a data glove worn on the user's hand; generating a set of key points of the user's hand based on preset rules and the skeletal data, wherein the set of key points of the hand includes the pose information of each key point of the user's hand in the hand coordinate system and the weight information corresponding to each key point of the hand; converting the pose information of each key point of the hand in the hand coordinate system into target pose information of each key point of the hand in the dexterous hand coordinate system; determining the target joint angle of each joint based on the joint angle limit range of each joint in the dexterous hand, the weight information of each key point of the hand, and the target pose information corresponding to each key point of the hand; wherein, when the current joint angle of each joint of the dexterous hand is the target joint angle, the error between the preset pose information of each key point of the dexterous hand in the dexterous hand coordinate system predicted based on the forward kinematics model of the dexterous hand and the target pose information of the corresponding key point of the hand satisfies a preset error condition; and controlling the dexterous hand to perform hand movements based on the target joint angle of each joint.

[0006] Therefore, compared to directly having the dexterous hand perform the overall hand movements (such as clenching a fist) sensed by the data glove, this application, by selecting key skeletal nodes to generate a set of key points, avoids interference from redundant nodes, thereby improving the smoothness of the dexterous hand's hand movements. Furthermore, by performing coordinate transformation on the pose information of the hand's key points, it ensures a precise match between the user's hand movement intention and the dexterous hand's hand movements, avoiding movement deviations caused by inconsistencies in coordinates. In addition, by incorporating the limiting constraints of each joint of the dexterous hand, when outputting commands to control the dexterous hand's movement, it avoids outputting control commands with angles exceeding limits or not conforming to the device's kinematics.

[0007] In one possible implementation of the first aspect described above, the aforementioned skeletal data includes skeletal identifiers for referring to user hand skeletal nodes, and pose information of the user hand skeletal nodes indicated by each skeletal identifier relative to the user's wrist reference node. Furthermore, the hand coordinate system uses the wrist reference node as its origin.

[0008] Furthermore, the aforementioned process of generating a set of key hand points for the user based on preset rules and skeletal data includes: obtaining a preset index mapping table; wherein the index mapping table stores the bone identifiers and weight information corresponding to j skeletal nodes, where j is a positive integer and the value of j is the same as the number of joint degrees of freedom of the dexterous hand; based on the index mapping table, mapping the j bone identifiers in the skeletal data to each skeletal node of the user, and obtaining the weight information corresponding to each skeletal node; based on the connection relationship between each skeletal node, selecting each key hand point from each skeletal node; and obtaining a set of key hand points based on the pose information of each key hand point relative to the wrist reference node and the weight information of each key hand point.

[0009] In one possible implementation of the first aspect above, the process of filtering out key hand points from each bone node based on the connection relationships between the bone nodes includes: if n bone nodes belong to the same finger joint of the user, delete n-1 proximal bone nodes from the n bone nodes; wherein the distance of the n-1 proximal bone nodes relative to the wrist reference node is shorter than the distance of the remaining distal bone node from the n bone nodes relative to the wrist reference node, and n is an integer greater than 1. And / or, in the case where the first finger of the dexterous hand is in underactuated mode, if m bone nodes belong to the user's first finger, delete m-1 proximal bone nodes from the m bone nodes; wherein the distance of the m-1 proximal bone nodes relative to the wrist reference node is shorter than the distance of the remaining distal bone node from the m bone nodes relative to the wrist reference node, and m is an integer greater than 1.

[0010] In one possible implementation of the first aspect described above, converting the pose information of each hand keypoint in the hand coordinate system into target pose information of each hand keypoint in the dexterous hand coordinate system includes: obtaining an extrinsic transformation matrix for transforming the hand coordinate system to the dexterous hand coordinate system, wherein the extrinsic transformation matrix is ​​obtained by performing hand-eye calibration on the hand coordinate system and the dexterous hand coordinate system using a calibration device. Based on the extrinsic transformation matrix and the homogeneous pose matrix corresponding to the pose information of each hand keypoint in the hand coordinate system, the target homogeneous pose matrix of each hand keypoint in the dexterous hand coordinate system is determined.

[0011] In one possible implementation of the first aspect above, determining the target homogeneous pose matrix of each hand keypoint in the dexterous hand coordinate system based on the extrinsic transformation matrix and the homogeneous pose matrix corresponding to the pose information of each hand keypoint in the hand coordinate system includes: determining the initial homogeneous pose matrix of each hand keypoint in the dexterous hand coordinate system based on the extrinsic transformation matrix and the homogeneous pose matrix corresponding to the pose information of each hand keypoint in the hand coordinate system; in the case of left-right mirror image or axial transformation between the data glove and the dexterous hand, correcting the initial homogeneous pose matrix of each hand keypoint based on preset parameters to obtain the target homogeneous pose matrix, so that after the dexterous hand performs hand movements, the thumb abduction direction and palm orientation of the dexterous hand are the same as the thumb abduction direction and palm orientation of the data glove, respectively.

[0012] In one possible implementation of the first aspect described above, determining the target joint angle of each joint based on the joint angle limit range of each joint in the dexterous hand, the weight information of each key hand point, and the target pose information corresponding to each key hand point includes: constructing an objective function based on the forward kinematics model of the dexterous hand and the target pose information of each key hand point; wherein, the objective function is used to characterize the error between the preset pose information of each key hand point determined based on the forward kinematics model and the target pose information of each key hand point when the dexterous hand is in the current joint angle combination; wherein, the joint angle combination includes the joint angles corresponding to each joint, and the joint angles of each joint in the joint angle combination are within the joint angle limit range of each joint. The joint angles of each joint are iteratively adjusted, and the joint angle combination corresponding to the target joint angle of each joint is determined according to the function value of the objective function under different joint angle combinations; wherein, when each joint is in its corresponding target joint angle, the function value of the objective function is less than the function value corresponding to each joint in other joint angles.

[0013] In one possible implementation of the first aspect above, the objective function is used to characterize the error between the preset pose information of each key point in the dexterity hand determined based on the forward kinematics model and the target pose information of each key point in the hand, given the current joint angle combination. This error includes: determining the preset pose information of each key point in the dexterity hand in the dexterity hand coordinate system using the forward kinematics model; determining the position error corresponding to each key point in the hand based on the position information in the preset pose information and the target position information in the target pose information; determining the posture error corresponding to each key point in the hand based on the posture information in the preset pose information and the target posture information; and determining the total error corresponding to the current joint angle combination based on the weight information, the position error, and the posture error.

[0014] In one possible implementation of the first aspect above, controlling the dexterous hand to perform hand movements based on the target joint angles of each joint includes: performing low-pass filtering on the target joint angles of each joint to obtain the filtered joint angles corresponding to each joint; and sending the filtered joint angles corresponding to each joint to the controller of the dexterous hand so that each joint of the dexterous hand executes the filtered joint angles.

[0015] In one possible implementation of the first aspect described above, the low-pass filtering of the target joint angles of each joint to obtain the filtered joint angles corresponding to each joint includes: determining the filtered joint angles of the first joint corresponding to the current frame of bone data based on a preset smoothing coefficient, the target joint angle of the first joint corresponding to the current frame of bone data, and the filtered joint angle of the first joint corresponding to the previous frame of bone data. Alternatively, a second-order Butterworth filter is used to filter the target joint angles of the first joint corresponding to the current frame of bone data to obtain the filtered joint angles of the first joint corresponding to the current frame of bone data. Alternatively, if the difference between the target joint angle of the first joint corresponding to the current frame of bone data and the filtered joint angle of the first joint corresponding to the previous frame of bone data exceeds a preset maximum change threshold, the target joint angle of the first joint is adjusted to obtain the filtered joint angles of the first joint corresponding to the current frame of bone data, such that the difference between the filtered joint angle of the first joint corresponding to the current frame of bone data and the filtered joint angle of the first joint corresponding to the previous frame of bone data is the maximum change threshold.

[0016] Secondly, this application provides a control system, including: a data acquisition module, a skeleton specification module, a coordinate preprocessing module, a target joint angle determination module, and a control module.

[0017] The data acquisition module is used to acquire skeletal data of the user's hand collected by the data gloves worn on the user's hand.

[0018] The skeleton specification module is used to generate a set of key points for the user's hand according to preset rules and skeleton data. The set of key points includes the pose information of each key point of the user's hand in the hand coordinate system and the weight information corresponding to each key point.

[0019] The coordinate preprocessing module is used to convert the pose information of each hand key point in the hand coordinate system into the target pose information of each hand key point in the dexterous hand coordinate system.

[0020] The target joint angle determination module is used to determine the target joint angle of each joint based on the joint angle limit range of each joint in the dexterous hand, the weight information of each key point of the hand, and the target pose information corresponding to each key point of the hand. Among them, when the current joint angle of each joint of the dexterous hand is the target joint angle, the error between the preset pose information of each key point in the dexterous hand in the dexterous hand coordinate system predicted by the forward kinematics model of the dexterous hand and the target pose information of the corresponding key point of the hand meets the preset error condition.

[0021] The control module is used to control the dexterous hand to perform hand movements based on the target joint angles of each joint.

[0022] Thus, in this application, the control system can generate a set of key points by filtering key skeletal nodes, avoiding interference from redundant nodes and improving the smoothness of hand movements. Furthermore, the control system can perform coordinate transformation on the pose information of the hand's key points, ensuring precise matching between the user's hand movement intention and the dexterous hand's movements, avoiding movement deviations caused by inconsistencies in coordinates. In addition, by incorporating the limiting constraints of each joint of the dexterous hand, the control system can prevent the output of control commands that exceed angle limits or do not conform to the device's kinematics when outputting commands to control the dexterous hand's movement.

[0023] Thirdly, this application also provides an electronic device, comprising: at least one memory and at least one processor, wherein the memory is coupled to the processor; the memory is used to store computer program code / instructions; when the computer program code / instructions are executed by the processor, the electronic device performs the control method mentioned in the first aspect and any possible implementation thereof.

[0024] Fourthly, this application also provides a readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the control methods mentioned in the first aspect and any possible implementation thereof.

[0025] Fifthly, this application also provides a computer program product, comprising: computer instructions that, when executed on an electronic device, cause the electronic device to perform the control method mentioned in the first aspect and any possible implementation thereof.

[0026] The beneficial effects of the third to fifth aspects mentioned above can be referred to the relevant descriptions in the first aspect and any possible implementation of the first aspect, which will not be repeated here. Attached Figure Description

[0027] Figure 1 According to some embodiments of this application, a flowchart of a control method is shown;

[0028] Figure 2 According to some embodiments of this application, a flowchart for obtaining a set of key points on a user's hand is shown;

[0029] Figure 3 According to some embodiments of this application, a schematic diagram of a coordinate system transformation process is shown;

[0030] Figure 4 According to some embodiments of this application, a flowchart for determining the target joint angle of each joint is shown;

[0031] Figure 5According to some embodiments of this application, a flowchart of another control method is shown;

[0032] Figure 6 According to some embodiments of this application, a schematic diagram of the structure of a control system is shown;

[0033] Figure 7 According to some embodiments of this application, a schematic diagram of the hardware structure of an electronic device is shown. Detailed Implementation

[0034] The illustrative embodiments of this application include, but are not limited to, a control method, system, device, storage medium, and program product.

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0036] As mentioned earlier, during telemanipulation of the dexterous hand, there are often significant differences between the human hand skeleton model and the dexterous hand joint model. Directly mapping the hand movements depicted by the data glove onto the dexterous hand joint model can result in noticeable distortions in scenarios involving multi-finger collaboration or fine manipulation. For example, a data glove can output the poses (such as movement, rotation, etc.) of 25 skeletal nodes of the human hand relative to the wrist bones (also known as wrist reference nodes), while a dexterous hand typically only has 21 degrees of freedom (also known as active degrees of freedom). If the dexterous hand directly performs the user's hand movements sensed by the data glove (such as clenching a fist), the limited number of joint degrees of freedom may prevent the dexterous hand from accurately replicating the hand movements depicted by the data glove.

[0037] To address the aforementioned issues, this application provides a control method. Specifically, an electronic device acquires skeletal data of a user's hand collected by a data glove (such as an optical data glove). Then, the electronic device generates a set of key points for the user's hand based on preset rules and the skeletal data. This set of key points includes the pose information of each key point in the hand coordinate system and the corresponding weight information. Next, the electronic device converts the pose information of each key point in the hand coordinate system into target pose information for each key point in the dexterous hand coordinate system. Then, the electronic device determines the target joint angles of each joint in the dexterous hand based on the joint angle limit range of each joint, the weight information of each key point, and the target pose information corresponding to each key point. Where the current joint angle of each joint in the dexterous hand is the target joint angle, the error between the pose information of each key point in the dexterous hand predicted by the forward kinematics model of the dexterous hand in the dexterous hand coordinate system and the corresponding target pose information of the key points satisfies a preset error condition. Finally, electronic devices can control the dexterous hand to perform hand movements based on the target joint angles of each joint.

[0038] Therefore, compared to directly having the dexterous hand perform the overall hand movements (such as clenching a fist) sensed by the data glove, this application, by generating a set of key points through filtering key skeletal nodes, avoids interference from redundant nodes, thereby improving the smoothness of the dexterous hand's hand movements. Furthermore, by performing coordinate transformation on the pose information of the hand's key points, it ensures a precise match between the user's hand movement intention and the device's movement, avoiding movement deviations caused by inconsistencies in coordinates. In addition, by incorporating the limiting constraints of each joint of the dexterous hand, when outputting commands to control the dexterous hand's movement, it avoids outputting control commands with angles exceeding limits or that do not conform to the device's kinematics.

[0039] It should be noted that the control method described in this application can be applied to any electronic device. Electronic devices include, but are not limited to, mobile stations (MS) and mobile terminals (MT). For example, electronic devices can be computers, tablets, laptops, terminals in remote surgery, terminals in industrial control, terminals in self-driving vehicles, terminals in smart grids, terminals in transportation safety, etc. This application does not limit the specific form of the electronic device.

[0040] The following is based on Figure 1The flowchart shown provides a brief overview of the control method mentioned in the embodiments of this application. This control method can be applied to electronic devices, such as any electronic device like the computer mentioned above. Figure 1 As shown, specifically, the method is as follows:

[0041] S11: Obtain the skeletal data of the user's hand collected by the data gloves worn on the user's hand.

[0042] In some embodiments, the data glove can be any glove capable of outputting data on the user's hand bones, such as an inertial data glove, an optical data glove, or a fiber optic data glove.

[0043] In some embodiments, the Software Development Kit (SDK) callback interface of the data glove can send skeletal data of the user's hand to an electronic device at a fixed frequency, for example, in the form of a JSON structure.

[0044] In some embodiments, the user's hand skeletal data output by the data glove may include bone identifiers (also called node numbers) to refer to the user's hand skeletal nodes. For example, in the skeletal data, bone identifiers 0 to 24 may refer to 25 skeletal nodes of the user's left hand, and bone identifiers 25 to 49 may refer to 25 skeletal nodes of the user's right hand. For instance, bone identifier 0 may represent the wrist skeletal node of the user's left hand, and bone identifier 3 may represent the thumb tip skeletal node of the user's left hand.

[0045] In some embodiments, the user hand skeletal data output by the data glove may also include the user hand skeletal nodes indicated by each skeletal identifier and their pose information relative to the user's wrist reference node (also referred to as the wrist skeletal node). For example, if skeletal identifier 3 represents the thumb tip skeletal node of the user's left hand, the skeletal data may include the pose information corresponding to skeletal identifier 3 to represent the pose information of the user's left thumb tip skeletal node relative to the left wrist reference node.

[0046] For example, the pose information (i.e., position and orientation information) of each bone node can be a 6D pose, used to represent the three translational degrees of freedom and three rotational degrees of freedom of each bone node. It is understood that the orientation information can be represented using various data structures such as unit quaternions or three-dimensional rotation matrices, and this application does not limit this. When representing orientation information using unit quaternions, the quaternions can be normalized (e.g., forced to be greater than or equal to 0) to avoid approximately 180° jumps caused by sign reversal.

[0047] In some embodiments, the electronic device may also perform a consistency check on the skeletal data, for example, by representing the positional information of each skeletal node using a uniform distance unit (such as a meter or millimeter) so that the relevant calculations in S12 to S15 can be performed subsequently.

[0048] S12: Generate a set of key points for the user's hand based on preset rules and skeletal data.

[0049] In some embodiments, the preset rules may refer to an index mapping table storing skeletal node information and a method for filtering a set of key hand points among multiple skeletal nodes, etc., which are not limited in this application.

[0050] The following is combined with Figure 2 As shown, the specific process by which an electronic device generates a set of key points on a user's hand according to preset rules is described. Specifically, as... Figure 2 As shown:

[0051] S121: Obtain the preset index mapping table; wherein the index mapping table stores the bone identifiers and weight information corresponding to j bone nodes respectively, where j is a positive integer and the value of j is the same as the number of joint degrees of freedom of the dexterous hand.

[0052] For example, the index mapping table can store the correspondence between each bone identifier and bone node, as well as the correspondence between each bone node and weight information, using "{bone identifier ID: {bone node, weight information}}".

[0053] For example, if a dexterous hand has 21 joint degrees of freedom (as an example of j), the index mapping table can store the bone nodes corresponding to each of the 21 bone identifiers, along with their weight information. The bone node information stored in the index mapping table (such as weight information or information about which bone nodes are present) can be determined by the developers based on experimentation or experience. For instance, the index mapping table may not include information about some palm or redundant intermediate bone nodes, or it may include information about the bone nodes at the fingertips of the five fingers.

[0054] S122: Based on the index mapping table, map the j bone identifiers in the bone data to the user's bone nodes, and obtain the weight information corresponding to each bone node.

[0055] For example, if the index mapping table stores {bone identifier 0: {left wrist bone node, weight coefficient 0.2}}, the electronic device can use this index mapping table to map bone identifier 0 in the bone data to the user's left wrist bone node and obtain the corresponding weight coefficient 0.2.

[0056] In some embodiments, the electronic device may not map bone identifiers that exist in the skeletal data but not in the index mapping table. For example, if the skeletal data output by the data glove includes bone identifier 5, but bone identifier 5 is not included in the index mapping table, the electronic device may not map bone identifier 5 in the skeletal data.

[0057] S123: Based on the connection relationship between the mapped bone nodes, select each key point of the hand from each bone node.

[0058] In some embodiments, during the screening of hand key points, for bone nodes with duplicate outputs at the parent-child level, a rule of "retaining only the farthest leaf node" can be adopted to avoid optimization oscillations caused by conflicting targets on the same phalanx chain. Specifically, among the mapped bone nodes, if there are n (n is an integer greater than 1) bone nodes belonging to the same finger phalanx of the user (such as the phalanx of the middle finger near the palm), the electronic device can delete n-1 proximal bone nodes from the n bone nodes. The distances of the deleted n-1 proximal bone nodes relative to the wrist reference node are all shorter than the distances of the remaining distal bone node among the n bone nodes relative to the wrist reference node.

[0059] And / or, in other embodiments, when a finger of the dexterous hand is in an underactuated (i.e., multi-joint coupled) working mode, the electronic device can reduce the number of hand key points corresponding to that finger to one distal bone node during the screening process. Specifically, when the first finger of the dexterous hand (e.g., the index finger) is in an underactuated working mode, if m (m is an integer greater than 1) bone nodes in the mapped bone nodes all belong to the user's first finger, the electronic device can delete m-1 proximal bone nodes from the m bone nodes. The distances of the deleted m-1 proximal bone nodes relative to the wrist reference node are all shorter than the distances of the remaining distal bone node among the m bone nodes relative to the wrist reference node.

[0060] In this way, by using the above method, bone nodes that are prone to contradictions or problems can be deleted from multiple bone nodes, thereby obtaining the key points of the user's hand.

[0061] S124: Based on the pose information of each hand key point relative to the wrist reference node and the weight information of each hand key point, a set of hand key points is obtained.

[0062] In some embodiments, the set of hand key points may include the pose information of each hand key point of the user in the hand coordinate system, and the weight information corresponding to each hand key point obtained through S122. Since the pose information corresponding to each bone node in the skeletal data is the pose information of each bone node relative to the wrist reference node, the hand coordinate system mentioned in this application can also be a coordinate system with the wrist reference node as the origin.

[0063] The process of constructing a hand coordinate system is described below as an example.

[0064] In some embodiments, after acquiring various hand keypoints, the electronic device can select three geometric reference points, such as wrist bone nodes and hand keypoints on the index finger palmar chain (represented as p0, p1, and p2). Then, the electronic device can use the difference (p2-p0) between one of the geometric reference points on the index finger palmar chain (e.g., p2) and the wrist reference node (e.g., p0) as an approximate lateral dimension of the user's palm. Next, the electronic device can determine the average position of each hand keypoint based on its position, and subtract the average position from the position of each hand keypoint to obtain a decentered point set. Then, the electronic device can perform Singular Value Decomposition (SVD) on the decentered point set and select the right singular vector corresponding to the smallest singular value as the normal unit vector n of the user's palm.

[0065] Next, the electronic device projects the lateral measurement of the palm onto the normal unit vector, removes the normal vector, and normalizes the resulting vector to obtain a unit vector x. Then, the electronic device performs a cross product (e.g., x×n) between the normal unit vector n and the unit vector x to obtain a unit vector z, and determines the sign of the normal based on the position of the index finger keypoint. For example, the sign of the scalar (p1-p2)•z determines whether to simultaneously flip the unit vector n and the unit vector z. In this way, a hand coordinate system represented by the orthogonal basis matrix [x, n, z] is obtained, where the unit vectors x, n, and z are mutually perpendicular and can serve as the three direction axes of the hand coordinate system to represent the left-right, up-down, and forward-backward directions of the user's palm, respectively. Thus, after constructing the hand coordinate system with the wrist reference node as the origin, the pose information of each hand keypoint in the hand coordinate system can be determined based on the pose information of each hand keypoint relative to the wrist reference node. Furthermore, this method does not rely on a magnetometer; it can be achieved using only the pose information from the skeletal data collected by the data glove.

[0066] In summary, by executing the methods described in S121 to S124, the electronic device can obtain the user's hand key point set.

[0067] S13: Convert the pose information of each hand key point in the hand coordinate system into the target pose information of each hand key point in the dexterous hand coordinate system.

[0068] The specific process of coordinate system transformation can be found in [reference needed]. Figure 3 As shown:

[0069] S131: Obtain the extrinsic transformation matrix used to transform the hand coordinate system to the dexterous hand coordinate system.

[0070] In some embodiments, the extrinsic transformation matrix can be obtained in advance by calibrating the hand coordinate system and the dexterous hand coordinate system using a calibration device (such as a calibration plate). For example, the extrinsic transformation matrix can be a 4×4 homogeneous transformation matrix, which belongs to the rigid body motion group SE(3), that is, it contains both 3×3 rotation subblocks and 3×1 translation subblocks.

[0071] S132: Based on the extrinsic transformation matrix and the homogeneous pose matrix corresponding to the pose information of each hand key point in the hand coordinate system, determine the target homogeneous pose matrix of each hand key point in the dexterous hand coordinate system.

[0072] In some embodiments, the electronic device can determine the initial homogeneous pose matrix of each hand keypoint in the dexterous hand coordinate system based on the extrinsic transformation matrix and the homogeneous pose matrix corresponding to the pose information of each hand keypoint in the hand coordinate system. For example, for each hand keypoint, the electronic device can multiply the homogeneous pose matrix (such as matrix w) corresponding to its pose information in the hand coordinate system with the extrinsic transformation matrix (such as matrix T) (such as w×T) to obtain the initial homogeneous pose matrix of that hand keypoint in the dexterous hand coordinate system.

[0073] If there is no mirror image of the left or right hand or axial transformation (i.e., axial convention difference) between the data glove and the dexterous hand, the electronic device can directly use the initial homogeneous pose matrix of each hand key point in the dexterous hand coordinate system as the target homogeneous pose matrix.

[0074] Conversely, when there is a mirror image of the left and right hands or an axial transformation between the data glove and the dexterous hand (such as a difference in the rotation axis when the fingers are bent), the electronic device can correct the initial homogeneous pose matrix based on preset parameters (such as rotating it 180° around a certain axis) to obtain the target homogeneous pose matrix, so that after the dexterous hand performs hand movements, the thumb abduction direction and palm orientation of the dexterous hand can be the same as the thumb abduction direction and palm orientation of the data glove, respectively.

[0075] Thus, by executing the methods shown in S131 to S132, the electronic device can convert the pose information of each hand key point in the hand coordinate system into the target pose information of each hand key point in the dexterous hand coordinate system.

[0076] S14: Based on the joint angle limit range of each joint in the dexterous hand, the weight information of each key point of the hand, and the target pose information corresponding to each key point of the hand, determine the target joint angle of each joint.

[0077] In some embodiments, the joint angle limit range of each joint may refer to the rotation angle range of each joint, which is limited by the structure or manufacturing of the dexterous hand itself, and this application does not limit it.

[0078] In some embodiments, when the current joint angle of each joint of the dexterous hand is the target joint angle, the error between the preset pose information of each key point in the dexterous hand in the dexterous hand coordinate system predicted by the forward kinematic model of the dexterous hand and the target pose information of the corresponding hand key points (also known as pose information error) satisfies the preset error condition.

[0079] In some embodiments, the preset error condition mentioned in this application may refer to the pose information error corresponding to the current joint angle of each joint of the dexterous hand being the target joint angle, which is less than the pose information error corresponding to the joint angle of each joint of the dexterous hand being other joint angles. Furthermore, by setting the joint angle of each joint of the dexterous hand as the target joint angle, the pose information error between the dexterous hand and the user's hand can be minimized, thereby fulfilling the requirement that the dexterous hand can accurately replicate the user's hand movements.

[0080] The following is combined with Figure 4 As shown, the process by which an electronic device determines the target joint angles of each joint by checking whether the pose information error between the dexterous hand and the user's hand meets a preset error condition is described. Specifically, as... Figure 4 As shown:

[0081] S141: Construct an objective function based on the forward kinematics model of the dexterous hand and the target pose information of each key point of the hand.

[0082] In some embodiments, the forward kinematic model can be used to calculate the preset pose information of each link coordinate system and each key point based on kinematic parameters such as the joint angles, link lengths, joint axis directions and parent-child link relationships of each joint of the dexterous hand.

[0083] In some embodiments, the objective function can be used to characterize the error between the preset pose information of each key point in the dexterous hand determined based on the forward kinematics model and the target pose information of each key point in the hand, given the current joint angle combination. In other words, the objective function constructed in this application can be used to calculate the aforementioned pose information error between the dexterous hand and the user's hand. The joint angle combination can include the joint angles corresponding to each joint of the dexterous hand, and the joint angles of each joint in the joint angle combination are all within the joint angle limit range of each joint.

[0084] The process of constructing the objective function is described below.

[0085] In some embodiments, the electronic device can preset the dexterous hand to be in different joint angle combinations. Furthermore, when the dexterous hand is preset to be in a certain joint angle combination A1, the electronic device can also determine the preset pose information of each key point in the dexterous hand in the dexterous hand coordinate system through the forward kinematic model of the dexterous hand.

[0086] Then, when the pre-set dexterous hand is in a certain joint angle combination A1, the electronic device can determine the position error corresponding to each key point of the hand based on the position information in the pre-set pose information corresponding to each key point of the dexterous hand and the target position information in the target pose information corresponding to each key point of the hand. For example, for a certain key point k (k is a positive integer), the position p in the pre-set pose information corresponding to the key point k can be calculated. k Position p in the target pose k The difference between ' (e.g., p) k -p k '), and the square of the Euclidean norm of the difference (e.g., ||p) k -p k '|| 2 This is the position error corresponding to the key point k.

[0087] Furthermore, when the dexterous hand is preset to be in a certain joint angle combination A1, the electronic device can also determine the posture error corresponding to each key point of the hand based on the posture information in the preset pose information corresponding to each key point of the dexterous hand, and the target posture information in the target pose information corresponding to each key point of the hand. For example, for a key point k (k is a positive integer), the electronic device can first determine the posture matrix R in the preset pose information of the key point k. k Transpose the matrix to obtain the transpose matrix R. k T Then transpose matrix R k T Right-multiply the pose matrix R in the target pose information of the key point k k '(e.g., R) kT •R k '), thus obtaining the relative rotation matrix "R k T •R k '". and the relative rotation matrix "R" k T •R k Process it by '” (e.g., calculate "||log(R)"). k T •R k The value of ')||” is used to obtain the attitude error corresponding to the key point k.

[0088] Finally, the electronic device can determine the total error corresponding to the current joint angle combination (that is, the pose information error between the dexterous hand and the user's hand mentioned above) based on the weight information corresponding to each hand key point, the position error corresponding to each hand key point, and the posture error corresponding to each hand key point. For example, the electronic device can use a formula... The total error Q corresponding to the current joint angle combination is determined. Wherein, WP k It is the position error weight corresponding to the k-th key point, EP k The positional error of the kth key point, WR k It is the pose error weight corresponding to the k-th keypoint, ER k The attitude error of the kth key point.

[0089] Thus, the objective function can be constructed using the methods described above.

[0090] S142: Iteratively adjust the joint angles of each joint, and determine the joint angle combination corresponding to the target joint angle for each joint based on the function value of the objective function under different joint angle combinations. Specifically, when each joint is at its corresponding target joint angle, the function value of the objective function is less than the function value corresponding to the joint at other joint angles.

[0091] In some embodiments, the electronic device uses various methods such as the gradient-based Gauss-Newton method, the Levenberg-Marquardt (LM) damped least squares method, the sequential quadratic programming method, or the interior point solution method to iteratively determine the joint angle combination corresponding to the target joint angle of each joint.

[0092] For example, when using the LM damped least squares method, electronic devices can be solved by... The joint angles of each joint are iteratively adjusted. Here, J refers to the Jacobian matrix, J... TThis refers to the transpose of the Jacobian matrix; μ is the damping coefficient, which can be adaptively adjusted according to the error reduction; I is the identity matrix; e is the residual vector between the preset pose and the target pose; and δ is the increment to be solved. When the electronic device solves for the increment δ in the above formula, it can use δ+θ pre Update the current joint angle, where θ pre It is the joint angle from the previous iteration.

[0093] In this way, the electronic device can iteratively adjust the joint angles of each joint, and after each iteration, it can calculate the total error corresponding to the current joint angle combination based on the objective function constructed in S141. Furthermore, the electronic device can use the joint angles in the joint angle combination with the minimum total error (i.e., satisfying the preset error condition) as the target joint angles for each joint.

[0094] In some embodiments, after each iteration of joint angle adjustment, the electronic device can trim the joint angle to within a joint angle limit range to satisfy hard constraints. Alternatively, in some embodiments, when the electronic device receives continuous multi-frame skeletal data, it can also use the movement speed of each joint as a limiting condition, so that the continuous changes of each joint meet the speed limit condition. This application does not limit this aspect.

[0095] In some embodiments, electronic devices can divide multiple joints of a dexterous hand into multiple subchain blocks, for example, optimizing the thumb subchain first and then optimizing the other fingers, thereby reducing the computational scale of a single iteration on a very low-power computing platform.

[0096] Thus, by executing the methods shown in S141 and S142, the electronic device can determine the target joint angle of each joint.

[0097] S15: Based on the target joint angles of each joint, control the dexterous hand to perform hand movements.

[0098] The specific process of controlling a dexterous hand to perform hand movements can be found in [reference needed]. Figure 5 As shown:

[0099] S151: Perform low-pass filtering on the target joint angles of each joint to obtain the filtered joint angles for each joint.

[0100] In some embodiments, the electronic device can determine the joint angle of the first joint corresponding to the current frame of bone data based on a preset smoothing coefficient 'a', the target joint angle 'θ' of the first joint corresponding to the current frame of bone data, and the joint angle 'θ0' of the filtered first joint corresponding to the previous frame of bone data. For example, the joint angle of the filtered first joint corresponding to the current frame of bone data can be calculated using the formula "a•θ+(1-a)•θ0", where the smoothing coefficient 'a' can range from (0,1], and the smaller 'a' is, the stronger the filtering effect but the greater the delay.

[0101] In other embodiments, the electronic device may also employ a second-order Butterworth filter to filter the target joint angle of the first joint corresponding to the current frame of skeletal data, thereby obtaining the filtered joint angle of the first joint corresponding to the current frame of skeletal data. For example, in a second-order Butterworth filter, the cutoff frequency can be set to 8-15 Hz.

[0102] In other embodiments, if the difference between the target joint angle of the first joint corresponding to the current frame of skeletal data and the filtered joint angle of the first joint corresponding to the previous frame of skeletal data exceeds a preset maximum change threshold, the electronic device can further adjust the target joint angle of the first joint to obtain the filtered joint angle of the first joint corresponding to the current frame of skeletal data. This ensures that the difference between the filtered joint angle of the first joint corresponding to the current frame of skeletal data and the filtered joint angle of the first joint corresponding to the previous frame of skeletal data is equal to the maximum change threshold.

[0103] Thus, through the above method, the electronic device can perform low-pass filtering on the target joint angle of each joint, thereby improving the continuity and stability of the teleoperation process and reducing jitter and control abrupt changes.

[0104] S152: Send the filtered joint angles corresponding to each joint to the controller of the dexterous hand so that each joint of the dexterous hand executes the filtered joint angles.

[0105] Thus, in this application, by selecting key skeletal nodes to generate a set of key points, interference from redundant nodes can be avoided, thereby improving the smoothness of hand movements. Furthermore, by performing coordinate transformation on the pose information of the hand's key points, it is ensured that the user's hand movement intentions are accurately matched with the device's movements, avoiding movement deviations caused by inconsistencies in coordinates. In addition, by incorporating the limiting constraints of each joint of the dexterous hand, control commands that exceed angle limits or do not conform to the device's kinematics can be avoided when outputting commands to control the dexterous hand's movement.

[0106] Furthermore, in some embodiments, corresponding to the above-described control method, this application also provides a control system.

[0107] For example, such as Figure 6 As shown, the control system 600 may include a data acquisition module 601, a skeleton specification module 602, a coordinate preprocessing module 603, a target joint angle determination module 604, and a control module 605.

[0108] The data acquisition module 601 can be used to acquire the skeletal data of the user's hand collected by the data glove worn on the user's hand.

[0109] The skeleton specification module 602 can be used to generate a set of key points for the user's hand according to preset rules and skeleton data. The set of key points for the hand includes the pose information of each key point of the user's hand in the hand coordinate system and the weight information corresponding to each key point.

[0110] The coordinate preprocessing module 603 can be used to convert the pose information of each hand key point in the hand coordinate system into the target pose information of each hand key point in the dexterous hand coordinate system.

[0111] The target joint angle determination module 604 can be used to determine the target joint angle of each joint based on the joint angle limit range of each joint in the dexterous hand, the weight information of each key point of the hand, and the target pose information corresponding to each key point of the hand. Specifically, when the current joint angle of each joint in the dexterous hand is the target joint angle, the error between the preset pose information of each key point in the dexterous hand in the dexterous hand coordinate system predicted based on the forward kinematics model of the dexterous hand and the target pose information of the corresponding key point satisfies a preset error condition.

[0112] The control module 605 can be used to control the dexterous hand to perform hand movements based on the target joint angles of each joint.

[0113] Thus, in this application, the control system 600 can generate a set of key points by filtering key skeletal nodes to avoid interference from redundant nodes, thereby improving the smoothness of hand movements. Furthermore, the control system 600 can perform coordinate transformation on the pose information of the hand key points to ensure accurate matching between the user's hand movement intention and the dexterous hand movement, avoiding movement deviations caused by inconsistencies in coordinates. In addition, by incorporating the limiting constraints of each joint of the dexterous hand, the control system 600 can avoid outputting control commands with angles exceeding limits or that do not conform to the kinematics of the device when outputting commands to control the movement of the dexterous hand.

[0114] Furthermore, in some other embodiments, the control system 600 may also include a feedback module (not shown in the figure) that can be used to acquire the state and operating information of the dexterous hand.

[0115] Furthermore, in some embodiments, this application also provides a readable storage medium storing instructions. When executed on an electronic device, the instructions cause the electronic device to implement the control method mentioned in this application.

[0116] Furthermore, in some embodiments, this application also provides a computer program product, including computer instructions. These computer instructions, when executed on an electronic device, cause the electronic device to implement the control method mentioned in this application.

[0117] Furthermore, in some embodiments, this application also provides an electronic device. The electronic device includes at least one memory and at least one processor, with the memory coupled to the processor. The memory stores computer program code / instructions, which, when executed by the processor, enable the electronic device to implement the control methods mentioned in this application.

[0118] For example, such as Figure 7 As shown, this application provides a schematic diagram of the structure of an electronic device. Figure 7 As shown, the electronic device 700 includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704.

[0119] The memory 703 stores a computer program that can run on the processor 701. The memory 703 and the processor 701 communicate with each other through the communication interface 702 and the communication bus 704.

[0120] Processor 701 may include general-purpose processors, including central processing units, neural network processors, etc.; it may also be digital signal processing (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0121] The memory 703 may include random access memory (RAM) or non-volatile memory, etc. Optionally, the memory 703 may also be at least one storage device located remotely from the processor 701.

[0122] The communication bus 704 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This communication bus 704 can be divided into an address bus, a data bus, a control bus, etc.

[0123] It should be noted that the connection method between units / modules in the accompanying drawings provided in this application is only one example. In other embodiments, units / modules can be directly connected via buses, signal lines, or connectors, or they can be indirectly connected.

[0124] Furthermore, some structural or methodological features may be shown in a specific arrangement and / or order in the accompanying drawings provided in this application. However, it should be understood that such a specific arrangement and / or order may not be necessary. For example, in other embodiments, these features may be arranged in a manner and / or order different from that shown in the drawings. In addition, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments; for example, in other embodiments, these features may be omitted or may be combined with other features.

[0125] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0126] The embodiments disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0127] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried on or stored thereon by one or more transient or non-transitory machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors.

[0128] It should be noted that in the examples and description of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0129] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. A control method, characterized in that, include: Acquire skeletal data of the user's hand from the data gloves worn on the user's hand; A set of key points for the user's hand is generated based on preset rules and the skeletal data. The set of key points includes the pose information of each key point of the user's hand in the hand coordinate system and the weight information corresponding to each key point. Each key point is selected from the skeletal nodes based on the connection relationship between the user's skeletal nodes corresponding to the skeletal data. The pose information of each hand key point in the hand coordinate system is converted into the target pose information of each hand key point in the dexterous hand coordinate system. Based on the joint angle limit range of each joint in the dexterous hand, the weight information of each key hand point, and the target pose information corresponding to each key hand point, the target joint angle of each joint is determined; wherein, when the current joint angle of each joint in the dexterous hand is the target joint angle, the error between the preset pose information of each key hand point in the dexterous hand in the dexterous hand coordinate system predicted based on the forward kinematics model of the dexterous hand and the target pose information of the corresponding key hand point satisfies the preset error condition. Based on the target joint angles of each joint, the dexterous hand is controlled to perform hand movements; The key hand points mentioned above are selected from the skeletal nodes of the user based on the connection relationships between the skeletal data, including: If, among the aforementioned skeletal nodes, n skeletal nodes belong to the same finger joint of the user, then delete n-1 proximal skeletal nodes from among the n skeletal nodes; wherein the distances of the n-1 proximal skeletal nodes relative to the user's wrist reference node are all shorter than the distances of the remaining distal skeletal node among the n skeletal nodes relative to the wrist reference node, where n is an integer greater than 1; and / or, When the first finger of the dexterous hand is in underactuated working mode, if m bone nodes belong to the user's first finger, delete m-1 proximal bone nodes from the m bone nodes; wherein the distance of the m-1 proximal bone nodes relative to the wrist reference node is shorter than the distance of the remaining distal bone node among the m bone nodes relative to the wrist reference node, and m is an integer greater than 1.

2. The control method according to claim 1, characterized in that, The skeletal data includes skeletal identifiers for referring to user hand skeletal nodes, and pose information of the user hand skeletal nodes indicated by each skeletal identifier relative to the user's wrist reference node, wherein the hand coordinate system is based on the wrist reference node as the origin. The process of generating a set of key points for the user's hand based on preset rules and the skeletal data includes: Obtain a preset index mapping table; wherein the index mapping table stores the bone identifiers and weight information corresponding to j bone nodes respectively, where j is a positive integer and the value of j is the same as the number of joint degrees of freedom of the dexterous hand; Based on the index mapping table, the j bone identifiers in the bone data are mapped to the user's bone nodes, and the weight information corresponding to each bone node is obtained. Based on the connection relationships between the bone nodes, the key points of the hand are selected from the bone nodes; The set of hand key points is obtained based on the pose information of each hand key point relative to the wrist reference node and the weight information of each hand key point.

3. The control method according to claim 2, characterized in that, The step of converting the pose information of each hand key point in the hand coordinate system into the target pose information of each hand key point in the dexterous hand coordinate system includes: An extrinsic transformation matrix is ​​obtained for transforming the hand coordinate system to the dexterous hand coordinate system. The extrinsic transformation matrix is ​​obtained by performing hand-eye calibration on the hand coordinate system and the dexterous hand coordinate system using a calibration device. Based on the extrinsic transformation matrix and the homogeneous pose matrix corresponding to the pose information of each hand key point in the hand coordinate system, the target homogeneous pose matrix of each hand key point in the dexterous hand coordinate system is determined.

4. The control method according to claim 3, characterized in that, The determination of the target homogeneous pose matrix of each hand key point in the dexterous hand coordinate system based on the extrinsic transformation matrix and the homogeneous pose matrix corresponding to the pose information of each hand key point in the hand coordinate system includes: Based on the extrinsic transformation matrix and the homogeneous pose matrix corresponding to the pose information of each hand key point in the hand coordinate system, the initial homogeneous pose matrix of each hand key point in the dexterous hand coordinate system is determined. In the case where there is a left-right mirror image or axial transformation between the data glove and the dexterous hand, the initial homogeneous pose matrix of each hand key point is corrected based on preset parameters to obtain the target homogeneous pose matrix, so that after the dexterous hand performs the hand action, the thumb abduction direction and palm orientation of the dexterous hand are the same as the thumb abduction direction and palm orientation of the data glove, respectively.

5. The control method according to claim 4, characterized in that, The method for determining the target joint angle of each joint based on the joint angle limit range of each joint in the dexterous hand, the weight information of each key point of the hand, and the target pose information corresponding to each key point of the hand includes: Based on the positive kinematics model of the dexterous hand and the target pose information of each key hand point, an objective function is constructed; wherein, The objective function is used to characterize the error between the preset pose information of each key point in the dexterous hand determined based on the positive kinematics model and the target pose information of each key point in the hand, when the dexterous hand is in the current joint angle combination; wherein, the joint angle combination includes the joint angles corresponding to each joint respectively, and the joint angles of each joint in the joint angle combination are within the joint angle limit range of each joint. The joint angles of each joint are iteratively adjusted, and the joint angle combination corresponding to the target joint angle of each joint is determined based on the function values ​​of the objective function under different joint angle combinations; wherein, When each joint is at its corresponding target joint angle, the function value of the objective function is less than the function value corresponding to when each joint is at other joint angles.

6. The control method according to claim 5, characterized in that, The objective function is used to characterize the error between the preset pose information of each key point in the dexterous hand determined based on the positive kinematics model and the target pose information of each key point in the hand, given the current joint angle combination. This error includes: The preset pose information of each key point in the dexterous hand in the dexterous hand coordinate system is determined by the positive kinematics model. Based on the position information in the preset pose information corresponding to each key point in the dexterous hand, and the target position information in the target pose information corresponding to each key point in the hand, the position error corresponding to each key point in the hand is determined respectively. Based on the posture information in the preset posture information corresponding to each key point in the dexterous hand, and the target posture information in the target posture information corresponding to each key point in the hand, the posture error corresponding to each key point in the hand is determined respectively. Based on the weight information corresponding to each hand key point, the position error corresponding to each hand key point, and the posture error corresponding to each hand key point, the total error corresponding to the current joint angle combination is determined.

7. The control method according to claim 6, characterized in that, The control of the dexterous hand to perform hand movements based on the target joint angles of each joint includes: The target joint angles of each joint are low-pass filtered to obtain the filtered joint angles corresponding to each joint. The filtered joint angles corresponding to each joint are sent to the controller of the dexterous hand, so that each joint of the dexterous hand executes the filtered joint angles.

8. The control method according to claim 7, characterized in that, The step of performing low-pass filtering on the target joint angles of each joint to obtain the filtered joint angles corresponding to each joint includes: Based on a preset smoothing coefficient, the target joint angle of the first joint corresponding to the current frame's skeletal data, and the filtered joint angle of the first joint corresponding to the previous frame's skeletal data, the filtered joint angle of the first joint corresponding to the current frame's skeletal data is determined; or, A second-order Butterworth filter is used to filter the target joint angle of the first joint corresponding to the current frame skeleton data to obtain the filtered joint angle of the first joint corresponding to the current frame skeleton data; or, If the difference between the target joint angle of the first joint corresponding to the current frame bone data and the filtered joint angle of the first joint corresponding to the previous frame bone data exceeds a preset maximum change threshold, the target joint angle of the first joint is adjusted to obtain the filtered joint angle of the first joint corresponding to the current frame bone data, so that the difference between the filtered joint angle of the first joint corresponding to the current frame bone data and the filtered joint angle of the first joint corresponding to the previous frame bone data is the maximum change threshold.

9. A control system, characterized in that, include: The data acquisition module is used to acquire skeletal data of the user's hand collected by the data gloves worn on the user's hand; The skeleton reduction module is used to generate a set of key points for the user's hand according to preset rules and the skeleton data. The set of key points includes the pose information of each key point of the user's hand in the hand coordinate system and the weight information corresponding to each key point. Each key point is selected from each skeleton node according to the connection relationship between each skeleton node of the user corresponding to the skeleton data. The coordinate preprocessing module is used to convert the pose information of each hand key point in the hand coordinate system into the target pose information of each hand key point in the dexterous hand coordinate system. The target joint angle determination module is used to determine the target joint angle of each joint based on the joint angle limit range of each joint in the dexterous hand, the weight information of each key point of the hand, and the target pose information corresponding to each key point of the hand; wherein, when the current joint angle of each joint of the dexterous hand is the target joint angle, the error between the preset pose information of each key point in the dexterous hand in the dexterous hand in the dexterous hand coordinate system predicted based on the forward kinematics model of the dexterous hand and the target pose information of the corresponding key point of the hand satisfies the preset error condition. The control module is used to control the dexterous hand to perform hand movements according to the target joint angles of each joint; The key hand points are selected from the skeletal nodes based on the connection relationships between the user's skeletal nodes corresponding to the skeletal data, including: If, among the aforementioned skeletal nodes, n skeletal nodes belong to the same finger joint of the user, then delete n-1 proximal skeletal nodes from among the n skeletal nodes; wherein the distances of the n-1 proximal skeletal nodes relative to the user's wrist reference node are all shorter than the distances of the remaining distal skeletal node among the n skeletal nodes relative to the wrist reference node, where n is an integer greater than 1; and / or, When the first finger of the dexterous hand is in underactuated working mode, if m bone nodes belong to the user's first finger, delete m-1 proximal bone nodes from the m bone nodes; wherein the distance of the m-1 proximal bone nodes relative to the wrist reference node is shorter than the distance of the remaining distal bone node among the m bone nodes relative to the wrist reference node, and m is an integer greater than 1.

10. An electronic device, characterized in that, include: At least one memory and at least one processor, the memory being coupled to the processor; the memory being used to store computer program code / instructions; when the computer program code / instructions are executed by the processor, causing the electronic device to implement the control method as described in any one of claims 1 to 8.

11. A readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the control method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, include: Computer instructions, when executed on an electronic device, cause the electronic device to perform the control method as described in any one of claims 1 to 8.

Images