A dexterous hand teleoperation method, device, system and electronic device

Abstract

Embodiments of the present application provide a dexterous hand teleoperation method, device, system and electronic equipment, and relate to the field of robots, the method comprising: receiving angle information collected by an angle information collection unit of an exoskeleton glove hand joint configuration; determining a first joint pose of a wearer's hand joint of the exoskeleton glove based on the angle information; using a motion reorientation algorithm to map the first joint pose to a second joint pose of a dexterous hand hand joint, sending the second joint pose to the dexterous hand to control the dexterous hand to reach the second joint pose; receiving haptic information fed back by the dexterous hand, determining contact area contour information of the exoskeleton glove based on the haptic information; and driving a haptic feedback unit provided on a corresponding finger joint of the exoskeleton glove to produce convex deformation based on the contact area contour information. The scheme provided by the embodiments of the present application can improve the accuracy of dexterous hand control.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method, apparatus, system and electronic device for dexterous hand remote manipulation. Background Technology

[0002] Dexterous hand telemanipulation technology is a research hotspot in the field of robot human-computer interaction, possessing significant engineering value in precision component assembly and high-risk unmanned operation scenarios. Furthermore, in the training of embodied intelligent robot models, dexterous hand telemanipulation technology can replicate the fine-grained hand control operations of humans, providing high-quality, realistic interaction samples for the models and demonstrating strong potential for scientific research application.

[0003] In view of the above, this application provides a dexterous hand remote control scheme to achieve precise control of the dexterous hand. Summary of the Invention

[0004] The purpose of this application is to provide a method, device, system, and electronic device for dexterous hand remote control, so as to achieve precise control of the dexterous hand. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a dexterous hand remote manipulation method applied to an exoskeleton glove, the method comprising:

[0006] The system receives angle information collected by the angle information acquisition unit configured in the hand joints of the exoskeleton glove, and determines the first joint pose of the wearer's hand joints based on the angle information.

[0007] An action redirection algorithm is used to map the first joint pose to the second joint pose of the dexterous hand to be manipulated, and the second joint pose is sent to the dexterous hand to manipulate the dexterous hand to reach the second joint pose.

[0008] Receive tactile information from the dexterous hand and determine the contact area contour information of the exoskeleton glove based on the tactile information;

[0009] Based on the contour information of the contact area, the tactile feedback unit set on the corresponding finger joint of the exoskeleton glove is driven to produce a raised deformation.

[0010] Secondly, embodiments of this application provide a dexterous hand remote manipulation method, applied to a dexterous hand, the method comprising:

[0011] Receive a second joint pose sent by the exoskeleton glove, the second joint pose being obtained based on a first joint pose of the wearer's hand joints;

[0012] Control the hand joints of the dexterous hand to achieve the second joint position;

[0013] The system receives tactile information collected by contact force sensors on each finger joint and sends the tactile information to the exoskeleton glove.

[0014] Thirdly, embodiments of this application provide a dexterous hand remote control device for use in exoskeleton gloves, the device comprising:

[0015] The first pose determination module is used to receive angle information collected by the angle information acquisition unit configured in the hand joints of the exoskeleton glove, and determine the first joint pose of the hand joints of the wearer wearing the exoskeleton glove based on the angle information.

[0016] The second pose determination module is used to map the first joint pose to the second joint pose of the dexterous hand joint to be controlled using a motion redirection algorithm, and send the second joint pose to the dexterous hand to control the dexterous hand to reach the second joint pose.

[0017] The contour information determination module is used to receive tactile information fed back by the dexterous hand, and determine the contour information of the contact area of ​​the exoskeleton glove based on the tactile information.

[0018] The contour reproduction module is used to drive the tactile feedback unit set on the corresponding finger joint of the exoskeleton glove to generate a raised deformation based on the contour information of the contact area.

[0019] Fourthly, embodiments of this application provide a dexterous hand remote control device, applied to a dexterous hand, the device comprising:

[0020] The pose receiving module receives the second joint pose sent by the exoskeleton glove. The second joint pose is obtained based on the first joint pose of the wearer's hand joints.

[0021] The pose control module is used to control the hand joints of the dexterous hand to reach the second joint pose;

[0022] The information feedback module is used to receive tactile information collected by the contact force sensors set in each finger joint and send the tactile information to the exoskeleton glove.

[0023] Fifthly, embodiments of this application provide a dexterous hand remote control system, including an exoskeleton glove and a dexterous hand, wherein...

[0024] The exoskeleton glove is used to receive angle information collected by the angle information acquisition unit configured on the hand joints of the exoskeleton glove, and based on the angle information, determine the first joint pose of the hand joints of the wearer wearing the exoskeleton glove; and use a motion redirection algorithm to map the first joint pose to the second joint pose of the dexterous hand to be controlled, and send the second joint pose to the dexterous hand.

[0025] The dexterous hand is used to receive the second joint pose sent by the exoskeleton glove; control the hand joints of the dexterous hand to reach the second joint pose; receive tactile information collected by the contact force sensors set on each finger joint, and send the tactile information to the exoskeleton glove;

[0026] The exoskeleton glove is also used to receive tactile information from the dexterous hand, determine the contact area contour information of the exoskeleton glove based on the tactile information, and drive the tactile feedback unit set on the corresponding finger joint of the exoskeleton glove to generate a protruding deformation based on the contact area contour information.

[0027] In a sixth aspect, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0028] Memory, used to store computer programs;

[0029] A processor, when executing a program stored in memory, implements the steps of the method described in the first or second aspect.

[0030] In a seventh aspect, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the first or second aspect.

[0031] Eighthly, embodiments of the present invention provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps of the method described in the first or second aspect.

[0032] Beneficial effects of the embodiments in this application:

[0033] As can be seen from the above, in the solution provided by this application embodiment, after the dexterous hand receives the second joint pose from the exoskeleton glove, it not only controls its own joints to reach the second joint pose, but also feeds back the tactile information it perceives to the exoskeleton glove. Thus, the exoskeleton glove can reproduce the object contact shape based on the feedback tactile information. It is evident that this solution integrates hand motion acquisition, cross-shape motion mapping, dexterous hand tactile feedback, and exoskeleton glove contour tactile reproduction into a complete closed loop. The wearer does not need to rely solely on vision to judge the contact state, and can perceive the object contact situation in real time and intuitively through the object contour reproduced by the exoskeleton glove, restoring the real tactile sensation of operation. This allows the wearer to perform more accurate and refined operations based on the contact situation, and transmits this operation to the dexterous hand, thereby improving the accuracy of dexterous hand control.

[0034] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0036] Figure 1 This is a schematic diagram illustrating one application scenario of an embodiment of this application;

[0037] Figure 2 A flowchart illustrating the first dexterous hand telemanipulation method provided in this application embodiment;

[0038] Figure 3 A flowchart illustrating the second dexterous hand telemanipulation method provided in this application embodiment;

[0039] Figure 4 A schematic diagram of the structure of a dexterous hand remote control system provided in this application embodiment;

[0040] Figure 5 This is a schematic diagram of an interaction process according to an embodiment of this application;

[0041] Figure 6 A schematic diagram of the structure of the first dexterous hand remote control device provided in the embodiments of this application;

[0042] Figure 7 This is a schematic diagram of the structure of the second type of dexterous hand remote control device provided in the embodiments of this application;

[0043] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0045] First, the application scenarios of the embodiments of this application will be introduced.

[0046] The solution provided in this application is applied to the scenario of dexterous hands of exoskeleton gloves for remote operation robots, specifically including scenarios such as embodied intelligent dexterous operation data acquisition and precision operations in high-risk environments.

[0047] Among them, such as Figure 1 As shown, the exoskeleton glove is worn on the wearer's hand. It collects hand movement information through joint sensors, and after the processing unit completes the pose calculation and retargeting mapping, it controls the dexterous hand to perform corresponding actions synchronously.

[0048] It should be noted that the above figures are for illustrative purposes only and do not limit the specific structure of exoskeleton gloves or dexterous hands.

[0049] In order to achieve precise control of a dexterous hand, this application provides a dexterous hand remote control solution.

[0050] See Figure 2 This is a schematic diagram of a dexterous hand remote control method applied to an exoskeleton glove, provided by an embodiment of this application. The method includes the following steps S201 to S204:

[0051] Step S201: Receive angle information collected by the angle information acquisition unit of the exoskeleton glove hand joint configuration, and determine the first joint pose of the wearer's hand joint based on the angle information.

[0052] The angle information acquisition unit is described below:

[0053] For fingers other than the thumb, the angle information acquisition unit may include encoders deployed at each joint, covering the flexion-extension and lateral swing degrees of freedom of the metacarpophalangeal joint (MCP), the flexion-extension degrees of freedom of the proximal interphalangeal joint (PIP), and the flexion-extension degrees of freedom of the distal interphalangeal joint (DIP). Preferably, the MCP joints of the fingers other than the thumb may consist of two approximately orthogonal joints, namely the extension / flexion (E / F) joint and the lateral swing / adduction / abduction (A / A) joint, which may be referred to as the MCP E / F joint and the MCP A / A joint, respectively; the PIP and DIP joints of the aforementioned fingers may each only include the E / F joint.

[0054] For the thumb, the angle information acquisition unit may include encoders deployed at each joint, covering multiple degrees of freedom of the thumb carpometacarpal joint (CMC) and corresponding degrees of freedom of the metacarpophalangeal and interphalangeal joints (IP). Preferably, the thumb CMC joint may consist of three approximately orthogonal joints: the E / F joint, the A / A joint, and the axial rotation / opposition rotation (Rot / Opp joint), which can be referred to as the CMC E / F joint, the CMC A / A joint, and the CMC Rot / Opp joint, respectively; the thumb MCP joint may consist of two approximately orthogonal joints: the E / F joint and the A / A joint, which can be referred to as the MCP E / F joint and the MCP A / A joint, respectively; the thumb IP joint may consist of only the E / F joint.

[0055] In this way, the angle information acquisition unit can collect the rotation angle data of each joint in real time.

[0056] After obtaining the angle information of the hand joints, the angle information is substituted into the pre-constructed kinematic model of the wearer's hand. Through forward kinematic mapping, complete hand motion information such as the six-dimensional pose of each finger joint and the angle of each joint is obtained, which is the first joint pose.

[0057] Considering the differences in actual hand parameters such as bone length and palm width among different individuals, it is difficult to accurately determine the actual hand posture of the wearer based on the joint angle of the exoskeleton glove. Therefore, in one possible implementation, the exoskeleton glove is equipped with a segmented knuckle self-alignment slide rail. The wearer can align their own knuckles with the knuckles of the exoskeleton glove by adjusting the position of the slider in the segmented knuckle self-alignment slide rail.

[0058] In this scenario, angle information collected in advance by multiple angle information acquisition units under different hand postures of the wearer can be used to calibrate the wearer's actual hand parameters as ergonomic parameters. Based on these ergonomic parameters, a realistic kinematic model of the wearer's hand can be constructed. This approach specifically calibrates the wearer's individualized ergonomic parameters, thereby reducing the impact of individual hand shape differences on determining hand posture and improving the accuracy of the determined hand posture.

[0059] Step S202: Using a motion redirection algorithm, the first joint pose is mapped to the second joint pose of the dexterous hand to be manipulated, and the second joint pose is sent to the dexterous hand to manipulate the dexterous hand to achieve the second joint pose.

[0060] Action redirection, also known as action remapping, can be achieved by using various action redirection algorithms to convert the wearer's hand pose to a dexterous hand pose. This application does not limit this.

[0061] To achieve adaptive redirection of hand movements under different operating scenarios and improve the scene adaptability and overall accuracy of the pose mapping scheme, one possible implementation is to determine the operation stage of the task performed by the dexterous hand based on the spatial distance between the dexterous hand and the task target; based on the operation stage, determine the optimized weights of each pose component in the first joint pose; and based on the optimized weights of each pose component, perform weighted fusion of each pose component to obtain the second joint pose of the dexterous hand joint.

[0062] In practice, a first threshold and a second threshold can be set, where the first threshold is a larger distance threshold and the second threshold is a smaller contact determination threshold. When the spatial distance between the dexterous fingertip and the task target is greater than the first threshold, the current task is determined to be in the distance phase, which corresponds to the stage where the wearer's hand moves in a wide range. When the spatial distance is between the first threshold and the second threshold, the current task is determined to be in the proximity phase, which corresponds to the stage where the wearer perceives that the dexterous hand is about to contact the task target and prepares to perform a set hand operation (such as clenching a fist, interlocking fingers, etc.). When the spatial distance is less than or equal to the second threshold, or when the dexterous hand detects a contact force signal, the current task is determined to be in the contact phase, which corresponds to the stage where the wearer actually performs a hand operation on the task target.

[0063] In one possible implementation, the adjustment rules corresponding to the aforementioned three operational stages are as follows:

[0064] In the distance phase, the core requirement is to restore the overall opening and closing shape of the human hand. Therefore, the default weight of the hand shape component is increased, while the default weight of the other components is maintained to ensure that the overall hand shape of the dexterous hand matches the human hand.

[0065] During the approach phase, the core requirement is to ensure the accuracy of fingertip-to-finger contact and the correct orientation of the contact. Therefore, the default weights of the relative position component of the fingertips and the normal vector component of the thumb are increased simultaneously to improve the accuracy of finger-to-finger operation.

[0066] During the contact phase, the core requirement is to reproduce the joint bending shape of a human hand after being subjected to force and to improve the object envelope effect. Therefore, the default weight of the multi-joint angle components is increased to restore the segmented force posture of the fingers.

[0067] Step S203: Receive tactile information from the dexterous hand and determine the contact area contour information of the exoskeleton glove based on the tactile information.

[0068] The purpose of this step is to restore the tactile information perceived by the dexterous hand through the exoskeleton glove, so that the wearer can perform subsequent operations based on the perceived tactile information.

[0069] In this embodiment, each finger joint of the dexterous hand may be equipped with a contact force sensor to measure the perceived tactile information. The specific type of the sensor is not limited in this application.

[0070] Preferably, the fingertip area of ​​the dexterous hand can be equipped with distributed three-dimensional tactile sensors. In this case, the exoskeleton glove can receive the fingertip pressure value sent by the dexterous hand, mark the sensing points with pressure values ​​higher than a preset threshold as effective contact areas, and then stitch together the coordinates of all effective points to generate two-dimensional contact area contour information.

[0071] Step S204: Based on the contour information of the contact area, drive the tactile feedback unit set on the corresponding finger joint of the exoskeleton glove to generate a raised deformation.

[0072] In this way, the outline information of the object perceived by the dexterous hand is replicated on the exoskeleton glove.

[0073] In one possible implementation, each fingertip of the exoskeleton can be equipped with an electroosmotic pump array tactile feedback unit. In this way, based on the effective point coordinates included in the contour information of the contact area, the medium can be delivered to the corresponding pump body to generate local bubbling protrusions, thereby accurately replicating the actual contour shape of the object in contact with the finger.

[0074] In another possible implementation, each fingertip of the exoskeleton can be equipped with an independent array of micro-airbags. This allows the corresponding airbags to inflate and expand according to the contour region information to form a protrusion. This solution is simpler in structure and lower in cost, but the contour reconstruction accuracy is weaker than that of the electroosmotic pump solution.

[0075] As can be seen from the above, in the solution provided by this application embodiment, the exoskeleton glove calculates the first joint pose of the wearer's real hand by collecting the angles of each joint; then, through motion redirection, the real hand pose is mapped to the second joint pose of the dexterous hand and sent down; next, it receives the tactile data returned by the dexterous hand and maps the contact contour of the dexterous hand to the contact area contour of the exoskeleton glove; finally, it can drive the local protrusion of the corresponding finger joint tactile unit to reproduce the object contact shape on the exoskeleton glove. It is evident that this solution integrates hand motion acquisition, cross-shape motion mapping, dexterous hand tactile feedback, and exoskeleton glove contour tactile reproduction into a complete closed loop. The wearer does not need to rely solely on vision to judge the contact state and can perceive the object contact situation in real time and intuitively through the object contour reproduced by the exoskeleton glove, restoring the real tactile sensation of operation. This allows the wearer to perform more accurate and refined operations based on the contact situation and transmits this operation to the dexterous hand, thereby improving the accuracy of dexterous hand control.

[0076] To reproduce the joint interaction forces caused by the compression and pulling of objects, and to form a dual sensory loop of touch and force, thereby further enhancing the wearer's sense of control realism, in one possible implementation, the exoskeleton glove's hand joints are equipped with elastic drive units. The exoskeleton glove can also receive force feedback from the dexterous hand, and based on the force feedback information, drive the elastic drive units of the corresponding joints of the exoskeleton glove to move.

[0077] The contact force sensor in the knuckles of the dexterous hand can also be used to measure the perceived force information. This application does not limit its specific type.

[0078] Preferably, a distributed three-dimensional tactile sensor can be provided on the fingertip area of ​​each finger of the dexterous hand, and positive pressure sensors can be provided on the two finger side areas and the finger back area. The aforementioned distributed three-dimensional tactile sensor and positive pressure sensor can be collectively referred to as a ring-type tactile sensor, wherein the distributed three-dimensional tactile sensor is used to sense tactile information as well as distributed force and tangential force, and the positive pressure sensor is used to sense positive pressure.

[0079] Preferably, the joints such as MCP, PIP, DIP, and CMC of the exoskeleton glove can all be equipped with independent series elastic actuators (SEA) as elastic driving units, which can receive the force data of the joints transmitted back by the dexterous hand in real time, and match the SEA unit of the corresponding finger joint to output buffer resistance, thereby reproducing the bidirectional interactive force of the object on the knuckles, such as compression and pulling.

[0080] In one possible implementation, to simplify the structure of the exoskeleton glove and improve the wearer's operational accuracy, the elastic drive unit is electrically connected to an external drive motor. In this case, the target deformation of the elastic drive unit of the corresponding joint of the exoskeleton glove can be determined based on the contact force data of each finger joint of the exoskeleton glove. Then, based on the target deformation, a drive command is generated for the drive motor connected to the elastic drive unit. Finally, a drive command is sent to the drive motor so that the drive motor controls the elastic drive unit to produce the target deformation through the transmission rope.

[0081] Among them, the stiffness coefficient k of the SEA elastic element can be pre-calibrated, and the component forces of each contact surface of the fingertip, side, and back of the finger can be used as input to solve the target deformation of the corresponding joint elastic element in reverse, distinguishing the bidirectional deformation values ​​of flexion-extension and lateral swing.

[0082] See Figure 3 This is a schematic flowchart of a dexterous hand remote manipulation method applied to a dexterous hand, provided by an embodiment of this application. The method includes the following steps S301 to S303:

[0083] Step S301: Receive the second joint pose sent by the exoskeleton glove.

[0084] The second joint pose is obtained based on the first joint pose of the wearer's hand joints in the exoskeleton glove. The specific method for determining the second joint pose has been described in the aforementioned embodiments applied to the exoskeleton glove, and will not be repeated here.

[0085] Step S302: Control the hand joints of the dexterous hand to reach the second joint position.

[0086] Step S303: Receive tactile information collected by the contact force sensors set in each finger joint and send the tactile information to the exoskeleton glove.

[0087] As can be seen from the above, in the solution provided by this application embodiment, after the dexterous hand receives the second joint pose from the exoskeleton glove, it not only controls its own joints to reach the second joint pose, but also feeds back the tactile information it perceives to the exoskeleton glove. Thus, the exoskeleton glove can reproduce the object contact shape based on the feedback tactile information. It is evident that this solution integrates hand motion acquisition, cross-shape motion mapping, dexterous hand tactile feedback, and exoskeleton glove contour tactile reproduction into a complete closed loop. The wearer does not need to rely solely on vision to judge the contact state, and can perceive the object contact situation in real time and intuitively through the object contour reproduced by the exoskeleton glove, restoring the real tactile sensation of operation. This allows the wearer to perform more accurate and refined operations based on the contact situation, and transmits this operation to the dexterous hand, thereby improving the accuracy of dexterous hand control.

[0088] In order to reproduce the joint interaction forces caused by the compression and pulling of objects on the side of the exoskeleton glove, forming a dual perception loop of touch and force, and further enhancing the wearer's sense of control realism, in one possible implementation, the dexterous hand can also receive force information collected by the contact force sensors set on each finger joint and send the force information to the exoskeleton glove.

[0089] The specific types of contact force sensors, tactile information, and force information set in each joint of the dexterous hand have been explained in the previous text and will not be repeated here.

[0090] Corresponding to the above-described dexterous hand remote control method, this application also provides a dexterous hand remote control system.

[0091] See Figure 4 The aforementioned system includes an exoskeleton glove 401 and a dexterous hand 402, wherein,

[0092] The exoskeleton glove 401 is used to receive angle information collected by the angle information acquisition unit of the hand joint configuration of the exoskeleton glove 401, and based on the angle information, determine the first joint pose of the hand joint of the wearer wearing the exoskeleton glove 401; and use a motion redirection algorithm to map the first joint pose to the second joint pose of the hand joint of the dexterous hand 402, and send the second joint pose to the dexterous hand 402.

[0093] The dexterous hand 402 is used to receive the second joint pose sent by the exoskeleton glove 401; control the corresponding joint to reach the second joint pose; receive tactile information collected by the contact force sensors set in each finger joint, and send tactile information to the exoskeleton glove 401.

[0094] The exoskeleton glove 401 is also used to receive tactile information fed back by the dexterous hand 402, and based on the tactile information, to determine the contour information of the contact area of ​​the exoskeleton glove 401; based on the contour information of the contact area, to drive the tactile feedback unit set on the corresponding finger joint of the exoskeleton glove 401 to produce a protruding deformation.

[0095] As can be seen from the above, in the solution provided by this application embodiment, after the dexterous hand receives the second joint pose from the exoskeleton glove, it not only controls its own joints to reach the second joint pose, but also feeds back the tactile information it perceives to the exoskeleton glove. Thus, the exoskeleton glove can reproduce the object contact shape based on the feedback tactile information. It is evident that this solution integrates hand motion acquisition, cross-shape motion mapping, dexterous hand tactile feedback, and exoskeleton glove contour tactile reproduction into a complete closed loop. The wearer does not need to rely solely on vision to judge the contact state, and can perceive the object contact situation in real time and intuitively through the object contour reproduced by the exoskeleton glove, restoring the real tactile sensation of operation. This allows the wearer to perform more accurate and refined operations based on the contact situation, and transmits this operation to the dexterous hand, thereby improving the accuracy of dexterous hand control.

[0096] The dexterous hand remote control solution provided in the embodiments of this application will be described more intuitively below with reference to the accompanying drawings.

[0097] See Figure 5 The operator wears an exoskeleton glove to perform hand movements. The exoskeleton glove collects hand joint angle data in real time through a built-in angle acquisition unit. Based on the pre-stored human factors parameters of the operator, it calculates the operator's complete hand pose. Then, it performs motion remapping processing on the hand pose, converts it into a target joint pose that is adapted to the dexterous hand structure, and transmits the pose command to the dexterous hand.

[0098] After receiving posture commands, it drives its joints to synchronously replicate human hand movements. During contact with objects, it provides tactile and force feedback: through the full-range ring-type sensing components of each finger joint, it synchronously collects tactile data and force data from multiple contact surfaces, and transmits the two types of sensory information in reverse to the exoskeleton glove.

[0099] After receiving tactile and force information from the dexterous hand, the exoskeleton glove performs feedback reproduction in two ways: First, it analyzes the contact contour of the object based on the tactile information and drives the fingertip tactile feedback unit to produce local protrusion deformation, allowing the operator to perceive the shape of the contact area; second, it calculates the resistance output of the corresponding joint based on the force data of each finger joint and controls the elastic drive unit of each joint to generate interactive resistance, reproducing the force sensation of the hand caused by the object's compression, sliding, and lateral pressure.

[0100] The operator perceives the real contact state based on the tactile contour and joint resistance feedback output by the exoskeleton glove, and autonomously adjusts the hand posture. The exoskeleton then collects new joint angle data and repeats all the above interaction steps to continuously form a real-time two-way teleoperation closed loop.

[0101] Corresponding to the above-described dexterous hand remote control method applied to exoskeleton gloves, this application also provides a dexterous hand remote control device applied to exoskeleton gloves.

[0102] See Figure 6 This is a schematic diagram of a dexterous hand remote control device for an exoskeleton glove, provided in an embodiment of this application. The device includes:

[0103] The first pose determination module 601 is used to receive angle information collected by the angle information acquisition unit of the hand joint configuration of the exoskeleton glove, and determine the first joint pose of the hand joint of the wearer wearing the exoskeleton glove based on the angle information.

[0104] The second pose determination module 602 is used to use an action redirection algorithm to map the first joint pose to the second joint pose of the dexterous hand, and send the second joint pose to the dexterous hand so that the dexterous hand reaches the second joint pose.

[0105] The contour information determination module 603 is used to receive tactile information fed back by the dexterous hand and determine the contour information of the contact area of ​​the exoskeleton glove based on the tactile information.

[0106] The contour reproduction module 604 is used to drive the tactile feedback unit set on the corresponding finger joint of the exoskeleton glove to generate a raised deformation based on the contour information of the contact area.

[0107] As can be seen from the above, in the solution provided by this application embodiment, the exoskeleton glove calculates the first joint pose of the wearer's real hand by collecting the angles of each joint; then, through motion redirection, the real hand pose is mapped to the second joint pose of the dexterous hand and sent down; next, it receives the tactile data returned by the dexterous hand and maps the contact contour of the dexterous hand to the contact area contour of the exoskeleton glove; finally, it can drive the local protrusion of the corresponding finger joint tactile unit to reproduce the object contact shape on the exoskeleton glove. It is evident that this solution integrates hand motion acquisition, cross-shape motion mapping, dexterous hand tactile feedback, and exoskeleton glove contour tactile reproduction into a complete closed loop. The wearer does not need to rely solely on vision to judge the contact state and can perceive the object contact situation in real time and intuitively through the object contour reproduced by the exoskeleton glove, restoring the real tactile sensation of operation. This allows the wearer to perform more accurate and refined operations based on the contact situation and transmits this operation to the dexterous hand, thereby improving the accuracy of dexterous hand control.

[0108] In one possible implementation, the exoskeleton glove's hand joints are provided with elastic drive units, and the device is further used for:

[0109] Receive force feedback from the dexterous hand;

[0110] Based on the force information, the elastic drive unit of the corresponding joint of the exoskeleton glove is driven to move.

[0111] In this way, the joint interaction forces caused by the squeezing and pulling of objects can be reproduced, forming a dual sensory loop of touch and force, further enhancing the wearer's sense of control realism.

[0112] In one possible implementation, the elastic drive unit is electrically connected to an external drive motor, and the step of driving the elastic drive unit of the exoskeleton glove to move based on the force information includes:

[0113] Based on the contact force data corresponding to each finger joint of the exoskeleton glove, the target deformation of the elastic drive unit of the corresponding joint of the exoskeleton glove is determined.

[0114] Based on the target deformation, drive commands are generated for the drive motor connected to the elastic drive unit;

[0115] The drive command is sent to the drive motor, causing the drive motor to control the elastic drive unit to produce the target deformation via a transmission rope. This simplifies the structure of the exoskeleton glove and improves the wearer's control precision.

[0116] In one possible implementation, the exoskeleton glove is equipped with a segmented knuckle self-alignment slide rail. The wearer aligns their own knuckles with the knuckles of the exoskeleton glove by adjusting the position of the slider in the segmented knuckle self-alignment slide rail. Determining the first joint pose of the wearer's hand joints based on the angle information includes:

[0117] The angle information is substituted into the pre-constructed kinematic model of the wearer's hand to obtain the first joint pose of the wearer's hand joint. The kinematic model of the wearer's hand is constructed based on the wearer's human factors parameters, which represent the wearer's actual hand parameters.

[0118] In this way, the individualized human factors parameters of the wearer are calibrated in a targeted manner, thereby reducing the impact of individual hand shape differences on the determination of hand position and improving the accuracy of the determined hand position of the wearer.

[0119] Corresponding to the above-described dexterity hand remote control method applied to dexterity hands, this application also provides a dexterity hand remote control device applied to dexterity hands.

[0120] See Figure 7 This is a schematic diagram of a dexterous hand remote control device for a dexterous hand, provided in an embodiment of this application. The device includes:

[0121] The pose receiving module 701 receives the second joint pose sent by the exoskeleton glove. The second joint pose is obtained based on the first joint pose of the wearer's hand joints.

[0122] The pose control module 702 is used to control the corresponding joint to achieve the second joint pose.

[0123] The information feedback module 703 is used to receive tactile information collected by the contact force sensors set in each finger joint and send the tactile information to the exoskeleton glove.

[0124] As can be seen from the above, in the solution provided by this application embodiment, after the dexterous hand receives the second joint pose from the exoskeleton glove, it not only controls its own joints to reach the second joint pose, but also feeds back the tactile information it perceives to the exoskeleton glove. Thus, the exoskeleton glove can reproduce the object contact shape based on the feedback tactile information. It is evident that this solution integrates hand motion acquisition, cross-shape motion mapping, dexterous hand tactile feedback, and exoskeleton glove contour tactile reproduction into a complete closed loop. The wearer does not need to rely solely on vision to judge the contact state, and can perceive the object contact situation in real time and intuitively through the object contour reproduced by the exoskeleton glove, restoring the real tactile sensation of operation. This allows the wearer to perform more accurate and refined operations based on the contact situation, and transmits this operation to the dexterous hand, thereby improving the accuracy of dexterous hand control.

[0125] In one possible implementation, the device is further configured to: receive force information collected by contact force sensors located on each finger joint, and send the force information to the exoskeleton glove.

[0126] In this way, the joint interaction forces caused by the compression and pulling of objects can be reproduced on the side of the exoskeleton glove, forming a dual sensory loop of touch and force, further enhancing the wearer's sense of control realism.

[0127] Corresponding to the above-described method for determining hand pose, this application also provides an electronic device, a storage medium, and a program product.

[0128] This application also provides an electronic device, such as... Figure 8As shown, the device includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, communication interface 802, and memory 803 communicate with each other via the communication bus 804. The memory 803 stores computer programs; the processor 801 executes the programs stored in the memory 803 to implement the aforementioned dexterous hand remote control method. The electronic device can be a charging compartment device, a battery transfer device, or a robot. The communication bus mentioned in the electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the electronic device and other devices. The memory can include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor. The aforementioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0129] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described dexterous hand telemanipulation methods.

[0130] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the dexterous hand remote control methods described above.

[0131] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0132] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0133] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system, apparatus, electronic device, and storage medium embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0134] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for dexterous hand-operated remote control, characterized in that, The method, applied to exoskeleton gloves, includes: The system receives angle information collected by the angle information acquisition unit configured in the hand joints of the exoskeleton glove, and determines the first joint pose of the wearer's hand joints based on the angle information. An action redirection algorithm is used to map the first joint pose to the second joint pose of the dexterous hand to be manipulated, and the second joint pose is sent to the dexterous hand to manipulate the dexterous hand to reach the second joint pose. Receive tactile information from the dexterous hand and determine the contact area contour information of the exoskeleton glove based on the tactile information; Based on the contour information of the contact area, the tactile feedback unit set on the corresponding finger joint of the exoskeleton glove is driven to produce a raised deformation.

2. The method according to claim 1, characterized in that, The exoskeleton glove has an elastic drive unit at its hand joints, and the method further includes: Receive force feedback from the dexterous hand; Based on the force information, the elastic drive unit of the corresponding joint of the exoskeleton glove is driven to move.

3. The method according to claim 2, characterized in that, The elastic drive unit is electrically connected to an external drive motor. The step of driving the elastic drive unit of the corresponding joint of the exoskeleton glove to move based on the force information includes: Based on the contact force data corresponding to each finger joint of the exoskeleton glove, the target deformation of the elastic drive unit of the corresponding joint of the exoskeleton glove is determined. Based on the target deformation, drive commands are generated for the drive motor connected to the elastic drive unit; The drive command is sent to the drive motor, causing the drive motor to control the elastic drive unit to generate the target deformation via the transmission rope.

4. The method according to any one of claims 1 to 3, characterized in that, The exoskeleton glove is equipped with a segmented knuckle self-alignment rail. The wearer aligns their own knuckles with the knuckles of the exoskeleton glove by adjusting the position of the slider in the segmented knuckle self-alignment rail. The step of determining the first joint position of the wearer's hand joints based on the angle information includes: Substituting the angle information into the constructed kinematic model of the wearer's hand, the first joint pose of the wearer's hand joint is obtained, wherein the kinematic model of the wearer's hand is constructed based on the wearer's personalized hand information, and the personalized hand information represents the wearer's actual hand parameters; And / or, The step of employing a motion redirection algorithm to map the first joint pose to the second joint pose of the dexterous hand to be manipulated includes: Based on the spatial distance between the dexterous hand to be controlled and the task target, the operation stage of the task performed by the dexterous hand is determined; based on the operation stage, the optimized weights of each pose component in the first joint pose are determined; based on the optimized weights of each pose component, the pose components are weighted and fused to obtain the second joint pose of the dexterous hand joint.

5. A method for dexterous hand-operated remote control, characterized in that, Applied to dexterous hands, the method includes: Receive a second joint pose sent by the exoskeleton glove, the second joint pose being obtained based on a first joint pose of the wearer's hand joints; Control the hand joints of the dexterous hand to achieve the second joint position; The system receives tactile information collected by contact force sensors on each finger joint and sends the tactile information to the exoskeleton glove.

6. The method according to claim 5, characterized in that, The method further includes: It receives force information collected by contact force sensors on each finger joint and sends the force information to the exoskeleton glove.

7. A dexterous hand-operated remote control system, characterized in that, Including exoskeleton gloves and dexterous hands, among which, The exoskeleton glove is used to receive angle information collected by the angle information acquisition unit configured on the hand joints of the exoskeleton glove, and based on the angle information, determine the first joint pose of the hand joints of the wearer wearing the exoskeleton glove; and use a motion redirection algorithm to map the first joint pose to the second joint pose of the dexterous hand to be controlled, and send the second joint pose to the dexterous hand. The dexterous hand is used to receive the second joint pose sent by the exoskeleton glove; control the hand joints of the dexterous hand to reach the second joint pose; receive tactile information collected by the contact force sensors set on each finger joint, and send the tactile information to the exoskeleton glove; The exoskeleton glove is also used to receive tactile information from the dexterous hand, determine the contact area contour information of the exoskeleton glove based on the tactile information, and drive the tactile feedback unit set on the corresponding finger joint of the exoskeleton glove to generate a protruding deformation based on the contact area contour information.

8. A dexterous hand-operated remote control device, characterized in that, The device, used in exoskeleton gloves, includes: The first pose determination module is used to receive angle information collected by the angle information acquisition unit configured in the hand joints of the exoskeleton glove, and determine the first joint pose of the hand joints of the wearer wearing the exoskeleton glove based on the angle information. The second pose determination module is used to map the first joint pose to the second joint pose of the dexterous hand joint to be controlled using a motion redirection algorithm, and send the second joint pose to the dexterous hand to control the dexterous hand to reach the second joint pose. The contour information determination module is used to receive tactile information fed back by the dexterous hand, and determine the contour information of the contact area of ​​the exoskeleton glove based on the tactile information. The contour reproduction module is used to drive the tactile feedback unit set on the corresponding finger joint of the exoskeleton glove to generate a raised deformation based on the contour information of the contact area.

9. A dexterous hand-operated remote control device, characterized in that, For use with dexterous hands, the device includes: The pose receiving module receives the second joint pose sent by the exoskeleton glove; The pose control module is used to enable the hand joints of the dexterous hand to achieve the second joint pose; The information feedback module is used to receive tactile information collected by the contact force sensors set in each finger joint and send the tactile information to the exoskeleton glove.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, a communication bus, and an image acquisition device. The processor, communication interface, and memory communicate with each other through the communication bus. Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1 to 4, or the method of any one of claims 5 to 6.

Images