Bionic dexterous hand control method and system based on grasping intention recognition

Abstract

The application discloses a bionic dexterous hand control method and system based on grabbing intention recognition and belongs to the technical field of robot grabbing control. The method comprises the following steps: acquiring sensing data of a target object, adopting an RGB-D main sensor and a laser radar auxiliary sensor to form a double-redundancy sensing architecture, taking a wrist coordinate origin as a spherical center, determining a safety boundary by superimposing an inflation threshold on the maximum stretching distance of the whole hand, and dividing a functional area and a non-functional area of a grabbing space; extracting appearance features such as the shape, size and structure of the object, mapping the appearance features to a force mode and a contact mode according to a preset intention matching rule, and determining a grabbing intention; selecting and executing a pinch mode, a grip mode or a grab mode according to the grabbing intention, wherein the pinch mode takes two fingers as carriers, constant force clamping is realized through three-stage path planning and proportional integral force control, the grip mode realizes stable holding through five-finger contact area calculation and multi-finger force control coordination, and the grab mode realizes accurate grabbing through palm positioning and geometric error control. The application realizes an interpretable and traceable grabbing decision in a rule-driven manner, does not need a large amount of training data, and improves the intelligent level and operation reliability of dexterous hand grabbing.

Description

Technical Field

[0001] This application relates to the field of robot grasping control technology, and in particular to a bionic dexterous hand control method, system, terminal and storage medium based on grasping intent recognition. Background Technology

[0002] In fields such as intelligent manufacturing and disability rehabilitation, the grasping control of bionic dexterous hands is the core of achieving autonomous operation. However, automatically selecting the appropriate grasping method when faced with objects of different shapes, sizes, and structures remains a challenge in this field.

[0003] Current dexterous hand grasping strategies primarily rely on deep learning-based grasping posture generation methods. For example, patent document CN121912362A discloses a deep learning-based underactuated dexterous hand grasping pose generation method. This method utilizes a large language model and a point cloud encoder to extract the semantic-geometric joint features of the object, constructing classification models for three grasping strategies: two-finger pinch, three-finger pinch, and five-finger grasping. It then uses a neural network to predict the six-dimensional pose and joint angles of the dexterous hand. However, this type of method relies on large amounts of labeled data and offline training, resulting in output results that lack interpretability, making it difficult to trace the basis for the selection of the grasping method. Furthermore, the mapping relationship between the generated grasping strategy and the appearance features (shape, size, structure) of the specific object is unclear.

[0004] Therefore, there is an urgent need for an interpretable and low-cost method for recognizing grasping intent, which can directly determine the grasping pattern based on the appearance features of the object and provide a corresponding engineering control scheme. Summary of the Invention

[0005] This invention addresses the problem that existing technologies rely on deep learning models for dexterous hand grasping strategy selection and lack interpretability. It provides a bionic dexterous hand control method, system, terminal, and storage medium based on grasping intent recognition, enabling interpretable and rule-based decision-making from object appearance features to grasping patterns.

[0006] The present invention adopts the following technical solution:

[0007] A bionic dexterous hand control method based on grasping intent recognition includes the following steps:

[0008] Step 1: Obtain the perception data of the target object, extract the three-dimensional spatial position information of the target object, and divide the grasping space into functional and non-functional areas.

[0009] Step 2: Extract the appearance features of the target object, which include at least one of shape, size, and structure; according to the preset intent matching rules, map the appearance features to the corresponding force application method and contact method to determine the grasping intent;

[0010] Step 3: Select and execute the corresponding grasping mode according to the grasping intention; the grasping modes include pinching mode, gripping mode and grabbing mode; among them, pinching mode uses two fingers as the execution carrier to complete the gripping and extraction; gripping mode uses all five fingers in full contact with the palm to complete the gripping and extraction; grabbing mode uses four fingers to form a claw with the fingertips for support to complete the grasping.

[0011] By extracting the appearance features of the object and combining them with intent matching rules, the grasping mode is determined in a rule-driven manner. This eliminates the need for a large amount of training data, and the decision-making process is interpretable and traceable. At the same time, it provides a complete control link from perception and recognition to execution, and designs motion planning and force control schemes for the three grasping modes respectively, which improves the intelligence level and operational reliability of dexterous hand grasping.

[0012] More preferably, in step 2, the intent matching rule includes: feature classification based on the shape, size, and structure of the target object; wherein, shape classification includes at least one of irregular curved surfaces, columnar bodies, objects with sharp edges and grooves; size classification includes large objects and small objects; and structure classification includes at least one of objects with sharp edges, grooves, and parts.

[0013] The corresponding force application method and contact method are matched according to the classification results. Among them, the force application method includes two-finger force application, multi-finger force application and full-finger force application. Two fingers refer to the index finger and thumb, multi-finger force application refers to the index finger, middle finger, ring finger and little finger, and full-finger force application refers to all fingers. The contact method includes large-area contact and small-area contact. Large-area contact is the surface contact formed between the fingertip and the object, and small-area contact is the point contact between the fingertip and the object.

[0014] When the force is applied using two fingers and the contact area is large, the grasping intention is mapped to the pinching mode; when the force is applied using all fingers and the contact area is large, the grasping intention is mapped to the gripping mode; when the force is applied using multiple fingers and the contact area is small, the grasping intention is mapped to the grabbing mode; when the force is applied using two fingers and the contact area is small, it is not mapped to any grasping mode.

[0015] By using preset intent matching rules, the appearance features of objects are directly mapped to the corresponding grasping modes, enabling interpretable decision-making for grasping strategies without relying on training data.

[0016] More preferably, in step 1, an RGB-D sensor is used as the main sensor and a LiDAR is used as the auxiliary sensor; when the main sensor is working normally, the auxiliary sensor is turned off; when the main sensor fails or its sensing fails, the auxiliary sensor is activated, and the LiDAR generates a sparse environmental point cloud to perform anti-collision, limit, and safety termination actions.

[0017] By employing a dual-sensor redundancy architecture, the system automatically switches to the auxiliary sensor and performs safety actions when the primary sensor fails, thereby improving the system's safety in the event of a sensing failure.

[0018] More preferably, in step 1, the wrist coordinate origin is used as the center of the sphere, the maximum extension distance of the whole hand is used as the base radius, and the expansion radius is obtained by superimposing the expansion threshold on the base radius. The expansion radius is used to determine the safe boundary of the grasping space. The functional area is the area determined by expanding based on the perceived object, and the area outside the functional area within the safe boundary is the non-functional area.

[0019] By defining safety boundaries, a clear boundary is formed between the safe movement area and the unsafe area of ​​the dexterous hand in the grasping space, avoiding the risk of collisions and boundary crossings.

[0020] More preferably, in step 3, the pinching mode includes: using the index finger and thumb as the execution carrier, while the other three fingers remain curled; the grasping is completed through a three-stage motion path planning, which includes an approach stage, a contact stage, and a pinching stage; in the contact stage, proportional-integral adjustment is used to control the contact force between the index finger and thumb to achieve constant clamping force control, and the maximum static friction is monitored in real time to prevent slippage; in the pinching stage, the wrist is lifted by forming a stable triangle posture with the center point of the index finger contact surface, the center point of the thumb contact surface, and the origin of the wrist coordinates.

[0021] By employing a three-stage motion path planning system for the pinching mode, proportional-integral force control for constant force clamping, and anti-slip monitoring, stable clamping and reliable extraction of small objects are achieved.

[0022] More preferably, in step 3, the gripping mode includes: using full contact of all five fingers in conjunction with palm contact as the execution method, calculating the three-dimensional coordinates of each finger joint in the wrist space, dividing the pentagon formed by the contact points of each finger into three triangles, calculating the area of ​​each triangle by vector cross product and then summing them to obtain the contact area; maintaining uniform clamping force at each contact point through multi-finger force control coordination strategy and proportional integral adjustment, and keeping the center of gravity of the object unchanged by adjusting the force at each contact point.

[0023] By calculating the contact area of ​​the five fingers and coordinating multi-finger force control for gripping modes, stable gripping of cylindrical and large objects is achieved.

[0024] More preferably, in step 3, the grasping mode includes: using four fingers forming a claw and the thumb touching as the execution method; forming a planar quadrilateral by the midpoint of the link between the index finger, little finger, and thumb and the origin of the wrist coordinates; determining the real-time position of the palm by the intersection of the diagonals of the quadrilateral; adjusting the geometric error by the deviation from the target position and the deviation between the fingertips and the corresponding target position; and using force control to constrain the judgment when lifting the object, limiting the force within a controllable range.

[0025] By adjusting the palm positioning and geometric error control for the gripping mode, it can accurately grasp objects with grooves or specific structures.

[0026] In another aspect, the present invention provides a bionic dexterous hand control system based on grasping intent recognition, comprising:

[0027] The perception module is used to acquire perception data of the target object, extract the three-dimensional spatial position information of the target object, and divide the grasping space into functional and non-functional areas.

[0028] The intent recognition module is used to extract the appearance features of the target object, including at least one of shape, size and structure, and to map the appearance features to the corresponding force application mode and contact mode according to the preset intent matching rules to determine the grasping intent.

[0029] The mode execution module is used to select and execute the corresponding grasping mode according to the grasping intent. The grasping modes include pinch mode, gripping mode and grab mode.

[0030] Through the modular design described above, complete functional coverage from perception and recognition to control is achieved.

[0031] The present invention also provides a bionic dexterous hand control terminal, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the steps of the above method.

[0032] The present invention also provides a computer-readable storage medium storing computer instructions that, when loaded by a processor, execute the steps of the above-described method. Attached Figure Description

[0033] Figure 1 This is a flowchart of the bionic dexterous hand control method in an embodiment of the present invention.

[0034] Figure 2 This is a flowchart illustrating the decision-making process for intent-matching rules in an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of the spatial safety boundary division in an embodiment of the present invention.

[0036] Figure 4 This is a schematic diagram of the three-stage movement of the pinching mode in an embodiment of the present invention.

[0037] Figure 5 This is a schematic diagram of the bionic dexterous hand structure in an embodiment of the present invention.

[0038] Figure 6 This is a schematic diagram of palm positioning in the grasping mode of this invention.

[0039] Figure 7 This is a schematic diagram illustrating the calculation of the five-point contact area in the gripping mode according to an embodiment of the present invention.

[0040] Figure 8 This is a block diagram of the bionic dexterous hand control system in an embodiment of the present invention. Detailed Implementation

[0041] Example 1

[0042] The overall process is as follows Figure 1 As shown, this embodiment provides a bionic dexterous hand control method based on grasping intent recognition, including the following steps.

[0043] Step 1: Perception and Coordinate System Transformation.

[0044] An RGB-D sensor is used as the primary sensor, and a LiDAR sensor as the auxiliary sensor, forming a redundant dual-sensor architecture. Upon power-up, the system prioritizes self-checking the RGB-D sensor. If hardware communication is normal and there are no large-area gaps or frame drops in the depth data stream, the primary sensor is deemed usable. At this point, the LiDAR is put into sleep mode and powered off, with all sensing data provided by the RGB-D sensor to reduce overall power consumption and computing power usage. The controller polls the RGB-D status in real time. In the event of hardware failure (camera communication interruption, power supply anomaly), sensing failure (large-area loss of depth of target objects within the field of view, image obstruction), or environmental failure (strong light or backlight causing depth accuracy to exceed limits), the LiDAR is immediately activated. A sparse environmental point cloud is generated, and detailed grasping planning is abandoned. Only basic safety actions such as collision avoidance, limit positioning, and slow retraction of the gripper are retained to prevent the gripper from colliding with objects or damaging the surrounding environment, thus safely terminating the grasping task.

[0045] Coordinate System Definition and Transformation: The wrist coordinate system has its origin at the center of the bionic wrist's dual bones. The positive y-axis points perpendicular to the palm, and the positive x-axis points towards the thumb. Using the right-hand rule, the positive z-axis points towards the robotic arm. The origin of the robotic arm's end-effector coordinate system coincides with the wrist coordinate origin. Its x-axis is perpendicular to the flange, and its y-axis points towards the thumb along the flange's tangent plane. The positive z-axis is obtained using the right-hand rule. The transformation relationship between the target wrist coordinate system and the robotic arm's end-effector coordinate system is: wrist x-axis = robotic arm end-effector y-axis, wrist y-axis = robotic arm end-effector - z-axis, wrist z-axis = robotic arm end-effector - x-axis. The corresponding rotation matrix R is:

[0046]

[0047] Verification using the identity matrix:

[0048]

[0049] A secondary 2D map is generated based on the wrist coordinate system. Combined with camera intrinsic parameters, 2D pixel values ​​and depth values ​​are converted into 3D spatial coordinates to generate point cloud data corresponding to the target object, achieving target recognition and 3D point cloud extraction based on visual and depth information. Hand-eye calibration is performed using the RGB-D sensor's intrinsic parameter matrix. The pose transformation matrix between the camera and the robotic arm is pre-solved using a calibration board, mapping the camera space point cloud to the robotic arm's global space, enabling the robotic arm controller to obtain the precise position of the object in the workspace.

[0050] Safety boundary delineation: such as Figure 3 As shown, the grasping space is divided into functional and non-functional zones. Using the wrist coordinate origin as the center of the spatial sphere and the maximum extension distance of the entire hand as the base radius, an expansion radius is obtained by superimposing an expansion threshold onto the base radius, forming a safety boundary. The functional zone expands based on the scanned object, and the boundary between the functional and non-functional zones serves as the limit of safe planning for the dexterous wrist's movement. Within the functional zone, the dexterous hand can quickly reach or withdraw from the target area, with a large path planning range. In the non-functional zone, the dexterous hand maintains its initial shape and does not change unless a collision occurs.

[0051] The formula for quantifying the radius of expansion:

[0052]

[0053] in, The outer radius of expansion. This is the distance from the wrist coordinate origin to the maximum extension of the entire hand. The expansion threshold is used to compensate for the deviation in the position of the fully extended or functionally extended end of the hand caused by external unstable factors (such as vibration, inertial impact, etc.) during the operation of the robotic arm. For example, the expansion threshold can be 1cm to 2cm. In actual deployment, it can be preset according to the working conditions of the robotic arm. The larger the threshold, the more conservative the safety boundary.

[0054] Step 2: Object feature extraction and crawling intent matching.

[0055] This step, based on a rule-based and feature-extraction-driven decision-making process, achieves crawling intent recognition. The core logic is: extracting appearance features such as shape, size, and structure; matching them with preset patterns; and categorizing them into one of three crawling modes. The decision-making process is as follows: Figure 2 As shown.

[0056] The appearance features of the point cloud data and image data of the target object are extracted, and three major appearance features are extracted: (1) Shape determination: the target object is classified into one or more of the following: irregular curved surface, column, edge, groove; (2) Size determination: the target object is distinguished as a large object or a small object, and small area contact includes point contact; (3) Structure determination: whether the target object has complex structures such as edge, groove, and parts.

[0057] Let shape feature be A1, size feature be A2, structural feature be A3, force application method be B1, and contact method be B2. Then the grasping mode category C is determined by the following formula:

[0058] C = f(A1,A2,A3) + g(B1,B2)

[0059] Where f(·) is the appearance feature classification piecewise function, which maps the appearance feature vector (A1,A2,A3) to the classification number according to the preset feature category; g(·) is the force contact matching piecewise function, which directly maps the combination of force application mode B1 and contact mode B2 to the corresponding grasping mode; "+" indicates the joint decision of f(·) and g(·), rather than the numerical addition.

[0060] The mapping relationship between force application method, contact method, and grasping mode is shown in the table below:

[0061]

[0062] In this embodiment, the force application methods include two-finger force application, multi-finger force application, and full-finger force application. Two fingers refer to the index finger and thumb, multi-finger force application refers to the index finger, middle finger, ring finger, and little finger, and full-finger force application refers to all fingers. The contact methods include large-area contact and small-area contact. Large-area contact is the surface contact formed between the fingertip and the object, and small-area contact is the point contact between the fingertip and the object.

[0063] In other words: when the force is applied with two fingers and the contact area is large, the grasping intention is mapped to the pinching mode; when the force is applied with all fingers and the contact area is large, the grasping intention is mapped to the gripping mode, and the palm must be in contact with the object's surface; when the force is applied with multiple fingers and the contact area is small, the grasping intention is mapped to the grabbing mode; when the force is applied with two fingers and the contact area is small, it is not mapped to any grasping mode.

[0064] By using the above intent matching rules, the appearance features of the target object are directly mapped to the corresponding grasping mode. The decision-making process is interpretable and traceable, and does not rely on a large amount of training data.

[0065] Step 3: Select and execute the capture mode.

[0066] Based on the grasping intent determined in step 2, select the corresponding grasping mode and execute it. Grasping modes include pinching mode, gripping mode, and grasping mode: pinching mode uses two fingers as the execution carrier to complete gripping and extraction; gripping mode uses all five fingers in full contact with the palm to complete gripping and extraction; grasping mode uses four fingers forming a claw with fingertip support to complete grasping.

[0067] The application of each mode is based on the functional and non-functional areas defined in step 1. After the bionic hand reaches the boundary between the functional and non-functional areas, it executes the corresponding grasping mode. Among them, the pinching mode is suitable for small, regular objects or objects that can be held between two fingers; the gripping mode is suitable for cylindrical or large objects; and the grasping mode is suitable for objects with grooves or specific structures.

[0068] The specific implementation methods of each grasping mode in terms of motion path planning, contact force control and stability maintenance are described in Examples 2, 3 and 4 below.

[0069] Example 2

[0070] This embodiment provides a specific implementation of the pinching mode in step 3. The pinching mode is suitable for small, regular-shaped objects or objects that can be held between two fingers. The index finger and thumb act as the actuating force, while the other three fingers remain curled. The fingertip of the index finger and its first joint, and the fingertip of the thumb and its first joint, form the contact area. The core of the pinching mode is the planning of the movement path of the two fingers, constant force clamping, and anti-slip monitoring, such as... Figure 4 As shown.

[0071] (a) Calculation of three-dimensional coordinates of the point of force

[0072] Based on the mechanical structure of the dexterous hand, this mode has two centers of force: the center between the index fingertip 10 and the index finger's first joint motor 11 (force point one 14), and the center between the thumbtip 50 and the thumb's first joint motor 51 (force point two 54). The three-dimensional coordinates of the dexterous hand are all based on the wrist coordinate system, which is a three-dimensional coordinate system with the wrist motor 6 as its origin. The direction perpendicular to the palm is defined as the positive y-axis, the direction pointing towards the thumb as the positive x-axis, and the direction pointing towards the robotic arm using the right-hand rule as the positive z-axis. The three-dimensional coordinates of the force points are calculated using the wrist coordinate system as a reference.

[0073] Taking the index finger as an example, the three-dimensional coordinates of the point of force 14 are calculated as follows:

[0074]

[0075] in, The three-dimensional coordinates of the index fingertip are 10. The three-dimensional coordinates of the motor of the first joint of the index finger; The center point between the tip of the index finger and the first joint of the index finger.

[0076] The three-dimensional coordinates of the second point (54) were calculated in the same way.

[0077] (II) Three-stage motion path planning

[0078] Set the spatial point of the index fingertip using the wrist coordinates as a reference system. Spatial point with the tip of the thumb The spatial-temporal sequence of the two fingertips during the entire pinching process is as follows: The spatial-temporal sequence of the index fingertip trajectory is as follows: The spatial-temporal sequence of the thumb tip trajectory is Where, subscript 1 represents the index finger, subscript 5 represents the thumb; t is the motion time parameter; , , These are functions representing the components of the index fingertip position in the X, Y, and Z coordinate directions as a function of time. , , Similarly, it is a function of the components of the thumb tip position in the X, Y, and Z coordinate directions as a function of time.

[0079] Based on the above spatial-temporal sequence, the pinching action is divided into the following three stages: approach stage, contact stage, and pinching stage.

[0080] Approaching phase ( ): Move the tips of both fingers toward the center of force and enter the grasping position of the object.

[0081] Adhesion stage ( ): By calculating the contact position between the two fingers and the object, the plane containing the index finger and thumb is adjusted so that the plane is parallel to the plane containing the corresponding point of the contact plane of the object, thus completing the contact between the two fingers and the object.

[0082] Pinch stage ( ): The wrist joint motor, the index finger first joint motor, and the thumb first joint motor output corresponding forces to lift the object upwards.

[0083] in, This marks the boundary between the approach phase and the bonding phase. This marks the dividing point between the bonding phase and the pinching phase. This represents the total duration of a single pinch action.

[0084] The above three stages require no collisions throughout, meaning that the two fingers must not collide with objects or the surrounding environment during the movement, and the movement in each stage must occur within the safety boundaries defined in step 1. Simultaneously, the three-stage movement process must satisfy the following motion constraints:

[0085]

[0086] Motion constraints include boundary conditions and kinematic limit constraints:

[0087] (1) Boundary conditions: At the start of the motion ( The tips of the index finger and thumb are in their respective initial positions. , Near the end of the phase ( The tips of the two fingers reach the target position of the approach stage respectively. , ;

[0088] (2) Velocity and acceleration constraints: The velocity and acceleration of the fingertips must not exceed the required limit values, i.e. , (i=1,5), where For the maximum permissible speed, This is the maximum permissible acceleration.

[0089] (III) Force-controlled constant force clamping

[0090] The system maintains a constant force-controlled grip after two fingers contact the object, simulating the pinching motion of a human hand. The contact force applied when the index finger contacts the object is set to... The contact force applied when the thumb makes contact with the object is The error in the contact force balance between the two fingers is defined as:

[0091]

[0092] The contact force between the two fingers is controlled by proportional-integral adjustment to minimize the error. As the force approaches zero, the contact force between the two fingers tends to be equal and remains constant.

[0093]

[0094] in, For contact force adjustment, This is the proportionality coefficient. The integral coefficient is... The variable is the integral. Proportional-integral control balances the error using the contact force of two fingers. As input, the output is the adjustment amount of the contact force between the two fingers, so that the force exerted by the two fingers on the target point remains constant, thereby achieving constant clamping force control.

[0095] (iv) Anti-slip monitoring

[0096] The main factor maintaining stability in a pinch is the maximum static friction. Let the coefficient of friction between the object and the fingers be μ, and the normal forces at the contact points between the two fingers (index finger and thumb) and the object be... and The frictional force between the two fingers (index finger and thumb) and the contact surface of the object is... and Real-time monitoring of non-slip conditions:

[0097]

[0098] in The frictional force is along the contact surface of the objects. denoted as normal force, i=1 for index finger, i=5 for thumb, and μ for coefficient of friction.

[0099] When the friction force at the contact point between the two fingers is detected to be close to the maximum static friction force, it is determined that there is a risk of slippage. The risk of slippage is eliminated by adjusting the contact force to ensure that the object being held will not slip.

[0100] (v) Stability maintenance

[0101] During the pinching phase, the finger is lifted in a stable triangular posture formed by the center point of the index finger contact surface, the center point of the thumb contact surface, and the origin of the wrist coordinate system. It is determined in real time that the center points of the two fingers contact surfaces and the origin of the wrist coordinate system cannot be in a straight line.

[0102] By using the above-mentioned two-finger movement path planning, force-controlled constant force clamping, and anti-slip monitoring based on maximum static friction, the stability and safety of the pinching process are ensured.

[0103] Example 3

[0104] This embodiment provides a specific implementation of the gripping mode in step 3. The gripping mode is suitable for cylindrical and large objects, using full contact of all five fingers combined with palm contact as the execution method, and wrist coordination to complete gripping and lifting. This embodiment transforms the experiential operational process of "full grasp—lifting—holding steadily" from traditional Chinese medicine gripping techniques into a robot-executable control strategy. The core is the calculation of multi-finger contact area and the coordination of multi-finger force control, achieving full-coverage gripping of cylindrical objects, such as... Figure 5 As shown.

[0105] (I) Calculation of three-dimensional coordinates of each finger joint

[0106] Based on the position information obtained from the dexterous hand's body sensors, the dexterous hand's three-dimensional coordinates are all based on the wrist coordinate system. The wrist coordinate system is a three-dimensional coordinate system with the wrist motor 6 as the origin. Let the direction perpendicular to the palm be the positive y-axis, the direction pointing to the thumb be the positive x-axis, and the direction pointing to the robotic arm be the positive z-axis according to the right-hand rule. Each joint of each finger is calculated in three-dimensional coordinates using the wrist coordinate system as the reference system. Specifically, the index, middle, ring, and little fingers each consist of 3 joint motors, and the thumb consists of 2 joint motors.

[0107] like Figure 5 As shown, the index finger has three motors: the index finger first joint motor 11, the index finger second joint motor 12, and the index finger third joint motor 13. The first segment is from the fingertip 10 to the index finger first joint motor 11, the second segment is from the index finger first joint motor 11 to the index finger second joint motor 12, and the third segment is from the index finger second joint motor 12 to the index finger third joint motor 13.

[0108] ① Index finger (finger number i=1, three segments in total)

[0109] Coordinates of the midpoint of the first segment of the index finger:

[0110]

[0111] Coordinates of the midpoint of the second segment of the index finger:

[0112]

[0113] Coordinates of the midpoint of the third segment of the index finger:

[0114]

[0115] in The three-dimensional coordinates of the index fingertip. , , These are the three-dimensional coordinates of the motors at the first, second, and third joints of the index finger, respectively. Let be the coordinates of the midpoint of the i-th segment and the m-th segment.

[0116] ② Middle finger (finger number i=2, three segments in total)

[0117] The middle finger has three motors: the first joint motor 21, the second joint motor 22, and the third joint motor 23. The middle finger is divided into three segments: the first segment from the fingertip 20 to the first joint motor 21, the second segment from the first joint motor 21 to the second joint motor 22, and the third segment from the second joint motor 22 to the third joint motor 23.

[0118] Coordinates of the midpoint of the first segment of the middle finger:

[0119]

[0120] The coordinates of the midpoint of the second segment of the middle finger:

[0121]

[0122] The coordinates of the midpoint of the third segment of the middle finger:

[0123]

[0124] The coordinates of the middle fingertip are: The coordinates of the motors for the first, second, and third joints of the middle finger are respectively , , , Let be the coordinates of the midpoint of the i-th segment and the m-th segment.

[0125] ③ Ring finger (finger number i=3, 3 segments)

[0126] The ring finger has three motors: the first joint motor 31, the second joint motor 32, and the third joint motor 33. The section from the fingertip 30 to the first joint motor 31 is the first segment of the ring finger; the section from the first joint motor 31 to the second joint motor 32 is the second segment; and the section from the second joint motor 32 to the third joint motor 33 is the third segment.

[0127] Coordinates of the midpoint of the first segment of the ring finger

[0128]

[0129] Coordinates of the midpoint of the second segment of the ring finger

[0130]

[0131] Coordinates of the midpoint of the third segment of the ring finger

[0132]

[0133] Among them, the coordinates of the ring fingertip are The coordinates of the first, second, and third joint motors are respectively , , , Let be the coordinates of the midpoint of the i-th segment and the m-th segment.

[0134] ④ Little finger (finger number i=4, 3 segments)

[0135] The little finger has three motors: the first joint motor 41, the second joint motor 42, and the third joint motor 43. The first segment of the little finger is from the fingertip 40 to the first joint motor 41, the second segment is from the first joint motor 41 to the second joint motor 42, and the third segment is from the second joint motor 42 to the third joint motor 43.

[0136] Coordinates of the midpoint of the first segment of the little finger

[0137]

[0138] Coordinates of the midpoint of the second segment of the little finger

[0139]

[0140] Coordinates of the midpoint of the third segment of the little finger

[0141]

[0142] The coordinates of the little fingertip are: The coordinates of the first, second, and third joint motors are respectively , , , Let be the coordinates of the midpoint of the i-th segment and the m-th segment.

[0143] ⑤ Thumb (finger number i=5, 2 segments)

[0144] The thumb has two motors: a first joint motor 51 and a second joint motor 52. The section from the thumb tip 50 to the first joint motor 51 is the first segment of the thumb, and the section from the first joint motor 51 to the second joint motor 52 is the second segment of the thumb.

[0145] Coordinates of the midpoint of the first segment of the thumb

[0146]

[0147] Coordinates of the midpoint of the second segment of the thumb

[0148]

[0149] The coordinates of the thumb tip are: The coordinates of the motors at the first and second joints of the thumb are respectively , , Let be the coordinates of the midpoint of the i-th segment and the m-th segment.

[0150] (ii) Calculation of contact area

[0151] When a human grasps a cylindrical object with all five fingers, it is in a wrapping state. The midpoints of the motors calculated using the index, middle, ring, little, and thumb are taken as contact points, creating a "wrapping" effect on the grasped cylindrical object. At this time, the cross-sectional area of ​​the cylindrical object and the positions of the midpoints of each finger constitute the reference frame. This embodiment calculates the contact area based on this reference frame: the surface integral of the contact area is approximated by the area of ​​a planar polygon. Based on the perceived three-dimensional position data of the object and the center coordinates of each finger segment calculated in step (a) above, the position of the center of each finger segment in the reference frame is quantized in three dimensions to obtain the contact points of the five fingers on the object's surface. The pentagon formed by the five contact points is divided into three triangles. The area of ​​each triangle is calculated using a vector cross product and then summed to obtain the contact area.

[0152] ① Determination of the five contact points

[0153] The thumb is designated as the starting point for area calculation. The starting points of the three triangles are fixed and all use this point as their initial point. See also Figure 7 Taking the index finger and thumb as examples, the five contact points are represented as follows:

[0154] (Example: The midpoint between the second joint motor 52 and the first joint motor 51 of the thumb is used as the fixed starting point)

[0155]

[0156]

[0157]

[0158]

[0159] in, The midpoint (fixed starting point) between the second joint motor 52 and the first joint motor 51 of the thumb 5. The midpoint between the first joint of the thumb (51) and the tip of the thumb (50) is the point on the right. The midpoint between the tip 10 of the index finger and the first joint motor 11. The midpoint between the first joint motor 11 and the second joint motor 12 of the index finger 1. The midpoint is between the second joint motor 12 and the third joint motor 13 of the index finger 1.

[0160] It should be noted that the five contact points mentioned above are merely examples to illustrate the method for determining contact points; the method for determining the contact points of the other fingers (middle finger, ring finger, and little finger) is the same, that is, the midpoint of each segment of the corresponding finger is used as the contact point. In actual calculation, based on the grasping contact of the target object, five contact points are selected from the five finger contact points to form a contact pentagon, and the contact area is calculated according to the above method.

[0161] ② Pentagonal partitioning and triangle area calculation

[0162] See Figure 7 For example, using P1 as a fixed common vertex, the pentagon is divided into three triangles: △P1P2P3, △P1P3P4, and △P1P4P5. The area of ​​each triangle is calculated using the vector cross product.

[0163] Area of ​​the first triangle:

[0164]

[0165] Area of ​​the second triangle:

[0166]

[0167] Area of ​​the third triangle:

[0168]

[0169] Where, ‖·‖ represents the magnitude of the orientation quantity, which is the length, and × represents the vector cross product.

[0170] ③ Total contact area and area limit

[0171] The contact area of ​​the pentagon formed by the five contact points is the sum of the areas of the three triangles:

[0172]

[0173] in, This represents the actual contact area. , , Let the areas of the first, second, and third triangles be respectively.

[0174] The actual contact area should meet the area limitation conditions:

[0175]

[0176] in, The reference area is the cross-sectional area of ​​the cylindrical object in the reference frame, which is the theoretical contact area required to completely enclose the cylinder. When the actual contact area is not less than the reference area, it indicates that the five fingers are effectively wrapping around the object and making sufficient contact.

[0177] (III) Multi-finger force control coordination

[0178] Secondly, a multi-finger force control coordination strategy is adopted to maintain a uniform and constant clamping force and avoid local stress concentration; the center of gravity of the object is kept constant by adjusting the force at each contact point, and the object is grasped and extracted by exerting force through the wrist.

[0179] ① Definition of contact force error

[0180] Let the real-time contact force at the j-th contact point of the i-th finger (i=1,2,3,4,5 corresponding to the index finger, middle finger, ring finger, little finger, and thumb, respectively) be . The target clamping force is The error between the two is defined as:

[0181]

[0182] Where i is the finger index number and j is the contact point number of that finger. Let be the contact force at the i-th and j-th contact points at time t. For the target clamping force, This represents the error between the two.

[0183] ② Proportional-integral control

[0184] The contact force at each contact point is controlled by proportional-integral regulation to make the error approach zero.

[0185]

[0186] in, For contact force adjustment, This is the proportionality coefficient. The integral coefficient is... The variable is the integral variable. Proportional-integral control uses error as the variable. As input, the output is the contact force adjustment amount, so that the contact force at each contact point tends to the target value and remains constant, thereby achieving uniform clamping force control and avoiding local stress concentration.

[0187] ③ Maintaining and stabilizing the center of gravity

[0188] By adjusting the force at each contact point, the center of gravity of the object remains unchanged, and the object is grasped and extracted by applying force with the wrist, thus achieving stable control.

[0189] Example 4

[0190] This embodiment provides a specific implementation of the grasping mode in step 3. The grasping mode is suitable for objects with grooves or specific structures. It uses four fingers forming a claw and the thumb pressed against the surface as the execution method. A planar quadrilateral is formed by the midpoints of the index finger, little finger, and thumb, and the origin of the wrist coordinate system. The real-time position of the palm is determined by the intersection of the quadrilateral's diagonals. Geometric error control is performed based on the deviation from the target position and the deviation between the fingertips and the corresponding target positions. When lifting the object, force control is used for constraint judgment, limiting the force within a controllable range. This embodiment transforms the empirical operation process of "four fingers together forming a claw - thumb pressed against - fingertips supporting force" from traditional Chinese medicine grasping techniques into a control strategy executable by the robot, achieving precise grasping of objects with grooves or specific structures.

[0191] (a) Determining the position of the palm and the fingertips

[0192] Real-time calculation of palm position: based on the midpoint of the third segment of the index finger ( ), the middle point of the third segment of the little finger ( ), the midpoint of the second segment of the thumb ( The four points, including the wrist coordinate origin, form a planar quadrilateral; the two diagonals of this quadrilateral (index finger) are taken as follows: With little finger The line connecting the dots, the thumb The intersection of the line connecting the wrist coordinate system and the origin of the wrist coordinate system is used as the real-time position coordinate of the palm. (The above...) The midpoint coordinates are taken from the calculation results of Example 3 (I) (see Example 3). Specific positioning is as follows: Figure 6 As shown.

[0193] The real-time position of the palm is:

[0194]

[0195] Let the position of the fingertip of the i-th finger be:

[0196]

[0197] Where i=1,2,3,4 correspond to the index finger, middle finger, ring finger, and little finger, respectively; Real-time position of the palm. The position of the fingertip of the i-th finger;

[0198] (ii) Geometric error control and adjustment

[0199] Geometric error control and adjustment are performed by measuring the deviations between the palm position and the target palm position, and between the fingertips and their corresponding target positions.

[0200] Palm position error:

[0201]

[0202] The sum of the positional errors of the four fingertips:

[0203]

[0204] in, The target position is in the palm of the hand. The target contact position is the fingertips of the four fingers, and ‖·‖ represents the modulus, i.e., the Euclidean distance. Due to palm position deviation, It is the sum of the positional deviations of the fingertips of the four fingers.

[0205] By minimizing geometric errors, the claw-like posture of the four fingers and the position of the palm are adjusted to achieve the desired effect. and Approaching 0, it achieves precise positioning and geometric matching of grooves or objects with specific structures.

[0206] (III) Determination of Force Control Constraints

[0207] When lifting an object, force control is used to determine the constraint, limiting the contact force of each finger to a controllable range:

[0208]

[0209] in, The contact force of the i-th fingertip , These represent the lower and upper limits of the fingertip contact force, respectively, with i=1,2,3,4 corresponding to the index, middle, ring, and little fingers. The force must be limited within this controllable range to ensure uniform force distribution; simultaneously, the claw-like posture is maintained through joint adjustments to adapt to the structural characteristics of the target object.

[0210] Example 5

[0211] This embodiment provides a bionic dexterous hand control system based on grasping intent recognition, realizing the systematic integration of the control methods described in the preceding embodiments. The system adopts a modular design, including a perception module, an intent recognition module, and a pattern execution module, forming a complete functional chain from perception and recognition to control, such as... Figure 7 As shown.

[0212] (a) Perception Module

[0213] The perception module is used to acquire perception data of the target object, extract the three-dimensional spatial position information of the target object, and divide the grasping space into functional and non-functional areas.

[0214] The sensing module includes a main sensor and an auxiliary sensor, forming a redundant dual-sensor architecture: the main sensor is an RGB-D sensor, and the auxiliary sensor is a LiDAR. After the system powers on, it prioritizes self-testing the RGB-D sensor. When the main sensor is working normally, the auxiliary sensor is turned off; when the main sensor fails or its sensing fails, the auxiliary sensor is activated, and the LiDAR generates a sparse environmental point cloud to perform collision avoidance, limit switching, and safety termination actions, preventing the dexterous hand platform from completely failing due to the failure of the main sensor.

[0215] The perception module is also used to define and transform the coordinate system: taking the center of the bionic wrist's dual bones as the origin of the wrist coordinate system, defining the direction of each coordinate axis, and combining the camera's intrinsic parameters to convert the two-dimensional pixel values ​​and depth values ​​into three-dimensional spatial coordinates, generating point cloud data corresponding to the target object, and realizing target recognition and three-dimensional point cloud extraction based on visual and depth information; and determining the safety boundary based on the wrist coordinate origin, the maximum extension distance of the whole hand and the expansion threshold, dividing the grasping space into functional areas and non-functional areas (see step 1 of embodiment 1 for specific implementation).

[0216] (ii) Intent recognition module

[0217] The intent recognition module is used to extract the appearance features of the target object, which include at least one of shape, size and structure, and to map the appearance features to the corresponding force application mode and contact mode according to the preset intent matching rules to determine the grasping intent.

[0218] Among them, appearance feature extraction includes: shape determination, classifying the target object into one or more of the following: irregular curved surface, columnar body, edged and angular, groove; size determination, distinguishing between large and small; and structure determination, determining whether it has complex structures such as edges, grooves, or parts.

[0219] The intent matching rule is as follows: features are categorized based on shape, size, and structural characteristics, and the corresponding force application and contact methods are matched. When the force application method is two fingers and the contact method is a large area, the grasping intent is mapped to a pinching mode; when the force application method is full fingers and the contact method is a large area, the grasping intent is mapped to a gripping mode; when the force application method is multiple fingers and the contact method is a small area, the grasping intent is mapped to a grasping mode; when the force application method is two fingers and the contact method is a small area, it is not mapped to any grasping mode. (For specific implementation, see step 2 of Example 1).

[0220] (III) Mode Execution Module

[0221] The mode execution module is used to select and execute the corresponding grasping mode according to the grasping intention. The grasping modes include pinch mode, gripping mode and grab mode.

[0222] The mode execution module takes the boundary between the functional area and the non-functional area as the execution premise: after the dexterous hand reaches the boundary between the functional area and the non-functional area, the corresponding grasping mode is executed. The pinching mode uses two fingers as the execution carrier to complete the gripping and extraction (see Example 2 for specific implementation); the holding mode uses all five fingers in full contact with the palm to complete the gripping and extraction (see Example 3 for specific implementation); the grasping mode uses four fingers to form a claw with the fingertips for support to complete the grasping (see Example 4 for specific implementation).

[0223] It should be understood that the application of this application is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims. Those skilled in the art can understand that implementing all or part of the processes of the above embodiments and making equivalent changes according to the claims of this application still fall within the scope of this application.

Claims

1. A bionic dexterous hand control method based on grasping intent recognition, characterized in that, Includes the following steps: Step 1: Obtain the perception data of the target object, extract the three-dimensional spatial position information of the target object, and divide the grasping space into functional and non-functional areas. Step 2: Extract the appearance features of the target object, which include at least one of shape, size, and structure; according to the preset intent matching rules, map the appearance features to the corresponding force application method and contact method to determine the grasping intent; Step 3: Select and execute the corresponding grasping mode according to the grasping intention; the grasping modes include pinching mode, gripping mode and grabbing mode; wherein, pinching mode uses two fingers as the execution carrier to complete clamping and extraction; gripping mode uses all five fingers in full contact with the palm to complete gripping and extraction; grabbing mode uses four fingers to form a claw with fingertip support to complete grasping.

2. The method according to claim 1, characterized in that, In step 2, the intent matching rules include: The target object is categorized based on its shape, size, and structure. Shape categorization includes at least one of irregular curved surfaces, cylindrical bodies, objects with sharp edges and corners, and grooves. Size categorization includes large objects and small objects. Structure categorization includes at least one of sharp edges, grooves, and parts. The corresponding force application method and contact method are matched according to the classification results; among them, the force application method includes two-finger force application, multi-finger force application, and full-finger force application, where two fingers are the index finger and thumb, multi-finger force application is the index finger, middle finger, ring finger, and little finger, and full-finger force application is all fingers; the contact method includes large-area contact and small-area contact, where large-area contact is the surface contact formed between the fingertip and the object, and small-area contact is the point contact between the fingertip and the object; When the force is applied using two fingers and the contact area is large, the grasping intention is mapped to the pinching mode; when the force is applied using all fingers and the contact area is large, the grasping intention is mapped to the gripping mode; when the force is applied using multiple fingers and the contact area is small, the grasping intention is mapped to the grabbing mode; when the force is applied using two fingers and the contact area is small, it is not mapped to any grasping mode.

3. The method according to claim 1, characterized in that, In step 1, an RGB-D sensor is used as the main sensor and a LiDAR as the auxiliary sensor. When the main sensor is working normally, the auxiliary sensor is turned off. When the main sensor fails or its sensing fails, the auxiliary sensor is activated, and the LiDAR generates a sparse environmental point cloud to perform anti-collision, limit, and safety termination actions.

4. The method according to claim 1, characterized in that, The method for dividing the functional area and non-functional area in step 1 is as follows: taking the origin of the wrist coordinate as the center of the sphere and the maximum extension distance of the whole hand as the base radius, the expansion radius is obtained by superimposing the expansion threshold on the base radius, and the safe boundary of the grasping space is determined by the expansion radius; the functional area is the area determined by the expansion based on the perceived object, and the area outside the functional area within the safe boundary is the non-functional area.

5. The method according to claim 1, characterized in that, In step 3, the pinching mode includes: The index finger and thumb are used as the execution carriers, while the other three fingers remain curled; the grasping is completed through a three-stage motion path planning, which includes the approach stage, the contact stage, and the pinching stage. During the bonding stage, proportional-integral adjustment is used to control the contact force between the index finger and thumb to achieve constant clamping force control, and the maximum static friction is monitored in real time to prevent slippage. During the pinching stage, the fingers are lifted in a stable triangular posture formed by the center point of the index finger contact surface, the center point of the thumb contact surface, and the origin of the wrist coordinate system.

6. The method according to claim 1, characterized in that, In step 3, the grip modes include: Using the five fingers in full contact with the palm as the execution method, the three-dimensional coordinates of the midpoint of each segment of each finger are calculated with the wrist coordinates as the reference system. The pentagon formed by the contact points of each finger is divided into three triangles. The area of ​​each triangle is calculated by vector cross product and then summed to obtain the contact area. By employing a multi-finger force control coordination strategy and proportional-integral adjustment, uniform clamping force is maintained at each contact point, and the object's center of gravity remains unchanged by adjusting the force at each contact point. The fingers are the index finger, middle finger, ring finger, little finger, and thumb; the segments are as follows: the index finger, middle finger, ring finger, and little finger each have three segments, which are the first segment, the second segment, and the third segment from top to bottom; the thumb has two segments, which are the first segment and the second segment from top to bottom.

7. The method according to claim 6, characterized in that, In step 3, the grabbing modes include: The method of execution is to form a claw with four fingers and press the thumb together. A plane quadrilateral is formed by the midpoint of the third segment of the index finger, the third segment of the little finger, the second segment of the thumb, and the origin of the wrist coordinates. The real-time position of the palm is determined by the intersection of the diagonals of the quadrilateral. Geometric error control adjustment is performed by measuring the deviation between the palm position and the target position, as well as the deviation between the fingertips of the four fingers and the corresponding target positions. When lifting an object, force control is used to determine the constraint and limit the force to a controllable range.

8. A bionic dexterous hand control system based on grasping intent recognition, characterized in that, include: The perception module is used to acquire perception data of the target object, extract the three-dimensional spatial position information of the target object, and divide the grasping space into functional and non-functional areas. An intent recognition module is used to extract the appearance features of a target object, the appearance features including at least one of shape, size and structure, and to map the appearance features to the corresponding force application mode and contact mode according to a preset intent matching rule to determine the grasping intent. The mode execution module is used to select and execute the corresponding grasping mode according to the grasping intention. The grasping modes include pinch mode, gripping mode and grasping mode.

9. A bionic dexterous hand control terminal based on grasping intent recognition, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions, characterized in that, When the instructions are loaded by the processor, they execute the steps of the method according to any one of claims 1-7.

Images