Autonomous thread-strapping method and system for robots based on tactile sensing multi-finger dexterous hands

Abstract

The present application discloses a method, system, device and storage medium for autonomous cable-straightening by a robot based on a tactile-sensing multi-finger dexterous hand, which belongs to the field of manipulator control and includes: obtaining mechanical input data collected from the fingertips of the tactile-sensing multi-finger dexterous hand; generating cable posture estimation data based on the mechanical input data; generating dynamic expected data of the tactile-sensing multi-finger dexterous hand based on the posture estimation data and / or mechanical input data; generating control instructions for the tactile-sensing multi-finger dexterous hand based on the dynamic expected data, so as to control the process of cable-straightening by the tactile-sensing multi-finger dexterous hand. The present application achieves precise control of the multi-finger dexterous hand by acquiring and utilizing tactile-sensing data in real time, and ultimately achieves stable and efficient grasping and following of the cable. The problem of the adaptability of the manipulator to the cable-straightening task and low operating efficiency is solved.

Description

Technical Field

[0001] The present application belongs to the field of manipulator control, and specifically relates to a robot autonomous thread-straightening method, system, equipment and storage medium based on tactile sensing multi-finger dexterous hands. Background Art

[0002] Handling deformable linear objects (DLOs), such as cables, ropes, and pipes, is a common yet challenging task in modern industrial and robotic applications. These tasks include cable routing, packaging, assembly, and maintenance operations. The process of cable routing requires the robot to slide along the object while maintaining a firm grip to prevent it from falling. This operation often requires a high level of dexterity and precision.

[0003] While some specialized end effectors exist in existing technologies and perform well for specific tasks, they lack the flexibility to handle different types of objects, are expensive, and lack versatility. Therefore, to achieve the threading process, rigid grasping and re-grasping methods are often used. Specifically, parallel grippers are used to perform multiple grasping and repositioning to achieve control of the object.

[0004] However, since the parallel gripper has only one degree of freedom, the rigid grasping and re-grasping method has difficulty handling deformable linear objects of various shapes and materials, and cannot adapt to complex wire-straightening tasks. At the same time, due to the difficulty in balancing the grasping force and the wire-straightening force, the parallel gripper needs to re-grasp frequently during execution, and there is also the problem of low operational efficiency. Summary of the Invention

[0005] The present application aims to provide a method, system, device and storage medium for autonomous thread-straightening of a robot based on a tactile sensing multi-finger dexterous hand, at least to solve the problems of the robot's adaptability to thread-straightening tasks and low operating efficiency.

[0006] In a first aspect, embodiments of the present application disclose a robot autonomous thread-straightening method based on a tactile-sensing multi-finger dexterous hand, comprising:

[0007] Acquiring mechanical input data collected from the fingertips of the tactile-sensing multi-finger dexterous hand, wherein the mechanical input data is generated by the fingertips grasping the cable during the process of the tactile-sensing multi-finger dexterous hand performing cable pulling;

[0008] generating, based on the mechanical input data, pose estimation data for the cable; wherein the pose estimation data is used to represent the spatial pose of the cable when the cable is grasped in the tactile-sensing multi-fingered dexterous hand;

[0009] generating expected dynamic data of the tactile-sensing multi-finger dexterous hand based on the pose estimation data and / or the mechanical input data; wherein the expected dynamic data is used to represent an expected change in the motion trajectory of the tactile-sensing multi-finger dexterous hand during a grasping process;

[0010] According to the expected dynamics data, a control instruction for the tactile-sensing multi-finger dexterous hand is generated to control the process of threading by the tactile-sensing multi-finger dexterous hand.

[0011] In a second aspect, the present application also discloses a robot autonomous thread-straightening system based on a tactile sensing multi-finger dexterous hand, comprising:

[0012] A tactile sensing multi-finger dexterous hand and controller having tactile sensors disposed on the fingertips;

[0013] The controller is used to obtain mechanical input data collected from the fingertips of the tactile-sensing multi-finger dexterous hand, and generate posture estimation data for the cable based on the mechanical input data, and generate dynamic expectation data of the tactile-sensing multi-finger dexterous hand based on the posture estimation data and / or the mechanical input data, and generate control instructions for the tactile-sensing multi-finger dexterous hand based on the dynamic expectation data to control the process of straightening the cable by the tactile-sensing multi-finger dexterous hand; the mechanical input data is generated by the fingertips grasping the cable during the process of straightening the cable by the tactile-sensing multi-finger dexterous hand; the posture estimation data is used to characterize the spatial posture of the cable when the cable is grasped in the tactile-sensing multi-finger dexterous hand; the dynamic expectation data is used to characterize the expected change in the motion trajectory of the tactile-sensing multi-finger dexterous hand during the grasping process.

[0014] In a third aspect, an embodiment of the present application further discloses an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application further discloses a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0016] In summary, in the embodiment of the present application, by generating accurate posture estimation data, the position and posture of the cable in the multi-finger dexterous hand can be grasped in real time; then, by comprehensively analyzing the posture estimation data and mechanical input data, the optimal motion trajectory of the multi-finger dexterous hand in the process of stringing is determined, and then the dynamic expectation data is converted into control instructions for the multi-finger dexterous hand, so as to achieve precise control of the stringing process, accurately guide the multi-finger dexterous hand to perform stringing operations, and ensure stability and accuracy in the stringing process. Therefore, based on the method of the embodiment of the present application, by acquiring and utilizing tactile sensing data in real time, precise control of the multi-finger dexterous hand is achieved, and finally stable and efficient grasping and following of the cable is achieved. The problems of the adaptability of the manipulator to the stringing task and low operating efficiency are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In the attached figure:

[0018] Figure 1 is a schematic diagram of the key structure of the tactile sensing multi-finger dexterous hand involved in this application;

[0019] Figure 2 This is a flowchart of the steps of a robot autonomous thread-straightening method based on a tactile sensing multi-finger dexterous hand provided in an embodiment of the present application;

[0020] Figure 3 This is a flowchart of another method for autonomous thread-straightening by a robot based on a tactile-sensing multi-finger dexterous hand provided by an embodiment of the present application;

[0021] Figure 4 This is a comparison diagram of the two robot wire drawing processes provided by the embodiments of the present application;

[0022] Figure 5 This is the data flow process under the embodiment of the present application;

[0023] Figure 6 This is a block diagram of a robot autonomous thread-straightening system based on a tactile-sensing multi-finger dexterous hand provided in an embodiment of the present application;

[0024] Figure 7 is a block diagram of an electronic device according to an embodiment of the present application;

[0025] Figure 8 is a block diagram of an electronic device according to another embodiment of the present application;

[0026] Among them: 301-tactile sensing multi-finger dexterous hand; 3011-finger tip; 30111-tactile sensor; 401-parallel gripper; A-cable. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0029] In this application, deformable linear objects (DLOs) will be collectively referred to as "cables" unless otherwise specified. Specifically, deformable linear objects refer to linear objects with flexible and bendable properties, including but not limited to cables, optical fibers, wires, etc. All DLO objects mentioned in the description and embodiments of this application are expressed using the term "cable". This unified name helps to simplify the expression of technical content and facilitates readers' understanding of the technical solutions of this application, and does not affect the comprehensive understanding and protection scope of this application.

[0030] In this application, "threading" refers to the process of grasping and moving the cable by using a tactile-sensing multi-finger dexterous hand to adjust the gripping force and motion trajectory in real time to achieve stable following and positioning of the cable. That is, the process of In-Hand Following of Deformable LinearObjects Using Dexterous Fingers with Tactile Sensing. Specifically, the dexterous hand uses tactile sensors to collect mechanical input data and posture estimation data, which are used to generate control instructions in real time to adjust the movement of each joint and the force applied. In this way, the dexterous hand can dynamically adapt to changes in the shape and position of the cable to ensure stable and precise control of the cable during operation.

[0031] The primary goal of the cable-straightening process is to improve the efficiency and precision of cable manipulation, preventing slippage or damage during movement. By adjusting grip force and motion trajectory in real time, the dexterous hand can better adapt to different operating environments and task requirements, thereby improving operational stability and reliability. This process applies not only to adjusting cables in space but also to precisely controlling their position and shape during installation and maintenance.

[0032] like Figure 1 As shown, the Dexterous Fingers with Tactile Sensing 301 is an advanced robotic device characterized by tactile sensors 30111 on each fingertip 3011. These sensors detect and collect mechanical contact data with the cable during the cable-winding process. By monitoring this mechanical input data in real time, the Dexterous Fingers with Tactile Sensing generates spatial pose estimation and motion trajectory data, enabling stable grasping and precise manipulation of objects. The Dexterous Fingers with Tactile Sensing not only possesses multi-degree-of-freedom flexibility but also uses intelligent control algorithms to optimize gripping posture and path, achieving efficient and precise cable-winding operations.

[0033] Based on the above ideas, if Figure 2 As shown in the figure, an autonomous thread-straightening method of a robot based on a tactile sensing multi-finger dexterous hand is provided in an embodiment of the present application.

[0034] The method may include the following steps:

[0035] Step 101: Acquire mechanical input data collected from the fingertips of a tactile sensing multi-finger dexterous hand.

[0036] Among them, the mechanical input data is generated by the fingertips grasping the cable during the process of stringing the cable through the tactile sensing multi-finger dexterous hand.

[0037] In some embodiments of the present application, the mechanical input data collected from the fingertips of the tactile-sensing multi-finger dexterous hand is obtained in order to monitor the force feedback during the grasping process in real time, so as to ensure the stability of the grasping and prevent the cable from slipping. In the process of straightening the cable using the tactile-sensing multi-finger dexterous hand, the mechanical input data of the fingertips can reflect the magnitude and direction of the force applied to the cable. The mechanical input data includes information such as pressure and shear force detected by the tactile sensor. The execution of this step ensures that the system can adjust the gripping force according to the real-time data, prevent the cable from falling off during the operation, and improve the stability and safety of the operation.

[0038] In a specific example, a multi-fingered dexterous hand using tactile sensing grasps a flexible cable and begins to reel it. The tactile sensors collect real-time mechanical input data from the fingertips on the cable, including changes in pressure and shear force. Based on this real-time data, the system adjusts the grip force of each fingertip to ensure the cable remains stable throughout the reeling process, ultimately achieving safe and efficient cable manipulation.

[0039] Step 102: Generate estimated data of the cable's posture based on the mechanical input data.

[0040] Among them, the pose estimation data is used to characterize the spatial pose of the cable when it is grasped in the tactile sensing multi-finger dexterous hand.

[0041] In some embodiments of the present application, the pose estimation data of the cable is generated based on the mechanical input data in order to accurately grasp the spatial position and posture of the cable in the multi-fingered dexterous hand. The mechanical input data reflects the changes in the force on the cable during the grasping process, and the spatial pose of the cable can be derived from these data. The pose estimation data is used to characterize the specific position and posture of the cable when it is grasped in the tactile sensing multi-fingered dexterous hand, thereby providing an accurate basis for subsequent motion control. The execution of this step ensures that the system can adjust the grasp and motion trajectory in real time, thereby improving the accuracy and reliability of the cable-winding operation.

[0042] In a specific example, a multi-finger dexterous hand, using tactile sensing, grasps a flexible cable and begins to string it. The tactile sensors collect real-time mechanical input data from the fingertips on the cable, including changes in pressure and shear force. Based on this real-time data, the system generates an estimated position of the cable, determining its exact position and posture within the dexterous hand. This information then adjusts the movement of the fingers and arm accordingly, ensuring stability and accuracy during the stringing process.

[0043] Step 103 : generating expected dynamic data of the tactile-sensing multi-finger dexterous hand based on the posture estimation data and / or the mechanical input data.

[0044] Among them, the dynamic expectation data is used to characterize the expected changes in the motion trajectory of the tactile sensing multi-finger dexterous hand during the grasping process.

[0045] In some embodiments of the present application, the expected dynamic data of the tactile sensing multi-finger dexterous hand is generated based on at least one of the posture estimation data and the mechanical input data in order to determine the optimal motion trajectory of the dexterous hand during the grasping process. The posture estimation data reflects the specific position and posture of the cable in the dexterous hand, while the mechanical input data provides the change in the force applied to the cable during the grasping process. The expected dynamic data is used to characterize the expected change in the motion trajectory of the tactile sensing multi-finger dexterous hand during the grasping process, thereby achieving precise control of the grasping process. The execution of this step ensures that the system can adjust the motion trajectory according to real-time data and optimize the flexibility and accuracy of the line-pulling operation.

[0046] In a specific example, a tactile multi-finger dexterous hand grasps a flexible cable and begins to string the cable. Tactile sensors collect real-time mechanical input data from the fingertips on the cable, and the system generates an estimated pose of the cable based on this real-time data. By comprehensively analyzing the pose estimation data and mechanical input data, the system generates expected dynamic data for the tactile multi-finger dexterous hand, which can adjust the hand's motion trajectory to ensure stability and accuracy during the cable stringing process.

[0047] Step 104 : generating control instructions for the tactile sensing multi-finger dexterous hand based on the expected dynamics data, so as to control the thread-straightening process performed by the tactile sensing multi-finger dexterous hand.

[0048] In some embodiments of the present application, control instructions for the tactile sensing multi-finger dexterous hand are generated based on the expected dynamic data in order to achieve precise control of the cable-straightening process. The expected dynamic data characterizes the changes in the motion trajectory of the tactile sensing multi-finger dexterous hand during the grasping process, and corresponding control instructions can be generated based on these data to control the movement of the dexterous hand. The control instructions include instructions for the specific position and force of each finger joint, thereby ensuring that the fingers can move according to the expected trajectory and force during the cable-straightening process. The execution of this step ensures that the system can make adjustments based on real-time data to achieve stable and efficient cable-straightening.

[0049] In a specific example, a multi-fingered dexterous hand, using tactile sensing, grasped a flexible cable and began to pull it. Based on previously generated dynamic expectation data, the system generated corresponding control instructions, including instructions for the specific position and force of each finger joint. The dexterous hand executed the pulling operation according to these control instructions, ensuring that each finger moved according to the expected trajectory and force, ultimately achieving precise control and stable pulling of the cable.

[0050] In summary, in the embodiment of the present application, by generating accurate posture estimation data, the position and posture of the cable in the multi-finger dexterous hand can be grasped in real time; then, by comprehensively analyzing the posture estimation data and mechanical input data, the optimal motion trajectory of the multi-finger dexterous hand in the process of stringing is determined, and then the dynamic expectation data is converted into control instructions for the multi-finger dexterous hand, so as to achieve precise control of the stringing process, accurately guide the multi-finger dexterous hand to perform stringing operations, and ensure stability and accuracy in the stringing process. Therefore, based on the method of the embodiment of the present application, by acquiring and utilizing tactile sensing data in real time, precise control of the multi-finger dexterous hand is achieved, and finally stable and efficient grasping and following of the cable is achieved. The problems of the adaptability of the manipulator to the stringing task and low operating efficiency are solved.

[0051] Figure 3 This is another robot autonomous thread-straightening method based on a tactile-sensing multi-finger dexterous hand provided in an embodiment of the present application.

[0052] The method may include the following steps:

[0053] Step 201 : determining the mechanical sampling distribution of each fingertip by setting a plurality of tactile sensors on each fingertip.

[0054] In some embodiments of the present application, by setting up multiple tactile sensors at each fingertip, the mechanical sampling distribution of each fingertip is determined in order to accurately obtain and analyze the mechanical information of the tactile sensor during the grasping process. This mechanical information reflects the specific force conditions of each fingertip when grasping the cable. The mechanical sampling distribution refers to the distribution of mechanical data collected by the tactile sensor over a period of time. The execution of this step ensures that the system can fully understand the mechanical characteristics of the grasping process, providing a basis for subsequent data processing and control instruction generation.

[0055] In a specific example, multiple tactile sensors installed on each fingertip of a tactile multi-fingered dexterous hand collect real-time mechanical data from each fingertip on the cable during a wire-pulling operation. This data can include pressure and shear forces at different locations and directions. The system analyzes this mechanical data to determine the distribution of mechanical samples for each fingertip, providing a comprehensive understanding of the forces acting on each fingertip during the grasping process.

[0056] Step 202 : Decompose the mechanical sampling distribution into two mechanical sampling sub-distributions, so that the two mechanical sampling sub-distributions constitute a two-component Gaussian mixture model.

[0057] Among them, each mechanical sampling sub-distribution has a corresponding distribution weight parameter.

[0058] In some embodiments of the present application, the mechanical sampling distribution is decomposed into two mechanical sampling sub-distributions to more accurately analyze and process the mechanical data collected by the tactile sensor. This process utilizes a two-component Gaussian mixture model (GMM) to model the mechanical sampling distribution of the tactile sensor, enabling the two mechanical sampling sub-distributions to correspond to different force conditions. The two-component Gaussian mixture model is a statistical model that fits complex data distributions through a weighted combination of two normal distributions. Each mechanical sampling sub-distribution has a corresponding distribution weight parameter to reflect its importance and contribution. Specifically, the expectation maximization (EM) algorithm is applied to approximate the mechanical input data as two Gaussian distributions: one representing the contact area and the other representing the non-contact area. Based on the data points within the distribution area, the mean (μ) and variance (σ) of each Gaussian distribution, as well as their respective weight parameters, are calculated and determined. This step ensures that the system can more accurately separate and analyze different force conditions, providing a more precise data foundation for subsequent pose estimation and control command generation.

[0059] In a specific example, suppose the system is handling a cable, and the mechanical sensors at each fingertip record a large amount of gripping mechanical data. These data points are distributed across different force ranges, with some reflecting actual contact between the fingertip and the cable surface (higher force values) and other reflecting non-contact areas (lower force values). By applying the expectation-maximization algorithm, the system can decompose this data into two Gaussian distributions, one representing the contact area and the other representing the non-contact area. The system then calculates the mean (μ) and variance (σ) of each Gaussian distribution, as well as the weight parameters for each distribution. In this way, the system can clearly distinguish between data in the contact and non-contact areas, providing accurate mechanical data support for subsequent posture estimation and trajectory generation.

[0060] In step 203 , the mechanical sampling sub-distribution corresponding to the smaller distribution weight parameter of the two mechanical sampling sub-distributions is determined as the mechanical input data of the fingertip corresponding to the mechanical sampling sub-distribution.

[0061] In some embodiments of the present application, the mechanical sampling sub-distribution corresponding to the smaller distribution weight parameter of the two mechanical sampling sub-distributions is determined as the mechanical input data of the fingertip corresponding to the mechanical sampling sub-distribution, in order to accurately identify and extract the effective force information perceived by the tactile sensor. In the previous step, the mechanical sampling data has been decomposed into two sub-distributions, and each sub-distribution has its corresponding distribution weight parameter. The distribution weight parameter reflects the relative importance of the sub-distribution in the overall data. By selecting the sub-distribution with a smaller distribution weight parameter as the mechanical input data, the noise data can be effectively removed and the accuracy of the data can be guaranteed. The execution of this step ensures that the subsequent posture estimation and control instruction generation are based on accurate and reliable force data, thereby improving the stability and accuracy of the system.

[0062] In a specific example, mechanical data collected over a period of time using the tactile sensors of a multi-fingered dexterous hand is decomposed into two sub-distributions. The system calculates distribution weights for these two sub-distributions: one with a distribution weight of 0.3 and the other with a distribution weight of 0.7. The system selects the sub-distribution with a distribution weight of 0.3 as the mechanical input data because it better represents the effective force information generated by the tactile sensors during grasping. This selection allows the system to eliminate a significant amount of noisy data, ensuring the accuracy of pose estimation and control command generation.

[0063] Step 204: Generate estimated data of the cable's posture based on the mechanical input data.

[0064] Among them, the pose estimation data is used to characterize the spatial pose of the cable when it is grasped in the tactile sensing multi-finger dexterous hand.

[0065] The method shown in this step has been described in step 102 and will not be repeated here.

[0066] Optionally, step 204 includes the following sub-steps:

[0067] Sub-step 2041 : generating contact surface simulation data between the fingertip and the cable according to the mechanical input data.

[0068] The contact surface simulation data is used to represent the force value at each contact point between the fingertip and the cable.

[0069] In some embodiments of the present application, contact surface simulation data between the fingertip and the cable is generated based on the mechanical input data in order to more accurately analyze the force conditions of the tactile sensor during the line drawing process. The contact surface simulation data reflects the specific force value at each contact point between the fingertip and the cable. In this process, the tactile sensor collects the mechanical input data in real time and converts it into corresponding contact surface simulation data. The contact surface simulation data includes the pressure and shear force values ​​of the fingertip and the cable at different contact points. Through these data, the force conditions of the fingertip and the cable can be accurately characterized, thereby providing a more accurate basis for subsequent posture estimation and control instruction generation.

[0070] In a specific example, the tactile sensors of a multi-fingered dexterous hand collect mechanical input data and initiate a cable-straightening operation. Based on this real-time data, the system generates simulated data of the contact surface between the fingertips and the cable. This simulated data reflects the pressure and shear force at different contact points, ensuring a precise representation of the forces acting on the fingertips and the cable. This processing enables more detailed force information to be obtained, providing accurate data support for pose estimation and motion control.

[0071] In sub-step 2042, the contact surface simulation data is used as the first boundary value, and the minimum value of the total variation consisting of the distance between all contact points and the cable and the radius of the cable in the contact surface simulation data is used as the first target value to construct a cable posture optimization model.

[0072] In some embodiments of the present application, the contact surface simulation data is used as the first boundary value, and the minimum value of the total variation composed of the distance between all contact points and the cable and the radius of the cable in the contact surface simulation data is used as the first target value to construct a cable posture optimization model in order to accurately estimate the spatial posture of the cable during the grasping process. The contact surface simulation data reflects the specific contact situation between the fingertip and the cable. These data are used as the boundary values ​​of the posture optimization model to ensure that the model can truly reflect the actual situation. The minimum value of the total variation refers to the minimum error composed of the difference in the distance between all contact points and the cable axis and the radius of the cable in the contact surface simulation data. By constructing a posture optimization model, the spatial posture of the cable can be estimated more accurately, thereby improving the cable control accuracy of the system.

[0073] In one specific example, the tactile sensors of a multi-fingered dexterous hand collect simulated data on the contact surface between the fingertips and the cable. The system uses this data as a first boundary value and constructs an optimized cable posture model based on the minimum total variation of the distance between all contact points and the cable axis and the cable radius. This model enables the system to accurately estimate the spatial position and posture of the cable within the dexterous hand and adjust grip and motion control accordingly, ensuring the accuracy of the cable-winding operation.

[0074] Sub-step 2043 , solving the pose optimization model to obtain pose estimation data.

[0075] In some embodiments of the present application, solving the posture optimization model to obtain posture estimation data is to accurately calculate the spatial position and posture of the cable during the grasping process. The posture optimization model is based on the previously generated contact surface simulation data and the minimum total variation. By solving the model, the posture estimation data of the cable can be obtained. The posture estimation data is used to characterize the specific position and direction of the cable in the tactile sensing multi-fingered dexterous hand, thereby providing an accurate basis for the generation of control instructions. By solving the optimization model, the system can adjust the grasping and motion trajectory in real time to ensure high precision and stability of the cable pulling operation.

[0076] In a specific example, the tactile sensors of a multi-fingered dexterous hand generated simulated contact surface data and constructed an optimized cable pose model. The system solved this model to obtain estimated cable pose data, including the cable's specific position and orientation in space. Based on this data, the system adjusted the dexterous hand's grip and motion control to ensure stability and precision during cable reeling.

[0077] Optionally, in a further embodiment, all contact points of the two sensors can be transformed into the same coordinate system using the forward kinematics of the finger to obtain the pose estimation data. In this case, the pose optimization model has the following form:

[0078] ;

[0079] Among them, Θ represents the optimization variable, and the difference term d is calculated. i (Θ)-r represents the distance from the i-th contact point to the cable (predicted straight line), W r is the predicted parameter, ψ represents the unit direction under the predicted posture, ξ x Represents the component of the parameter point of the straight line in the x direction, o i,x Represents the component in the x direction at the i-th contact point.

[0080] Step 205 : Generate expected dynamic data of the tactile-sensing multi-finger dexterous hand based on the posture estimation data and / or the mechanical input data.

[0081] Among them, the dynamic expectation data is used to characterize the expected changes in the motion trajectory of the tactile sensing multi-finger dexterous hand during the grasping process.

[0082] The method shown in this step has been described in step 103 and will not be repeated here.

[0083] like Figure 4As shown, the figure shows the parallel gripper 401 (parallel gripper) ( Figure 4 -a) and the comparison results of grasping cable A by the tactile-sensing multi-finger dexterous hand 301. The dashed arrow in the figure represents the gravity acting on cable A, the thick solid arrow represents the maximum contact feedback force that can be applied to the fingertips of the parallel gripper 401 and the tactile-sensing multi-finger dexterous hand 301, and the thin solid arrow represents the contact force directly exerted by cable A on the parallel gripper 401 and the tactile-sensing multi-finger dexterous hand 301. It can be seen that the parallel gripper 401 has difficulty finding an appropriate gripping force to balance following and holding. However, because the tactile-sensing multi-finger dexterous hand 301 has more degrees of freedom, these two goals can be decoupled by forming a "V" shape: that is, for holding, a larger gripping force can be applied to ensure close contact between the bottoms of the two fingertips; for following, the angle of the V shape (i.e., the gripping angle θ) can be adjusted to apply a moderate contact force.

[0084] Experiments ultimately revealed that the grip angle θ for a specific cable is initially determined during the grasping process: the fingers first grasp the cable parallel to the cable; then, the grip angle is increased until any tactile sensor loses contact with the cable. During the following process, if the shear force on the tactile sensor exceeds a threshold, the grip angle is increased; if the in-hand contact does not meet the requirements for reliable perception, the grip angle is reduced. Limiting the grip angle to between 15° and 50° and maintaining a constant grip force is a relatively preferred approach.

[0085] There are the following further options for step 205:

[0086] Optionally, the expected dynamic data includes expected data on the angle change of the cable clamping angle of the tactile sensing multi-finger dexterous hand. Step 205 includes the following sub-steps:

[0087] Sub-step 2051 : when the mechanical input data indicates that the friction force between the fingertip and the cable in the tangential direction exceeds a preset tangential force threshold, determining the expected angle change data to decrease.

[0088] Among them, the tangential direction is the movement direction of the tactile sensing multi-fingered dexterous hand.

[0089] In some embodiments of the present application, when the mechanical input data is characterized by the friction between the fingertip and the cable in the tangential direction exceeding a preset tangential force threshold, the angle change expected data is determined to be reduced in order to prevent the cable from sliding or being damaged due to excessive tangential friction. In this process, the mechanical input data reflects the friction between the fingertip and the cable in the tangential direction. When the friction exceeds the preset threshold, the system will automatically set the angle change expected data to decrease to reduce the friction. The tangential direction refers to the direction of movement of the tactile sensing multi-finger dexterous hand. The execution of this step can effectively reduce friction, ensure that the cable is not damaged during the grasping process, and maintain the stability and safety of the operation.

[0090] In one specific example, a multi-fingered dexterous hand uses tactile sensing to initiate a cable-straightening operation. When the tactile sensor detects that the tangential friction between the fingertips and the cable exceeds a preset tangential force threshold, the system automatically adjusts the desired angle change to a smaller value, reducing the friction applied to the cable. This adjustment ensures that the cable does not slip or become damaged due to excessive friction during the cable-straightening process, enabling stable grip and cable-straightening operations.

[0091] Sub-step 2052 : when the mechanical input data indicates that the contact area between the fingertip and the cable is smaller than a preset contact area threshold, determining the expected angle change data to be increasing.

[0092] In some embodiments of the present application, when the mechanical input data indicates that the area of ​​the contact area between the fingertip and the cable is less than a preset contact area threshold, the expected angle change data is determined to increase in order to ensure stable grip of the tactile-sensing multi-finger dexterous hand during the threading process. The mechanical input data reflects the specific area of ​​the contact area between the fingertip and the cable. When the contact area is less than the preset threshold, the system automatically sets the expected angle change data to increase to increase the contact area. Through this adjustment, it is possible to ensure that the contact area with the cable during the gripping process reaches an ideal state, thereby improving grip stability and control accuracy.

[0093] In a specific example, a multi-finger dexterous hand uses tactile sensing to initiate a cable-straightening operation. When the tactile sensor detects that the contact area between the fingertips and the cable is less than a preset contact area threshold, the system automatically increases the expected angle change to increase the contact area between the fingertips and the cable. This adjustment ensures a stable grip on the cable during the stringing process, enabling more precise and reliable stringing.

[0094] Optionally, the expected dynamics data includes expected data on the change in the palm direction of the tactile-sensing multi-finger dexterous hand. Step 205 includes the following sub-steps:

[0095] In sub-step 2053, when the pose estimation data indicates that the grip angle formed by the cable and the tactile sensing multi-fingered dexterous hand does not match a preset grip angle threshold, the expected palm direction change data is determined as a rotation in the target rotation direction.

[0096] The target rotation direction is the minimum rotation direction that makes the grasping angle match the grasping angle threshold.

[0097] In some embodiments of the present application, when the posture estimation data characterizes that the gripping angle formed by the cable and the tactile sensing multi-finger dexterous hand does not match the preset gripping angle threshold, the expected palm direction change data is determined to rotate in the target rotation direction in order to ensure that the gripping angle is consistent with the preset ideal gripping angle. The posture estimation data provides the specific position and posture of the cable relative to the dexterous hand. When it is detected that the gripping angle does not meet the preset value, the system adjusts the expected palm direction change data to rotate it in the target rotation direction. The target rotation direction refers to the minimum rotation direction that matches the gripping angle with the preset gripping angle threshold. The execution of this step can optimize the gripping angle and improve gripping stability and operation accuracy.

[0098] In one specific example, a multi-fingered dexterous hand initiates a cable-straightening operation using tactile sensing. Based on pose estimation data, the system detects that the grip angle between the cable and the dexterous hand deviates from a preset grip angle threshold. To adjust the grip angle, the system sets the expected palm orientation change data to rotate toward the target rotation direction—the minimum rotation direction that allows the grip angle to match the preset value. This adjustment ensures that the cable maintains a stable and precise grip angle during the cable-straightening process, ultimately achieving an efficient and stable cable-straightening operation.

[0099] Step 206 : generating control instructions for the tactile-sensing multi-finger dexterous hand based on the expected dynamics data, so as to control the thread-straightening process performed by the tactile-sensing multi-finger dexterous hand.

[0100] The method shown in this step has been described in step 104 and will not be repeated here.

[0101] Optionally, step 206 includes the following sub-steps:

[0102] In sub-step 2061, the running trajectory limit data of the tactile sensing multi-finger dexterous hand is used as the second boundary value, and the actual dynamic state data and dynamic expected data of the tactile sensing multi-finger dexterous hand, and the dynamic state error data of the tactile sensing multi-finger dexterous hand determined as the second target value, to construct a joint motion optimization model of the tactile sensing multi-finger dexterous hand.

[0103] In some embodiments of the present application, the operation trajectory limit data of the tactile sensing multi-finger dexterous hand is used as the second boundary value, and the dynamic state error data of the tactile sensing multi-finger dexterous hand determined by the actual dynamic state data and the dynamic expected data of the tactile sensing multi-finger dexterous hand is used as the second target value. The joint motion optimization model of the tactile sensing multi-finger dexterous hand is constructed to ensure that the dexterous hand can accurately control its joint motion during the execution process. The operation trajectory limit data defines the maximum range of motion that the dexterous hand can achieve during the operation process, and the dynamic state error data reflects the difference between the actual motion state and the expected state. Through this optimization model, higher operation accuracy and efficiency can be achieved while ensuring the stability of the dexterous hand's motion.

[0104] In a specific example, a multi-fingered dexterous hand uses tactile sensing to collect trajectory limit data and record its actual dynamic state data and expected dynamic data. Based on this data, the system calculates dynamic state error data and uses this as an optimization target to construct a joint motion optimization model. In this model, the trajectory limit data serves as a boundary value to ensure that the dexterous hand does not exceed its physical limitations during movement. By solving this optimization model, the system can generate control instructions that enable the dexterous hand to maintain high precision and stability when performing cable-winding tasks, ultimately achieving efficient and stable cable manipulation.

[0105] Sub-step 2062 , solving the joint motion optimization model to obtain control instructions.

[0106] In some embodiments of the present application, solving the joint motion optimization model to obtain control instructions is intended to achieve precise control of the tactile-sensing multi-fingered dexterous hand. By solving the joint motion optimization model, the optimal motion path and control parameters for each joint of the dexterous hand can be calculated based on the optimization objectives and boundary values. The control instructions include specific motion instructions for each joint to achieve the desired dynamic state and operational task. Through this optimization solution, the system can ensure that the dexterous hand maintains high precision and stability when performing tasks, thereby improving operational efficiency and task success rate.

[0107] In a specific example, a tactile-sensing multi-fingered dexterous hand collected trajectory limit data and dynamic state error data, and constructed a joint motion optimization model. The system solved this optimization model and obtained control instructions for each joint. These control instructions included the specific movement angle and force for each joint. The dexterous hand executed the cable-winding operation based on these instructions, ensuring that each joint moved according to the expected path and parameters, ultimately achieving efficient and stable grasping and cable-winding operations.

[0108] Optionally, in a further embodiment, considering that the motion trajectory of the tactile sensing multi-finger dexterous hand during the threading process depends on the positions of the two fingertips in the space W formed by the world coordinate system and the space H formed by the hand coordinate system ( w P f,i and H P f,i ), the joint motion optimization model is expressed as:

[0109] ;

[0110] Where C represents the dynamic state error data, 、 、 They represent the errors between the current pose and the desired pose calculated by the joint position q and the robot forward kinematics in space W and space H, respectively. fw,i 、W fh,i 、W rfh 、W hw Represent the predicted parameter matrix, q lb and q ub They represent the lower and upper limits of the value of q respectively.

[0111] Optionally, the expected dynamic data includes expected position data for characterizing the spatial position of the tactile-sensing multi-fingered dexterous hand. Step 206 includes the following sub-steps:

[0112] Sub-step 2063 , bringing the expected position data into the dynamic balance model of the tactile sensing multi-finger dexterous hand.

[0113] Among them, the dynamic balance model is used to characterize the relationship between the applied force value of each joint of the tactile sensing multi-finger dexterous hand and the movement trajectory when grasping the tactile sensing multi-finger dexterous hand during the process of threading.

[0114] In some embodiments of the present application, the expected position data is incorporated into the dynamic balance model of the tactile sensing multi-fingered dexterous hand to ensure that the dexterous hand can achieve precise grasping control when performing the thread-straightening task. The expected position data characterizes the expected position of the dexterous hand in space. The dynamic balance model is used to characterize the relationship between the applied force value and the trajectory of each joint of the dexterous hand when grasping during the thread-straightening process. By incorporating the expected position data into the model, the mechanical state of each joint can be accurately calculated, thereby ensuring the stability of the grasp and the accuracy of the control.

[0115] In a specific example, the desired position data for a tactile-sensing multi-fingered dexterous hand is set and incorporated into its dynamic balance model. This model optimizes grasping control by calculating the relationship between the applied force and the trajectory of each joint during the task. By inputting the desired position data into the model, the system can accurately calculate the mechanical state of each joint, ensuring that the dexterous hand maintains a stable and accurate grasp during the thread-straightening task.

[0116] Sub-step 2064 , solving the dynamic equilibrium model to determine the mechanical output value of each joint.

[0117] In some embodiments of the present application, solving the dynamic balance model to determine the mechanical output value of each joint is to accurately calculate the force required to be applied by each joint when the tactile sensing multi-fingered dexterous hand performs a grasping task. The dynamic balance model calculates the mechanical output value of each joint by analyzing the expected position data and the mechanical state in actual operation. The model includes the relationship between the range of motion of each joint, the applied force value and the travel trajectory. By solving this model, the optimal mechanical output value can be obtained, thereby ensuring the stability and control accuracy of the dexterous hand during the threading process.

[0118] In a specific example, a multi-fingered dexterous hand using tactile sensing performs a string-straightening task. The system inputs the desired position data and the actual mechanical state data into a dynamic equilibrium model and solves the model to determine the mechanical output value of each joint. The system calculates the force required to perform the task at each joint, ensuring stable and precise control of the dexterous hand during the string-straightening process.

[0119] Sub-step 2065 , according to the mechanical output value of each joint, rotating the joint corresponding to each mechanical output value to control the grasping of the cable.

[0120] In some embodiments of the present application, the joints corresponding to each mechanical output value are rotated according to the mechanical output value of each joint to control the grasping of the cable, in order to achieve precise control of the cable by the tactile-sensing multi-fingered dexterous hand during the threading process. The mechanical output value of each joint is obtained through the previously solved dynamic equilibrium model, and these mechanical output values ​​indicate the applied force required for each joint to perform the task. Based on these mechanical output values, the system controls each joint to rotate accordingly to achieve a stable grasp of the cable. The execution of this step ensures that the dexterous hand can adjust the gripping force according to the mechanical data calculated in real time during the threading process, thereby improving the stability and accuracy of the threading operation.

[0121] In a specific example, a multi-fingered dexterous hand using tactile sensing begins performing a cable-pulling task. Based on a previously solved dynamic equilibrium model, the system determines the mechanical output value for each joint. Based on these mechanical output values, the system adjusts the rotation angle of each joint to achieve a stable grip on the cable. For example, if the mechanical output value of a joint indicates a need for increased grip force, the system increases the rotation angle of that joint accordingly to ensure a secure grip on the cable.

[0122] In some further embodiments of the present application, considering that for some tactile sensing multi-finger dexterous hands, the position control mode of proportional-integral-derivative (PID) is used to control the joints, when the integral gain is set to zero, the joint torque τ is proportional to the desired joint position q d The correlation with the current joint position q can be simplified into a trajectory model:

[0123] τ=k p (q d -q)-k v q';

[0124] Where q' represents the first-order derivative of q with respect to time. In this scheme, the stable grasping of the tactile sensing multi-finger dexterous hand is a quasi-static process, so the velocity term in the above model is almost zero, so the expected joint position q of the finger can be assigned d To control the finger tip to apply force F to the object d , that is,

[0125] ;

[0126] where J(q) is the Jacobian matrix of the finger.

[0127] Optionally, the multiple joints of the tactile-sensing multi-fingered dexterous hand are arranged in sequence according to a preset arrangement direction, and sub-step 2065 includes the following sub-steps:

[0128] Sub-step 20651: rotate the joints corresponding to each mechanical output value in sequence according to the arrangement direction.

[0129] In some embodiments of the present application, the joints corresponding to each mechanical output value are rotated in sequence according to the arrangement direction to ensure that each joint of the tactile sensing multi-fingered dexterous hand can move in a coordinated and consistent manner during operation. When executing this process, the system will rotate and adjust each joint in sequence starting from the first joint according to the preset arrangement direction. The rotation of each joint is based on the previously calculated mechanical output values, which indicate the force and rotation angle that should be applied to each joint. This ensures the coordinated movement of all joints, thereby achieving stable grasping and threading operations.

[0130] In a specific example, a multi-fingered dexterous hand using tactile sensing performs a thread-straightening task. Based on a previously solved dynamic equilibrium model, the system determines the mechanical output value for each joint. Starting with the first joint, each joint is rotated and adjusted sequentially according to the preset arrangement, ensuring that each joint moves according to the calculated mechanical output value. This sequential rotation ensures that all joints move in a coordinated and consistent manner.

[0131] Optionally, between step 201 and step 202, the method further includes the following additional steps:

[0132] Step 207: Delete the target mechanical sampling sub-distribution in the mechanical sampling distribution to update the mechanical sampling distribution.

[0133] The target mechanical sampling sub-distribution is the mechanical sampling sub-distribution of the contact area between each fingertip and other fingertips in the mechanical sampling distribution.

[0134] In some embodiments of the present application, the target mechanical sampling sub-distribution in the mechanical sampling distribution is deleted to update the mechanical sampling distribution in order to remove the force data of the contact area with other fingertips and improve the accuracy and reliability of the data. The target mechanical sampling sub-distribution refers to the mechanical sampling sub-distribution in the mechanical sampling distribution that represents the contact area of ​​each fingertip with other fingertips. By deleting these sub-distributions, the mechanical data of the contact area can be prevented from interfering with subsequent analysis. The execution of this step ensures that the remaining mechanical sampling data is purer, providing a clear data basis for generating more accurate posture estimation and control instructions in subsequent steps.

[0135] In a specific example, the tactile sensors of a multi-fingered dexterous hand collected mechanical data from each fingertip during a grasping process. The system identified mechanical sampling subdistributions within this mechanical data that correlated with the contact areas of other fingertips. To ensure data accuracy, the system deleted these target mechanical sampling subdistributions and updated the mechanical sampling distribution. This processing enables more accurate mechanical input data, generating reliable pose estimation data and control instructions, enabling precise manipulation of the cable.

[0136] like Figure 5 As shown in the figure, it shows the specific flow of data during the control process of this solution:

[0137] Step S1: Acquiring mechanical data: The tactile sensor collects mechanical input values ​​of the fingertip and the cable during the process of stringing.

[0138] Step S2: Based on the mechanical data collected, the cable's position and posture are estimated to generate estimated position and posture data. This step uses the mechanical input data to calculate the cable's spatial position and posture.

[0139] Step S3: Use the pose estimation data to predict the trajectory of the tactile multi-fingered dexterous hand and generate expected dynamic data. This step plans the motion path of the fingers during the threading process.

[0140] Step S4: Using the expected dynamics data to control the tactile sensing multi-finger dexterous hand:

[0141] Step S4.1: Use the expected dynamics data to control the posture. Using this data, the system adjusts the finger posture in real time to ensure smooth and stable threading.

[0142] Step S4.2: Use the desired dynamics data to control the output force. This step ensures that the finger applies the appropriate force to enable it to firmly grasp and move the object.

[0143] Finally, the control data generated by the joint action of step S4.1 and step S4.2 realizes the line-smoothing control of the tactile-sensing multi-finger dexterous hand, and forms a loop from step S1 to S4.

[0144] In summary, in the embodiment of the present application, by generating accurate posture estimation data, the position and posture of the cable in the multi-finger dexterous hand can be grasped in real time; then, by comprehensively analyzing the posture estimation data and mechanical input data, the optimal motion trajectory of the multi-finger dexterous hand in the process of stringing is determined, and then the dynamic expectation data is converted into control instructions for the multi-finger dexterous hand, so as to achieve precise control of the stringing process, accurately guide the multi-finger dexterous hand to perform stringing operations, and ensure stability and accuracy in the stringing process. Therefore, based on the method of the embodiment of the present application, by acquiring and utilizing tactile sensing data in real time, precise control of the multi-finger dexterous hand is achieved, and finally stable and efficient grasping and following of the cable is achieved. The problems of the adaptability of the manipulator to the stringing task and low operating efficiency are solved.

[0145] refer to Figure 6 , which shows a robot autonomous thread-straightening system 30 based on a tactile-sensing multi-finger dexterous hand provided by an embodiment of the present application, comprising:

[0146] A tactile sensing multi-finger dexterous hand 301 with tactile sensors 30111 provided at the fingertips 3011 and a controller 302;

[0147] The controller 302 is used to obtain mechanical input data collected from the fingertips 3011 of the tactile sensing multi-finger dexterous hand 301, and generate posture estimation data for the cable A based on the mechanical input data, and generate dynamic expected data of the tactile sensing multi-finger dexterous hand 301 based on the posture estimation data and / or mechanical input data, and generate control instructions for the tactile sensing multi-finger dexterous hand 301 based on the dynamic expected data to control the process of straightening the cable through the tactile sensing multi-finger dexterous hand 301; the mechanical input data is generated by the fingertips 3011 grasping the cable A during the process of straightening the cable through the tactile sensing multi-finger dexterous hand 301; the posture estimation data is used to characterize the spatial posture of the cable A when the cable A is grasped in the tactile sensing multi-finger dexterous hand 301; the dynamic expected data is used to characterize the expected change in the motion trajectory of the tactile sensing multi-finger dexterous hand 301 during the grasping process.

[0148] Optionally, when obtaining mechanical input data collected from the fingertips of the tactile-sensing multi-fingered dexterous hand, the controller 302 is specifically used to determine the mechanical sampling distribution of each fingertip 3011 by setting multiple tactile sensors 30111 at each fingertip 3011, and decompose the mechanical sampling distribution into two mechanical sampling sub-distributions, so that the two mechanical sampling sub-distributions constitute a two-component Gaussian mixture model, and determine the mechanical sampling sub-distribution corresponding to the smaller distribution weight parameter of the two mechanical sampling sub-distributions as the mechanical input data of the fingertip corresponding to the mechanical sampling sub-distribution; each mechanical sampling sub-distribution has a corresponding distribution weight parameter.

[0149] Optionally, the controller 302 is further configured to delete a target mechanical sampling sub-distribution in the mechanical sampling distribution to update the mechanical sampling distribution; the target mechanical sampling sub-distribution is the mechanical sampling sub-distribution of the contact area between each fingertip and other fingertips in the mechanical sampling distribution.

[0150] Optionally, when the controller 302 generates posture estimation data for the cable based on the mechanical input data, it is specifically used to generate contact surface simulation data between the fingertip and the cable based on the mechanical input data, and use the contact surface simulation data as the first boundary value, and use the minimum value of the total variation composed of the distance between all contact points and the cable and the radius of the cable in the contact surface simulation data as the first target value to construct a posture optimization model of the cable, and solve the posture optimization model to obtain posture estimation data; the contact surface simulation data is used to characterize the force value at each contact point between the fingertip and the cable.

[0151] Optionally, the expected dynamic data includes expected data on the angle change of the clamping angle of the tactile sensing multi-finger dexterous hand on the cable. When the controller 302 generates the expected dynamic data of the tactile sensing multi-finger dexterous hand based on the posture estimation data and / or mechanical input data, it is specifically used to determine the expected angle change data to decrease when the mechanical input data is characterized by the friction force between the fingertip and the cable in the tangential direction exceeding a preset tangential force threshold, and to determine the expected angle change data to increase when the mechanical input data is characterized by the area of ​​the contact area between the fingertip and the cable being less than a preset contact area threshold; the tangential direction is the movement direction of the tactile sensing multi-finger dexterous hand.

[0152] Optionally, the expected dynamic data includes expected data on the change in the palm direction of the tactile-sensing multi-finger dexterous hand. When the controller 302 generates the expected dynamic data of the tactile-sensing multi-finger dexterous hand based on the posture estimation data and / or mechanical input data, it is specifically used to determine the expected data on the change in the palm direction as a rotation toward a target rotation direction when the posture estimation data characterizes that the grasping angle formed by the cable and the tactile-sensing multi-finger dexterous hand does not match a preset grasping angle threshold; the target rotation direction is the minimum rotation direction that makes the grasping angle match the grasping angle threshold.

[0153] Optionally, when the controller 302 generates control instructions for the tactile sensing multi-finger dexterous hand based on the expected dynamic data to control the process of threading through the tactile sensing multi-finger dexterous hand, it is specifically used to use the running trajectory limit data of the tactile sensing multi-finger dexterous hand as the second boundary value, and the actual dynamic state data and dynamic expected data of the tactile sensing multi-finger dexterous hand, and the dynamic state error data of the tactile sensing multi-finger dexterous hand determined as the second target value, to construct a joint motion optimization model of the tactile sensing multi-finger dexterous hand, and solve the joint motion optimization model to obtain control instructions.

[0154] Optionally, the dynamic expected data includes expected position data used to characterize the spatial position of the tactile sensing multi-finger dexterous hand. When the controller 302 generates control instructions for the tactile sensing multi-finger dexterous hand based on the dynamic expected data to control the process of straightening the cable through the tactile sensing multi-finger dexterous hand, the controller 302 is specifically used to bring the expected position data into the dynamic balance model of the tactile sensing multi-finger dexterous hand, and solve the dynamic balance model to determine the mechanical output value of each joint, and rotate the joint corresponding to each mechanical output value according to the mechanical output value of each joint to control the grasping of the cable; the dynamic balance model is used to characterize the relationship between the applied force value and the travel trajectory of each joint of the tactile sensing multi-finger dexterous hand when grasping the cable through the tactile sensing multi-finger dexterous hand.

[0155] Optionally, the multiple joints of the tactile-sensing multi-finger dexterous hand are arranged in sequence according to a preset arrangement direction. The controller 302 rotates the joint corresponding to each mechanical output value according to the mechanical output value of each joint to control the grasping of the cable. It is specifically used to rotate the joint corresponding to each mechanical output value in sequence according to the arrangement direction.

[0156] In summary, in the embodiment of the present application, by generating accurate posture estimation data, the position and posture of the cable in the multi-finger dexterous hand can be grasped in real time; then, by comprehensively analyzing the posture estimation data and mechanical input data, the optimal motion trajectory of the multi-finger dexterous hand in the process of stringing is determined, and then the dynamic expectation data is converted into control instructions for the multi-finger dexterous hand, so as to achieve precise control of the stringing process, accurately guide the multi-finger dexterous hand to perform stringing operations, and ensure stability and accuracy in the stringing process. Therefore, based on the method of the embodiment of the present application, by acquiring and utilizing tactile sensing data in real time, precise control of the multi-finger dexterous hand is achieved, and finally stable and efficient grasping and following of the cable is achieved. The problems of the adaptability of the manipulator to the stringing task and low operating efficiency are solved.

[0157] Reference Figure 7 , electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0158] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 502 may include one or more modules to facilitate interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate interaction between the multimedia component 508 and the processing component 502.

[0159] The memory 504 is used to store various types of data to support operations on the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, multimedia, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0160] The power supply assembly 506 provides power to the various components of the electronic device 500. The power supply assembly 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0161] The multimedia component 508 includes an interface that provides an output interface between the electronic device 500 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensors can not only sense the demarcation of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. In some embodiments, the multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When the electronic device 500 is in an operating mode, such as a capture mode or a multimedia mode, the front-facing camera and / or the rear-facing camera can receive external multimedia data. Each front-facing camera and the rear-facing camera can have a fixed optical lens system or have focal length and optical zoom capabilities.

[0162] The audio component 510 is used to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC) that receives external audio signals when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, or a voice recognition mode. The received audio signals may be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0163] The input / output I / O interface 512 provides an interface between the processing component 502 and peripheral interface modules, such as a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0164] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500. The sensor assembly 514 can also detect changes in the position of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and temperature changes of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0165] The communication component 516 is used to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0166] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the methods provided in the embodiments of the present application.

[0167] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, which can be executed by the processor 520 of the electronic device 500 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0168] Figure 8FIG2 is a block diagram of an electronic device 600 according to another embodiment of the present invention. For example, the electronic device 600 may be provided as a server.

[0169] Reference Figure 8 The electronic device 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632 for storing instructions executable by the processing component 622, such as an application. The application stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute the instructions to perform the method provided in the embodiments of the present application.

[0170] The electronic device 600 may further include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0171] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0172] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A robot autonomous thread-straightening method based on tactile sensing multi-finger dexterous hand, characterized in that: include: obtaining mechanical input data collected from the fingertips of a tactile sensing multi-fingered dexterous hand; The mechanical input data is generated by the fingertips grasping the cable during the process of stringing the cable with the tactile sensing multi-finger dexterous hand; generating pose estimation data for the cable according to the mechanical input data; The pose estimation data is used to represent the spatial pose of the cable when the cable is grasped in the tactile-sensing multi-finger dexterous hand; generating expected dynamic data of the tactile-sensing multi-finger dexterous hand based on the pose estimation data and / or the mechanical input data; wherein the expected dynamic data is used to represent an expected change in the motion trajectory of the tactile-sensing multi-finger dexterous hand during a grasping process; According to the expected dynamics data, a control instruction for the tactile-sensing multi-finger dexterous hand is generated to control the process of threading by the tactile-sensing multi-finger dexterous hand.

2. The method according to claim 1, wherein The obtaining of mechanical input data collected from the fingertips of the tactile sensing multi-fingered dexterous hand includes: Determining the mechanical sampling distribution of each fingertip by arranging a plurality of tactile sensors at each fingertip; Decomposing the mechanical sampling distribution into two mechanical sampling sub-distributions, so that the two mechanical sampling sub-distributions constitute a two-component Gaussian mixture model; each of the mechanical sampling sub-distributions has a corresponding distribution weight parameter; The mechanical sampling sub-distribution corresponding to the smaller distribution weight parameter of the two mechanical sampling sub-distributions is determined as the mechanical input data of the fingertip corresponding to the mechanical sampling sub-distribution.

3. The method according to claim 1, wherein Generating pose estimation data for the cable according to the mechanical input data includes: generating contact surface simulation data between the fingertip and the cable according to the mechanical input data; wherein the contact surface simulation data is used to represent the force value at each contact point between the fingertip and the cable; Constructing a posture optimization model for the cable using the contact surface simulation data as a first boundary value and a minimum value of a total variation consisting of the distances between all contact points and the cable and the radius of the cable in the contact surface simulation data as a first target value; Solve the pose optimization model to obtain the pose estimation data.

4. The method according to claim 1, wherein The expected dynamics data includes expected data on the angle change of the clamping angle of the tactile-sensing multi-fingered dexterous hand on the cable. Generating the expected dynamics data of the tactile-sensing multi-fingered dexterous hand based on the posture estimation data and / or the mechanical input data includes: When the mechanical input data indicates that the friction force between the fingertip and the cable in a tangential direction exceeds a preset tangential force threshold, the expected angle change data is determined to decrease; the tangential direction is the movement direction of the tactile-sensing multi-finger dexterous hand; When the mechanical input data indicates that the area of ​​the contact region between the fingertip and the cable is smaller than a preset contact area threshold, the expected angle change data is determined to be increasing.

5. The method according to claim 1, wherein The expected dynamics data includes expected data on changes in the palm direction of the tactile-sensing multi-finger dexterous hand. Generating the expected dynamics data of the tactile-sensing multi-finger dexterous hand based on the posture estimation data and / or the mechanical input data includes: When the posture estimation data indicates that the gripping angle formed by the cable and the tactile-sensing multi-finger dexterous hand does not match a preset gripping angle threshold, the expected palm direction change data is determined as a rotation toward a target rotation direction; the target rotation direction is the minimum rotation direction that makes the gripping angle match the gripping angle threshold.

6. The method according to claim 1, wherein Generating a control instruction for the tactile-sensing multi-finger dexterous hand according to the expected dynamics data to control the thread-straightening process by the tactile-sensing multi-finger dexterous hand includes: Using the trajectory limit data of the tactile-sensing multi-finger dexterous hand as a second boundary value, and using the actual dynamic state data of the tactile-sensing multi-finger dexterous hand and the dynamic expected data, and the dynamic state error data of the tactile-sensing multi-finger dexterous hand determined as a second target value, a joint motion optimization model of the tactile-sensing multi-finger dexterous hand is constructed; Solve the joint motion optimization model to obtain the control instruction.

7. The method according to claim 1, wherein The expected dynamic data includes expected position data for characterizing the spatial position of the tactile-sensing multi-finger dexterous hand. Generating a control instruction for the tactile-sensing multi-finger dexterous hand based on the expected dynamic data to control a thread-straightening process performed by the tactile-sensing multi-finger dexterous hand includes: The expected position data is introduced into a dynamic balance model of the tactile-sensing multi-finger dexterous hand; the dynamic balance model is used to characterize the relationship between the applied force value of each joint of the tactile-sensing multi-finger dexterous hand and the travel trajectory when the tactile-sensing multi-finger dexterous hand is grasped during the process of threading; Solving the dynamic equilibrium model to determine the mechanical output value of each joint; According to the mechanical output value of each joint, the joint corresponding to each mechanical output value is rotated to control the gripping of the cable.

8. A robot autonomous thread-straightening system based on a tactile sensing multi-finger dexterous hand, characterized in that: include: A tactile sensing multi-finger dexterous hand and controller having tactile sensors disposed on the fingertips; The controller is used to obtain mechanical input data collected from the fingertips of the tactile-sensing multi-finger dexterous hand, and generate posture estimation data of the cable based on the mechanical input data, and generate dynamic expected data of the tactile-sensing multi-finger dexterous hand based on the posture estimation data and / or the mechanical input data, and generate control instructions for the tactile-sensing multi-finger dexterous hand based on the dynamic expected data, so as to control the process of threading the cable by the tactile-sensing multi-finger dexterous hand; the mechanical input data is generated by the fingertips grasping the cable during the process of threading the cable by the tactile-sensing multi-finger dexterous hand; The pose estimation data is used to characterize the spatial pose of the cable when the cable is grasped in the tactile sensing multi-finger dexterous hand; the dynamic expectation data is used to characterize the expected change in the motion trajectory of the tactile sensing multi-finger dexterous hand during the grasping process.

9. An electronic device, characterized in that: include: a processor, a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 7.

Images