Robot autonomous wire stroking method and system based on tactile sensing multi-fingered dexterous hand

Abstract

The invention discloses a robot autonomous wire stroking method, system and device based on a tactile sensing multi-fingered dexterous hand and a storage medium, and belongs to the field of manipulator control. The robot autonomous wire stroking method comprises the steps that mechanical input data of finger tips of the tactile sensing multi-fingered dexterous hand is acquired and collected; generating pose estimation data of the cable according to the mechanical input data; according to the pose estimation data and / or the mechanical input data, dynamic expected data of the touch sensing multi-fingered dexterous hand is generated; according to the dynamics expected data, a control instruction for the touch sensing multi-fingered dexterous hand is generated so as to control the wire stroking process through the touch sensing multi-fingered dexterous hand. According to the multi-fingered dexterous hand, the touch sensing data is obtained and utilized in real time, accurate control over the multi-fingered dexterous hand is achieved, and finally stable and efficient grabbing and following of the cable are achieved. The problems that a manipulator is low in adaptability to a wire stroking task and low in operation efficiency are solved.

Description

Technical Field

[0001] The present application belongs to the field of robot control, and specifically relates to a robot autonomous thread-winding method, system, device and storage medium based on tactile sensing multi-finger dexterous hand. Background Art

[0002] Handling deformable linear objects (DLOs), such as cables, ropes, and pipes, is a common but challenging task in modern industrial and robotic applications. Such tasks include operations such as cable routing, packaging, assembly, and maintenance. The wiring process requires the robot to be able to slide along the object while maintaining a firm grip to prevent the object from falling off. This operation usually requires a high level of dexterity and precision.

[0003] In the prior art, although there are some dedicated end effectors that perform well in specific tasks, they are not flexible in handling different types of objects, are expensive and lack versatility. Therefore, in order to achieve the threading process, a rigid grasping and re-grasping method is usually used. Specifically, parallel grippers are used to perform multiple grasping and repositioning to achieve control of the object.

[0004] However, since the parallel gripper has only one degree of freedom, the rigid grasping and re-grasping method has difficulty in handling deformable linear objects of various shapes and materials, and cannot adapt to complex wire-straightening tasks. At the same time, since it is difficult to balance the grasping force and the wire-straightening force, the parallel gripper needs to re-grasp frequently during execution, and there is also the problem of low operating efficiency. Summary of the invention

[0005] The present application aims to provide a method, system, device and storage medium for autonomous thread-threading of a robot based on a tactile sensing multi-finger dexterous hand, at least to solve the problems of the adaptability of the robot to thread-threading tasks and low operating efficiency.

[0006] In a first aspect, the present application discloses a robot autonomous threading method based on a tactile sensing multi-finger dexterous hand, comprising: Acquiring mechanical input data collected from the fingertips of the tactile sensing multi-finger dexterous hand; the mechanical input data is generated by the fingertips grasping the cable during the process of the tactile sensing multi-finger dexterous hand pulling the cable; Generating position and posture estimation data of the cable according to the mechanical input data; the position and posture estimation data is used to characterize the spatial position and posture of the cable when the cable is grasped in the tactile sensing multi-finger dexterous hand; Generate expected dynamics data of the tactile-sensing multi-finger dexterous hand according to the posture estimation data and / or the mechanical input data; the expected dynamics data is used to characterize the expected change of the motion trajectory of the tactile-sensing multi-finger dexterous hand during the grasping process; According to the expected dynamics data, a control instruction for the tactile sensing multi-finger dexterous hand is generated to control the process of threading by the tactile sensing multi-finger dexterous hand.

[0007] In a second aspect, the present application also discloses a robot autonomous thread-straightening system based on a tactile sensing multi-finger dexterous hand, comprising: A tactile sensing multi-finger dexterous hand and a controller having tactile sensors disposed at the fingertips; The controller is used to obtain mechanical input data collected from the fingertips of the tactile sensing multi-finger dexterous hand, and generate posture estimation data for the cable based on the mechanical input data, and generate dynamic expected data of the tactile sensing multi-finger dexterous hand based on the posture estimation data and / or the mechanical input data, and generate control instructions for the tactile sensing multi-finger dexterous hand based on the dynamic expected data to control the process of smoothing the cable by the tactile sensing multi-finger dexterous hand; the mechanical input data is generated by the grasping of the cable by the fingertips during the process of smoothing the cable by the tactile sensing multi-finger dexterous hand; the posture estimation data is used to characterize the spatial posture of the cable when the cable is grasped in the tactile sensing multi-finger dexterous hand; the dynamic expected data is used to characterize the expected changes in the motion trajectory of the tactile sensing multi-finger dexterous hand during the grasping process.

[0008] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application further discloses a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In summary, in the embodiment of the present application, by generating accurate posture estimation data, the position and posture of the cable in the multi-fingered dexterous hand can be grasped in real time; then, by comprehensively analyzing the posture estimation data and mechanical input data, the optimal motion trajectory of the multi-fingered dexterous hand in the process of pulling the wire is determined, and then the dynamic expectation data is converted into the control instructions of the multi-fingered dexterous hand, so as to achieve precise control of the wire pulling process, accurately guide the multi-fingered dexterous hand to perform the wire pulling operation, and ensure the stability and accuracy in the wire pulling process. Therefore, based on the method of the embodiment of the present application, by acquiring and utilizing tactile sensing data in real time, precise control of the multi-fingered dexterous hand is achieved, and finally stable and efficient grasping and following of the cable is achieved. The problems of adaptability of the manipulator to the wire pulling task and low operating efficiency are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the attached picture: Figure 1 is a schematic diagram of the key structure of the tactile sensing multi-finger dexterous hand involved in this application; Figure 2 It is a flowchart of the steps of a robot autonomous thread-straightening method based on a tactile sensing multi-finger dexterous hand provided in an embodiment of the present application; Figure 3 is a flowchart of another method for autonomous thread-straightening by a robot based on a tactile sensing multi-finger dexterous hand provided in an embodiment of the present application; Figure 4 This is a comparison diagram of the two robot wire drawing processes provided by the embodiments of the present application; Figure 5 It is the data flow process under the embodiment of the present application; Figure 6 It is a block diagram of a robot autonomous thread-straightening system based on a tactile sensing multi-finger dexterous hand provided in an embodiment of the present application; Figure 7 is a block diagram of an electronic device according to an embodiment of the present application; Figure 8 is a block diagram of an electronic device according to another embodiment of the present application; Wherein: 301-tactile sensing multi-finger dexterous hand; 3011-finger tip; 30111-tactile sensor; 401-parallel gripper; A-cable. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0013] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0014] In this application, deformable linear objects (DLO) will be collectively referred to as "cables" unless otherwise specified. Specifically, deformable linear objects refer to linear objects with flexible and bendable properties, including but not limited to cables, optical fibers, wires, etc. All DLO objects mentioned in the description and embodiments of this application are expressed using the term "cable". This unified name helps to simplify the expression of technical content and facilitates readers to understand the technical solution of this application, and does not affect the comprehensive understanding and protection scope of this application.

[0015] In this application, "threading" refers to the process of grasping and moving cables by using a tactile-sensing multi-finger dexterous hand to adjust the gripping force and motion trajectory in real time to achieve stable following and positioning of the cable. That is, the process of In-Hand Following of Deformable LinearObjects Using Dexterous Fingers with Tactile Sensing. Specifically, the dexterous hand uses tactile sensors to collect mechanical input data and posture estimation data, which are used to generate control instructions in real time to adjust the movement of each joint and the force applied. In this way, the dexterous hand can dynamically adapt to changes in the shape and position of the cable, ensuring stable and precise control of the cable during operation.

[0016] The main goal of the cable-winding process is to improve the efficiency and accuracy of cable operations and prevent the cables from slipping or getting damaged during movement. By adjusting the gripping force and motion trajectory in real time, the dexterous hand can better adapt to different operating environments and task requirements, thereby improving the stability and reliability of operations. This process is not only applicable to the adjustment of cables in space, but also includes the precise control of the position and shape of cables during installation and maintenance.

[0017] like Figure 1As shown, the tactile sensing multi-finger dexterous hand (Dexterous Fingers with Tactile Sensing) 301 is an advanced manipulator device, which is characterized in that each fingertip 3011 is equipped with a tactile sensor 30111. These sensors can detect and collect contact mechanical data with the cable during the wire drawing process. By real-time monitoring of these mechanical input data, the tactile sensing multi-finger dexterous hand can generate spatial posture estimation data and motion trajectory data, thereby achieving stable grasping and precise operation of objects. The tactile sensing multi-finger dexterous hand not only has the flexibility of multiple degrees of freedom, but also can optimize the grasping posture and path through intelligent control algorithms to achieve efficient and precise wire drawing operations.

[0018] Based on the above ideas, Figure 2 As shown, a robot autonomous thread-straightening method based on a tactile sensing multi-finger dexterous hand is provided in an embodiment of the present application.

[0019] The method may include the following steps: Step 101, obtaining mechanical input data collected from the fingertips of the tactile sensing multi-finger dexterous hand.

[0020] Among them, the mechanical input data is generated by the fingertips grasping the cable during the process of stringing the cable through the tactile sensing multi-finger dexterous hand.

[0021] In some embodiments of the present application, the mechanical input data collected from the fingertips of the tactile sensing multi-finger dexterous hand is obtained in order to monitor the force feedback in the grasping process in real time, so as to ensure the stability of the grasping and prevent the cable from slipping. In the process of pulling the cable by the tactile sensing multi-finger dexterous hand, the mechanical input data of the fingertips can reflect the magnitude and direction of the force applied to the cable. The mechanical input data includes information such as pressure and shear force detected by the tactile sensor. The execution of this step ensures that the system can adjust the gripping force according to the real-time data, prevent the cable from falling off during the operation, and improve the stability and safety of the operation.

[0022] In a specific example, a tactile sensing multi-finger dexterous hand grasps a section of flexible cable and starts to perform a cable-winding operation. The tactile sensor collects real-time mechanical input data applied by the fingertips on the cable, including changes in pressure and shear force. The system will adjust the gripping force of each fingertip based on these real-time data, so that the cable remains stable throughout the entire cable-winding process, ultimately achieving safe and efficient cable operation.

[0023] Step 102: Generate pose estimation data for the cable based on the mechanical input data.

[0024] Among them, the posture estimation data is used to characterize the spatial posture of the cable when the cable is grasped in the tactile sensing multi-finger dexterous hand.

[0025] In some embodiments of the present application, the pose estimation data of the cable is generated based on the mechanical input data in order to accurately grasp the spatial position and posture of the cable in the multi-fingered dexterous hand. The mechanical input data reflects the changes in the force on the cable during the grasping process, and the spatial pose of the cable can be derived from these data. The pose estimation data is used to characterize the specific position and posture of the cable when it is grasped in the tactile sensing multi-fingered dexterous hand, thereby providing an accurate basis for subsequent motion control. The execution of this step ensures that the system can adjust the grasp and motion trajectory in real time, improving the accuracy and reliability of the line-pulling operation.

[0026] In a specific example, a tactile sensing multi-finger dexterous hand grasps a section of flexible cable and starts to perform a cable-winding operation. The tactile sensor collects the mechanical input data applied by the fingertips on the cable in real time, including the changes in pressure and shear force. Based on these real-time data, the system generates estimated data for the cable's posture, clarifies the specific position and posture of the cable in the dexterous hand, and adjusts the movement trajectory of the fingers and arms accordingly to ensure the stability and accuracy of the cable during the cable-winding process.

[0027] Step 103: Generate expected dynamic data of the tactile sensing multi-finger dexterous hand according to the posture estimation data and / or the mechanical input data.

[0028] Among them, the dynamic expectation data is used to characterize the expected changes in the motion trajectory of the tactile sensing multi-finger dexterous hand during the grasping process.

[0029] In some embodiments of the present application, the expected dynamic data of the tactile sensing multi-finger dexterous hand is generated based on at least one of the posture estimation data and the mechanical input data in order to determine the optimal motion trajectory of the dexterous hand during the grasping process. The posture estimation data reflects the specific position and posture of the cable in the dexterous hand, while the mechanical input data provides the change in the force applied to the cable during the grasping process. The expected dynamic data is used to characterize the expected change in the motion trajectory of the tactile sensing multi-finger dexterous hand during the grasping process, thereby achieving precise control of the grasping process. The execution of this step ensures that the system can adjust the motion trajectory according to the real-time data and optimize the flexibility and accuracy of the line-pulling operation.

[0030] In a specific example, a tactile sensing multi-finger dexterous hand grasps a section of flexible cable and starts to perform a cable-winding operation. The tactile sensor collects the mechanical input data applied by the fingertips on the cable in real time, and the system generates the cable's posture estimation data based on these real-time data. By comprehensively analyzing the posture estimation data and mechanical input data, the dynamic expectation data of the tactile sensing multi-finger dexterous hand generated by the system can adjust the motion trajectory of the dexterous hand and ensure the stability and accuracy of the cable during the cable-winding process.

[0031] Step 104, generating control instructions for the tactile sensing multi-finger dexterous hand according to the dynamics expectation data, so as to control the process of threading by the tactile sensing multi-finger dexterous hand.

[0032] In some embodiments of the present application, control instructions for the tactile sensing multi-finger dexterous hand are generated based on the dynamic expectation data in order to achieve precise control of the cable drawing process. The dynamic expectation data characterizes the changes in the motion trajectory of the tactile sensing multi-finger dexterous hand during the grasping process, and corresponding control instructions can be generated through these data to control the movement of the dexterous hand. The control instructions include instructions for the specific position and strength of each finger joint, thereby ensuring that the fingers can move according to the expected trajectory and strength during the cable drawing process. The execution of this step ensures that the system can be adjusted according to the real-time data to achieve stable and efficient cable drawing.

[0033] In a specific example, a tactile sensing multi-finger dexterous hand grasps a section of flexible cable and starts to perform a cable-winding operation. Based on the previously generated dynamic expectation data, the system generates corresponding control instructions, which include instructions for the specific position and force of each finger joint. The dexterous hand performs the cable-winding operation according to these control instructions, ensuring that each finger moves according to the expected trajectory and force, ultimately achieving precise control of the cable and stable cable-winding.

[0034] In summary, in the embodiment of the present application, by generating accurate posture estimation data, the position and posture of the cable in the multi-fingered dexterous hand can be grasped in real time; then, by comprehensively analyzing the posture estimation data and mechanical input data, the optimal motion trajectory of the multi-fingered dexterous hand in the process of pulling the wire is determined, and then the dynamic expectation data is converted into the control instructions of the multi-fingered dexterous hand, so as to achieve precise control of the wire pulling process, accurately guide the multi-fingered dexterous hand to perform the wire pulling operation, and ensure the stability and accuracy in the wire pulling process. Therefore, based on the method of the embodiment of the present application, by acquiring and utilizing tactile sensing data in real time, precise control of the multi-fingered dexterous hand is achieved, and finally stable and efficient grasping and following of the cable is achieved. The problems of adaptability of the manipulator to the wire pulling task and low operating efficiency are solved.

[0035] Figure 3 Another robot autonomous thread-straightening method based on tactile sensing multi-finger dexterous hands is provided in an embodiment of the present application.

[0036] The method may include the following steps: Step 201 : determining the mechanical sampling distribution of each fingertip by setting a plurality of tactile sensors on each fingertip.

[0037] In some embodiments of the present application, by setting up multiple tactile sensors at each fingertip, the mechanical sampling distribution of each fingertip is determined in order to accurately obtain and analyze the mechanical information of the tactile sensor during the grasping process. These mechanical information reflects the specific force conditions of each fingertip when grasping the cable. The mechanical sampling distribution refers to the distribution of mechanical data collected by the tactile sensor over a period of time. The execution of this step ensures that the system can fully understand the mechanical characteristics of the grasping process, providing a basis for subsequent data processing and control instruction generation.

[0038] In a specific example, according to the multiple tactile sensors installed on each fingertip of the tactile sensing multi-finger dexterous hand, when performing the cable-straightening operation, the tactile sensors collect the mechanical data applied by each fingertip on the cable in real time, which can include pressure and shear force in different positions and directions. By analyzing these mechanical data, the system determines the mechanical sampling distribution of each fingertip, and then fully understands the force conditions of each fingertip during the grasping process.

[0039] Step 202: decompose the mechanical sampling distribution into two mechanical sampling sub-distributions, so that the two mechanical sampling sub-distributions constitute a two-component Gaussian mixture model.

[0040] Among them, each mechanical sampling sub-distribution has a corresponding distribution weight parameter.

[0041] In some embodiments of the present application, the mechanical sampling distribution is decomposed into two mechanical sampling sub-distributions in order to more accurately analyze and process the mechanical data collected by the tactile sensor. This process uses a two-component Gaussian mixture model (Gaussian Mixture Model, GMM) to model the mechanical sampling distribution of the tactile sensor, so that the two mechanical sampling sub-distributions can correspond to different force conditions respectively. The two-component Gaussian mixture model is a statistical model that fits complex data distributions by a weighted combination of two normal distributions. Each mechanical sampling sub-distribution has a corresponding distribution weight parameter to reflect its respective importance and contribution. Specifically, the mechanical input data can be approximated into two Gaussian distributions by applying the expectation maximization (EM) algorithm, one representing the contact area and the other representing the non-contact area. According to the data points in the distribution area, the mean (μ) and variance (σ) of each Gaussian distribution, as well as their respective weight parameters, are calculated and determined. The execution of this step ensures that the system can more accurately separate and analyze different force conditions, thereby providing a more accurate data basis for subsequent posture estimation and control instruction generation.

[0042] In a specific example, suppose the system is handling a cable, and the mechanical sensors at each fingertip record a large amount of gripping mechanical data. These data points are distributed in different force value ranges, some of which reflect the actual contact between the fingertip and the cable surface (higher force value), while other data points reflect the non-contact area (lower force value). By applying the expectation maximization algorithm, the system can decompose this data into two Gaussian distributions, one representing the contact area and the other representing the non-contact area. Then, the system calculates the mean (μ) and variance (σ) of each Gaussian distribution, as well as the weight parameters of each distribution. In this way, the system can clearly distinguish the data in the contact and non-contact areas, providing accurate mechanical data support for subsequent posture estimation and trajectory generation.

[0043] Step 203 , determining the mechanical sampling sub-distribution corresponding to the smaller distribution weight parameter of the two mechanical sampling sub-distributions as the mechanical input data of the fingertip corresponding to the mechanical sampling sub-distribution.

[0044] In some embodiments of the present application, the mechanical sampling sub-distribution corresponding to the smaller distribution weight parameter of the distribution weight parameters of the two mechanical sampling sub-distributions is determined as the mechanical input data of the fingertip corresponding to the mechanical sampling sub-distribution, in order to accurately identify and extract the effective force information sensed by the tactile sensor. In the previous step, the mechanical sampling data has been decomposed into two sub-distributions, and each sub-distribution has its corresponding distribution weight parameter. The distribution weight parameter reflects the relative importance of the sub-distribution in the overall data. By selecting the sub-distribution with a smaller distribution weight parameter as the mechanical input data, the noise data can be effectively removed to ensure the accuracy of the data. The execution of this step ensures that the subsequent posture estimation and control instruction generation are based on accurate and reliable force data, thereby improving the stability and accuracy of the system.

[0045] In a specific example, mechanical data over a period of time has been collected by the tactile sensor of the tactile sensing multi-fingered dexterous hand and decomposed into two sub-distributions. The system calculates the distribution weight parameters of the two sub-distributions, one is 0.3 and the other is 0.7. The system selects the sub-distribution with a distribution weight parameter of 0.3 as the mechanical input data because it better represents the effective force information generated by the tactile sensor during the grasping process. Through this selection, the system can eliminate more noise data and ensure the accuracy of pose estimation and control command generation.

[0046] Step 204: Generate pose estimation data for the cable based on the mechanical input data.

[0047] Among them, the posture estimation data is used to characterize the spatial posture of the cable when the cable is grasped in the tactile sensing multi-finger dexterous hand.

[0048] The method shown in this step has been explained in step 102 and will not be repeated here.

[0049] Optionally, step 204 includes the following sub-steps: Sub-step 2041, generating contact surface simulation data between the finger tip and the cable according to the mechanical input data.

[0050] The contact surface simulation data is used to characterize the force value at each contact point between the fingertip and the cable.

[0051] In some embodiments of the present application, contact surface simulation data between the fingertip and the cable is generated based on the mechanical input data in order to more accurately analyze the force conditions of the tactile sensor during the line drawing process. The contact surface simulation data reflects the specific force value at each contact point between the fingertip and the cable. In this process, the tactile sensor collects the mechanical input data in real time and converts it into corresponding contact surface simulation data. The contact surface simulation data includes the pressure and shear force values ​​of the fingertip and the cable at different contact points. Through these data, the force conditions of the fingertip and the cable can be accurately characterized, thereby providing a more accurate basis for subsequent posture estimation and control instruction generation.

[0052] In a specific example, the tactile sensor of the tactile sensing multi-finger dexterous hand collects mechanical input data and starts to perform the cable pulling operation. The system generates simulated data of the contact surface between the fingertips and the cable based on these real-time collected data. These simulated data reflect the pressure and shear force values ​​at different contact points, ensuring accurate characterization of the force conditions of the fingertips and the cable. Through this processing, more detailed force information can be obtained, thereby providing accurate data support for posture estimation and motion control.

[0053] Sub-step 2042 uses the contact surface simulation data as the first boundary value, and uses the minimum value of the total variation of the distance between all contact points and the cable and the radius of the cable in the contact surface simulation data as the first target value to construct a cable posture optimization model.

[0054] In some embodiments of the present application, the contact surface simulation data is used as the first boundary value, and the minimum value of the total variation formed by the distance between all contact points and the cable and the radius of the cable in the contact surface simulation data is used as the first target value to construct a cable posture optimization model in order to accurately estimate the spatial posture of the cable during the grasping process. The contact surface simulation data reflects the specific contact situation between the fingertips and the cable. These data are used as the boundary values ​​of the posture optimization model to ensure that the model can truly reflect the actual situation. The minimum value of the total variation refers to the minimum error formed by the difference in the distance between all contact points and the cable axis and the radius of the cable in the contact surface simulation data. By constructing a posture optimization model, the spatial posture of the cable can be estimated more accurately, thereby improving the accuracy of the system's wire drawing control.

[0055] In a specific example, the tactile sensors of the tactile sensing multi-finger dexterous hand have been used to collect simulated data on the contact surface between the fingertips and the cable. The system uses this data as the first boundary value, and based on the minimum value of the total variation of the distance between all contact points and the cable axis and the cable radius in these data, constructs a cable posture optimization model. Through this model, the system can accurately estimate the spatial position and posture of the cable in the dexterous hand, and adjust the grip and motion control accordingly to ensure the accuracy of the cable pulling operation.

[0056] Sub-step 2043, solving the pose optimization model to obtain pose estimation data.

[0057] In some embodiments of the present application, solving the posture optimization model to obtain posture estimation data is to accurately calculate the spatial position and posture of the cable during the grasping process. The posture optimization model is based on the previously generated contact surface simulation data and the minimum total variation. By solving the model, the posture estimation data of the cable can be obtained. The posture estimation data is used to characterize the specific position and direction of the cable in the tactile sensing multi-fingered dexterous hand, thereby providing an accurate basis for the generation of control instructions. By solving the optimization model, the system can adjust the grasping and motion trajectory in real time to ensure the high precision and stability of the line-winding operation.

[0058] In a specific example, the tactile sensors of the tactile sensing multi-fingered dexterous hand have generated contact surface simulation data and constructed a cable pose optimization model. The system solves the model and obtains the cable pose estimation data, which includes the specific position and orientation of the cable in space. Based on this data, the system adjusts the grasping and motion control of the dexterous hand to ensure that the cable remains stable and precise during the cable pulling process.

[0059] Optionally, in a further embodiment, the forward kinematics of the finger can be used to transform all contact points of the two sensors into the same coordinate system, and then solve the pose estimation data. In this case, the specific pose optimization model has the following form: ; Among them, Θ represents the optimization variable, and the difference term d is calculated. i (Θ)-r represents the distance from the ith contact point to the cable (predicted straight line), W r is the predicted parameter, ψ represents the unit direction under the predicted posture, ξ x Represents the component of the parameter point of the straight line in the x direction, o i,x Represents the component in the x direction at the ith contact point.

[0060] Step 205: Generate expected dynamic data of the tactile sensing multi-finger dexterous hand according to the posture estimation data and / or the mechanical input data.

[0061] Among them, the dynamic expectation data is used to characterize the expected changes in the motion trajectory of the tactile sensing multi-finger dexterous hand during the grasping process.

[0062] The method shown in this step has been explained in step 103 and will not be repeated here.

[0063] like Figure 4 As shown, the figure shows the parallel gripper 401 (parallel gripper) ( Figure 4 -a) and the comparison results of grasping cable A by the tactile sensing multi-finger dexterous hand 301. In the figure, the dotted arrow indicates the gravity on cable A, the thick solid arrow indicates the maximum contact feedback force that the parallel gripper 401 and the tactile sensing multi-finger dexterous hand 301 can receive, and the thin solid arrow indicates the contact force of cable A directly acting on the parallel gripper 401 and the tactile sensing multi-finger dexterous hand 301. It can be seen that it is difficult for the parallel gripper 401 to find a suitable gripping force to balance following and holding, and because the tactile sensing multi-finger dexterous hand 301 has more degrees of freedom, these two goals can be decoupled by forming a "V" shape: that is, for holding, a larger gripping force can be applied to make the bottoms of the two fingertips in close contact; for following, a moderate contact force can be applied by adjusting the angle of the V shape (i.e., the gripping angle θ).

[0064] Through experiments, we finally found that the grip angle θ of a specific cable is initially determined during the grip process: the fingers first grip the cable in parallel; then, the grip angle is increased until any tactile sensor loses contact with the cable. During the following process, if the shear force on the tactile sensor is greater than the threshold, the grip angle is increased; if the in-hand contact does not meet the requirements for reliable perception, the grip angle is reduced. Limiting the grip angle to between 15° and 50° and setting the grip force to a constant value is a relatively more preferred result.

[0065] There are the following further options for step 205: Optionally, the expected dynamics data includes expected data of angle change of the cable clamping angle of the tactile sensing multi-finger dexterous hand, and step 205 includes the following sub-steps: Sub-step 2051 , when the mechanical input data is characterized as the friction force between the finger tip and the cable in the tangential direction exceeding a preset tangential force threshold, determining the angle change expected data as decreasing.

[0066] Among them, the tangential direction is the movement direction of the tactile sensing multi-fingered dexterous hand.

[0067] In some embodiments of the present application, when the mechanical input data is characterized as the friction between the fingertip and the cable in the tangential direction exceeding the preset tangential force threshold, the angle change expected data is determined to be reduced in order to prevent the cable from sliding or being damaged due to excessive tangential friction. In this process, the mechanical input data reflects the friction between the fingertip and the cable in the tangential direction. When the friction exceeds the preset threshold, the system will automatically set the angle change expected data to decrease to reduce the friction. The tangential direction refers to the movement direction of the tactile sensing multi-finger dexterous hand. The execution of this step can effectively reduce the friction, ensure that the cable is not damaged during the grasping process, and maintain the stability and safety of the operation.

[0068] In a specific example, the multi-finger dexterous hand starts to perform a cable-straightening operation through tactile sensing. When the tactile sensor detects that the friction between the fingertips and the cable in the tangential direction exceeds the preset tangential force threshold, the system automatically sets the angle change expectation data to decrease to reduce the friction applied to the cable. Through this adjustment, it can be ensured that the cable will not slip or be damaged due to excessive friction during the cable-straightening process, thereby achieving stable grasping and cable-straightening operations.

[0069] Sub-step 2052, when the mechanical input data indicates that the area of ​​the contact region between the fingertip and the cable is smaller than a preset contact area threshold, determining the angle change expected data to be increased.

[0070] In some embodiments of the present application, when the mechanical input data is characterized as the area of ​​the contact area between the fingertip and the cable being less than a preset contact area threshold, the angle change expected data is determined to be increased in order to ensure the stable grasping of the tactile sensing multi-finger dexterous hand during the line drawing process. The mechanical input data reflects the specific area of ​​the contact area between the fingertip and the cable. When the contact area is less than the preset threshold, the system will automatically set the angle change expected data to increase to increase the contact area. Through this adjustment, it can be ensured that the contact area with the cable during the grasping process reaches an ideal state, thereby improving the grasping stability and control accuracy.

[0071] In a specific example, the multi-finger dexterous hand starts to perform a wire-straightening operation through tactile sensing. When the tactile sensor detects that the area of ​​the contact area between the fingertip and the cable is less than a preset contact area threshold, the system automatically sets the angle change expectation data to increase to increase the contact area between the fingertip and the cable. Through this adjustment, it is possible to ensure that the cable maintains a stable grip during the wire-straightening process, thereby achieving a more accurate and reliable wire-straightening operation.

[0072] Optionally, the expected dynamics data includes expected data on the change in the palm direction of the tactile sensing multi-finger dexterous hand, and step 205 includes the following sub-steps: Sub-step 2053, when the posture estimation data indicates that the gripping angle formed by the cable and the tactile sensing multi-finger dexterous hand does not match the preset gripping angle threshold, the expected data of the palm direction change is determined as rotation in the target rotation direction.

[0073] The target rotation direction is the minimum rotation direction that makes the grasping angle match the grasping angle threshold.

[0074] In some embodiments of the present application, when the posture estimation data characterizes that the grasping angle formed by the cable and the tactile sensing multi-fingered dexterous hand does not match the preset grasping angle threshold, the expected data of the palm direction change is determined to rotate in the target rotation direction in order to ensure that the grasping angle is consistent with the preset ideal grasping angle. The posture estimation data provides the specific position and posture of the cable relative to the dexterous hand. When it is detected that the grasping angle does not meet the preset value, the system will adjust the expected data of the palm direction change to rotate it in the target rotation direction. The target rotation direction refers to the minimum rotation direction that matches the grasping angle with the preset grasping angle threshold. The execution of this step can optimize the grasping angle and improve the grasping stability and operation accuracy.

[0075] In a specific example, a multi-finger dexterous hand starts to perform a wire-straightening operation through tactile sensing. Based on the posture estimation data, the system detects that the grip angle between the cable and the dexterous hand deviates from the preset grip angle threshold. In order to adjust the grip angle, the system sets the expected data of the palm direction change to rotate in the target rotation direction, that is, to the minimum rotation direction that can make the grip angle match the preset value. Through this adjustment, it is possible to ensure that the cable maintains a stable and precise grip angle during the wire-straightening process, and ultimately achieve an efficient and stable wire-straightening operation.

[0076] Step 206, generating control instructions for the tactile sensing multi-finger dexterous hand according to the dynamics expectation data, so as to control the process of threading by the tactile sensing multi-finger dexterous hand.

[0077] The method shown in this step has been described in step 104 and will not be repeated here.

[0078] Optionally, step 206 includes the following sub-steps: In sub-step 2061, the running trajectory limit data of the tactile sensing multi-finger dexterous hand is used as the second boundary value, and the actual dynamic state data and dynamic expectation data of the tactile sensing multi-finger dexterous hand, and the dynamic state error data of the tactile sensing multi-finger dexterous hand determined as the second target value, to construct a joint motion optimization model of the tactile sensing multi-finger dexterous hand.

[0079] In some embodiments of the present application, the running trajectory limit data of the tactile sensing multi-finger dexterous hand is used as the second boundary value, and the dynamic state error data of the tactile sensing multi-finger dexterous hand determined by the actual dynamic state data of the tactile sensing multi-finger dexterous hand and the dynamic expected data is used as the second target value. The joint motion optimization model of the tactile sensing multi-finger dexterous hand is constructed to ensure that the dexterous hand can accurately control its joint motion during the execution process. The running trajectory limit data defines the maximum range of motion that the dexterous hand can achieve during the operation, and the dynamic state error data reflects the difference between the actual motion state and the expected state. Through this optimization model, higher operation accuracy and efficiency can be achieved while ensuring the stability of the dexterous hand's motion.

[0080] In a specific example, the tactile sensing multi-finger dexterous hand collects the running trajectory limit data, and records its actual dynamic state data and dynamic expectation data. Based on this data, the system calculates the dynamic state error data, and uses this as the optimization target to build a joint motion optimization model. In the model, the running trajectory limit data is used as the boundary value to ensure that the dexterous hand does not exceed its physical limitations during movement. By solving the optimization model, the system can generate control instructions to enable the dexterous hand to maintain high precision and stability when performing the wire pulling task, and finally achieve efficient and stable operation of the cable.

[0081] Sub-step 2062, solving the joint motion optimization model to obtain control instructions.

[0082] In some embodiments of the present application, solving the joint motion optimization model to obtain control instructions is to achieve precise control of the tactile sensing multi-fingered dexterous hand. By solving the joint motion optimization model, the optimal motion path and control parameters of each joint of the dexterous hand can be calculated according to the optimization target and boundary value. The control instructions include specific motion instructions for each joint to achieve the expected dynamic state and operation task. Through this optimization solution, the system can ensure that the dexterous hand maintains high precision and stability when performing tasks, thereby improving operational efficiency and task success rate.

[0083] In a specific example, the tactile sensing multi-finger dexterous hand collected the running trajectory limit data and dynamic state error data, and constructed a joint motion optimization model. The system solved the optimization model and obtained the control instructions for each joint. These control instructions include the specific movement angle and force of each joint. The dexterous hand performs the cable pulling operation according to these instructions, ensuring that each joint moves according to the expected path and parameters, and finally achieves efficient and stable grasping and pulling of the cable.

[0084] Optionally, in a further embodiment, considering that the motion trajectory of the tactile sensing multi-finger dexterous hand during the threading process depends on the positions of the two fingertips in the space W formed by the world coordinate system and the space H formed by the hand coordinate system ( w P f,i and H P f,i ), the joint motion optimization model is expressed as: ; Where C represents the dynamic state error data, , , They represent the error between the current pose and the desired pose calculated by the joint position q and the robot forward kinematics in space W and space H, respectively. fw,i , W fh,i , W rfh , W hw They represent the predicted parameter matrices, q lb and q ub They represent the lower and upper limits of the value of q respectively.

[0085] Optionally, the dynamic expected data includes expected position data for characterizing the spatial position of the tactile sensing multi-fingered dexterous hand, and step 206 includes the following sub-steps: Sub-step 2063, bringing the expected position data into the dynamic balance model of the tactile sensing multi-finger dexterous hand.

[0086] Among them, the dynamic balance model is used to characterize the relationship between the applied force value and the travel trajectory of each joint of the tactile sensing multi-finger dexterous hand when grasping the thread through the tactile sensing multi-finger dexterous hand.

[0087] In some embodiments of the present application, the expected position data is brought into the dynamic balance model of the tactile sensing multi-fingered dexterous hand to ensure that the dexterous hand can achieve precise grasping control when performing the thread-straightening task. The expected position data characterizes the expected position of the dexterous hand in space. The dynamic balance model is used to characterize the relationship between the applied force value and the travel trajectory of each joint of the dexterous hand when grasping during the thread-straightening process. By bringing the expected position data into the model, the mechanical state of each joint can be accurately calculated, thereby ensuring the stability of the grasp and the accuracy of the control.

[0088] In a specific example, the expected position data of the tactile sensing multi-finger dexterous hand is set and brought into the dynamic balance model of the dexterous hand. The model optimizes the grasping control by calculating the relationship between the applied force value and the travel trajectory of each joint when performing the task. By inputting the expected position data into the model, the system can accurately calculate the mechanical state of each joint, ensuring that the dexterous hand maintains a stable and accurate grasping state when performing the thread-pulling task.

[0089] Sub-step 2064, solving the dynamic equilibrium model to determine the mechanical output value of each joint.

[0090] In some embodiments of the present application, solving the dynamic balance model to determine the mechanical output value of each joint is to accurately calculate the force required to be applied by each joint when the tactile sensing multi-fingered dexterous hand performs a grasping task. The dynamic balance model calculates the mechanical output value of each joint by analyzing the expected position data and the mechanical state in actual operation. The model includes the relationship between the range of motion of each joint, the applied force value and the travel trajectory. By solving the model, the optimal mechanical output value can be obtained, thereby ensuring the stability and control accuracy of the dexterous hand during the line drawing process.

[0091] In a specific example, a multi-finger dexterous hand with tactile sensing performs a thread-straightening task. The system inputs the expected position data and the mechanical state data in the actual operation into the dynamic equilibrium model and solves the model to determine the mechanical output value of each joint. The system calculates the force required to be applied by each joint when performing the task, ensuring that the dexterous hand maintains stable and precise control during the thread-straightening process.

[0092] Sub-step 2065, according to the mechanical output value of each joint, rotate the joint corresponding to each mechanical output value to control the grasping of the cable.

[0093] In some embodiments of the present application, the joints corresponding to each mechanical output value are rotated according to the mechanical output value of each joint to control the grasping of the cable, so as to achieve precise control of the cable by the tactile sensing multi-fingered dexterous hand during the process of pulling the cable. The mechanical output value of each joint is obtained through the previously solved dynamic equilibrium model, and these mechanical output values ​​indicate the applied force required for each joint to perform the task. According to these mechanical output values, the system controls each joint to rotate accordingly to achieve a stable grasp of the cable. The execution of this step ensures that the dexterous hand can adjust the gripping force according to the mechanical data calculated in real time during the process of pulling the cable, thereby improving the stability and accuracy of the pulling operation.

[0094] In a specific example, the multi-finger dexterous hand starts to perform the task of pulling the cable through tactile sensing. The system obtains the mechanical output value of each joint based on the previously solved dynamic equilibrium model. According to these mechanical output values, the rotation angle of each joint is adjusted to achieve a stable grasp of the cable. For example, when the mechanical output value of a joint indicates that the grasping force needs to be increased, the system will increase the rotation angle of the joint accordingly to ensure that the cable is firmly grasped.

[0095] In some further embodiments of the present application, considering that for some tactile sensing multi-finger dexterous hands, a proportional-integral-derivative (PID) position control mode is used to control the joints, when the integral gain is set to zero, the joint torque τ is proportional to the desired joint position q d The correlation with the current joint position q can be simplified into a trajectory model: τ = k p (q d -q)-k v q'; Where q' represents the first-order derivative of q with respect to time. In this scheme, the stable grasping of the tactile sensing multi-finger dexterous hand belongs to a quasi-static process, so the velocity term in the above model is almost zero, so the expected joint position q of the finger can be assigned d To control the finger tip to apply force F to the object d , that is, ; where J(q) is the Jacobian matrix of the finger.

[0096] Optionally, the multiple joints of the tactile sensing multi-fingered dexterous hand are arranged in sequence according to a preset arrangement direction, and sub-step 2065 includes the following sub-steps: Sub-step 20651, rotate the joints corresponding to each mechanical output value in sequence according to the arrangement direction.

[0097] In some embodiments of the present application, the joints corresponding to each mechanical output value are rotated in sequence according to the arrangement direction to ensure that each joint of the tactile sensing multi-fingered dexterous hand can move in a coordinated manner during operation. When executing this process, the system will rotate and adjust each joint in sequence, starting from the first joint, according to the preset arrangement direction. The rotation of each joint is based on the previously calculated mechanical output values, which indicate the force and rotation angle to be applied to each joint. This ensures the coordinated movement of all joints, thereby achieving stable grasping and threading operations.

[0098] In a specific example, a tactile multi-finger dexterous hand performs a thread-straightening task. Based on the previously solved dynamic equilibrium model, the system obtains the mechanical output value of each joint. According to the preset arrangement direction, each joint will be rotated and adjusted in turn, starting from the first joint, to ensure that each joint moves according to the calculated mechanical output value. This sequential rotation method can ensure that all joints move in a coordinated manner.

[0099] Optionally, between step 201 and step 202, the method further includes the following additional steps: Step 207: Delete the target mechanical sampling sub-distribution in the mechanical sampling distribution to update the mechanical sampling distribution.

[0100] The target mechanical sampling sub-distribution is the mechanical sampling sub-distribution of the contact area between each fingertip and other fingertips in the mechanical sampling distribution.

[0101] In some embodiments of the present application, the target mechanical sampling sub-distribution in the mechanical sampling distribution is deleted to update the mechanical sampling distribution in order to remove the force data of the contact area with other fingertips and improve the accuracy and reliability of the data. The target mechanical sampling sub-distribution refers to the mechanical sampling sub-distribution in the mechanical sampling distribution that represents the contact area between each fingertip and other fingertips. By deleting these sub-distributions, it is possible to avoid the mechanical data of the contact area interfering with subsequent analysis. The execution of this step ensures that the remaining mechanical sampling data is purer, providing a clear data basis for generating more accurate pose estimation and control instructions in subsequent steps.

[0102] In a specific example, the tactile sensors of the tactile sensing multi-finger dexterous hand collected the mechanical data of each fingertip during the grasping process. The system identified the mechanical sampling sub-distributions related to the contact areas of other fingertips in these mechanical data. To ensure the accuracy of the data, the system deleted these target mechanical sampling sub-distributions and updated the mechanical sampling distribution. Through this processing, more accurate mechanical input data can be obtained, and then reliable posture estimation data and control instructions can be generated to achieve precise operation of the cable.

[0103] like Figure 5 As shown, it shows the specific flow of data during the control process of this solution: Step S1: collecting mechanical data. The tactile sensor collects mechanical input values ​​of the fingertips and the cable during the process of stroking the cable.

[0104] Step S2: Based on the data collected by mechanics, the position and posture of the cable are estimated to generate position and posture estimation data. This step uses the mechanical input data to calculate the spatial position and posture of the cable.

[0105] Step S3: Use the pose estimation data to predict the trajectory of the tactile sensing multi-finger dexterous hand and generate dynamic expectation data. This step plans the movement path of the fingers during the threading process.

[0106] Step S4: Using the expected dynamics data to control the tactile sensing multi-finger dexterous hand: Step S4.1: Use the dynamic expectation data to control the posture. Through this data, the system adjusts the finger posture in real time to ensure the smoothness and stability of the threading operation.

[0107] Step S4.2: Use the dynamics expectation data to control the output force. This step ensures that the finger applies the appropriate force to enable it to firmly grasp and move the object.

[0108] Finally, the control data generated by the joint action of step S4.1 and step S4.2 realizes the line-smoothing control of the tactile-sensing multi-finger dexterous hand, and forms a loop from step S1 to S4.

[0109] In summary, in the embodiment of the present application, by generating accurate posture estimation data, the position and posture of the cable in the multi-fingered dexterous hand can be grasped in real time; then, by comprehensively analyzing the posture estimation data and mechanical input data, the optimal motion trajectory of the multi-fingered dexterous hand in the process of pulling the wire is determined, and then the dynamic expectation data is converted into the control instructions of the multi-fingered dexterous hand, so as to achieve precise control of the wire pulling process, accurately guide the multi-fingered dexterous hand to perform the wire pulling operation, and ensure the stability and accuracy in the wire pulling process. Therefore, based on the method of the embodiment of the present application, by acquiring and utilizing tactile sensing data in real time, precise control of the multi-fingered dexterous hand is achieved, and finally stable and efficient grasping and following of the cable is achieved. The problems of adaptability of the manipulator to the wire pulling task and low operating efficiency are solved.

[0110] refer to Figure 6 , which shows a robot autonomous thread-straightening system 30 based on tactile sensing multi-finger dexterous hand provided in an embodiment of the present application, comprising: A tactile sensing multi-finger dexterous hand 301 having tactile sensors 30111 disposed at fingertips 3011 and a controller 302; The controller 302 is used to obtain mechanical input data collected from the fingertips 3011 of the tactile sensing multi-finger dexterous hand 301, and generate posture estimation data for the cable A based on the mechanical input data, and generate dynamic expected data of the tactile sensing multi-finger dexterous hand 301 based on the posture estimation data and / or mechanical input data, and generate control instructions for the tactile sensing multi-finger dexterous hand 301 based on the dynamic expected data, so as to control the process of smoothing the wire through the tactile sensing multi-finger dexterous hand 301; the mechanical input data is generated by the grasping of the cable A by the fingertips 3011 during the process of smoothing the wire through the tactile sensing multi-finger dexterous hand 301; the posture estimation data is used to characterize the spatial posture of the cable A when the cable A is grasped in the tactile sensing multi-finger dexterous hand 301; the dynamic expected data is used to characterize the expected change in the motion trajectory of the tactile sensing multi-finger dexterous hand 301 during the grasping process.

[0111] Optionally, when the controller 302 obtains the mechanical input data collected from the fingertips of the tactile-sensing multi-fingered dexterous hand, it is specifically used to determine the mechanical sampling distribution of each fingertip 3011 by setting a plurality of tactile sensors 30111 at each fingertip 3011, and decompose the mechanical sampling distribution into two mechanical sampling sub-distributions, so that the two mechanical sampling sub-distributions constitute a two-component Gaussian mixture model, and determine the mechanical sampling sub-distribution corresponding to the smaller distribution weight parameter of the distribution weight parameters of the two mechanical sampling sub-distributions as the mechanical input data of the fingertip corresponding to the mechanical sampling sub-distribution; each mechanical sampling sub-distribution has a corresponding distribution weight parameter.

[0112] Optionally, the controller 302 is further configured to delete a target mechanical sampling sub-distribution in the mechanical sampling distribution to update the mechanical sampling distribution; the target mechanical sampling sub-distribution is a mechanical sampling sub-distribution of the contact area between each fingertip and other fingertips in the mechanical sampling distribution.

[0113] Optionally, when the controller 302 generates posture estimation data for the cable based on the mechanical input data, it is specifically used to generate contact surface simulation data between the fingertip and the cable based on the mechanical input data, and use the contact surface simulation data as the first boundary value, and use the minimum value of the total variation composed of the distance between all contact points and the cable and the radius of the cable in the contact surface simulation data as the first target value to construct a posture optimization model of the cable, and solve the posture optimization model to obtain the posture estimation data; the contact surface simulation data is used to characterize the force value at each contact point between the fingertip and the cable.

[0114] Optionally, the expected dynamic data includes expected data on the angle change of the clamping angle of the cable by the tactile sensing multi-finger dexterous hand. When the controller 302 generates the expected dynamic data of the tactile sensing multi-finger dexterous hand based on the posture estimation data and / or the mechanical input data, it is specifically used to determine the expected angle change data to decrease when the mechanical input data is characterized by the friction force between the fingertip and the cable in the tangential direction exceeding a preset tangential force threshold, and to determine the expected angle change data to increase when the mechanical input data is characterized by the area of ​​the contact area between the fingertip and the cable being smaller than a preset contact area threshold; the tangential direction is the movement direction of the tactile sensing multi-finger dexterous hand.

[0115] Optionally, the expected dynamic data includes expected data on the change in the palm direction of the tactile-sensing multi-finger dexterous hand. When the controller 302 generates the expected dynamic data for the tactile-sensing multi-finger dexterous hand based on the posture estimation data and / or the mechanical input data, the controller 302 is specifically used to determine the expected data on the change in the palm direction as a rotation in a target rotation direction when the posture estimation data characterizes that the grasping angle formed by the cable and the tactile-sensing multi-finger dexterous hand does not match a preset grasping angle threshold; the target rotation direction is the minimum rotation direction that matches the grasping angle with the grasping angle threshold.

[0116] Optionally, when the controller 302 generates control instructions for the tactile sensing multi-finger dexterous hand based on the expected dynamic data to control the process of threading through the tactile sensing multi-finger dexterous hand, it is specifically used to use the running trajectory limit data of the tactile sensing multi-finger dexterous hand as the second boundary value, and the actual dynamic state data and the expected dynamic data of the tactile sensing multi-finger dexterous hand, and the dynamic state error data of the tactile sensing multi-finger dexterous hand determined as the second target value, to construct a joint motion optimization model of the tactile sensing multi-finger dexterous hand, and solve the joint motion optimization model to obtain control instructions.

[0117] Optionally, the dynamic expected data includes expected position data for characterizing the spatial position of the tactile sensing multi-finger dexterous hand. When the controller 302 generates a control instruction for the tactile sensing multi-finger dexterous hand based on the dynamic expected data to control the process of straightening the cable by the tactile sensing multi-finger dexterous hand, the controller 302 is specifically used to bring the expected position data into the dynamic balance model of the tactile sensing multi-finger dexterous hand and solve the dynamic balance model to determine the mechanical output value of each joint, and rotate the joint corresponding to each mechanical output value according to the mechanical output value of each joint to control the grasping of the cable; the dynamic balance model is used to characterize the relationship between the applied force value and the travel trajectory of each joint of the tactile sensing multi-finger dexterous hand when grasping in the process of straightening the cable by the tactile sensing multi-finger dexterous hand.

[0118] Optionally, the multiple joints of the tactile-sensing multi-finger dexterous hand are arranged in sequence according to a preset arrangement direction. The controller 302 rotates the joint corresponding to each mechanical output value according to the mechanical output value of each joint to control the grasping of the cable. It is specifically used to rotate the joint corresponding to each mechanical output value in sequence according to the arrangement direction.

[0119] In summary, in the embodiment of the present application, by generating accurate posture estimation data, the position and posture of the cable in the multi-fingered dexterous hand can be grasped in real time; then, by comprehensively analyzing the posture estimation data and mechanical input data, the optimal motion trajectory of the multi-fingered dexterous hand in the process of pulling the wire is determined, and then the dynamic expectation data is converted into the control instructions of the multi-fingered dexterous hand, so as to achieve precise control of the wire pulling process, accurately guide the multi-fingered dexterous hand to perform the wire pulling operation, and ensure the stability and accuracy in the wire pulling process. Therefore, based on the method of the embodiment of the present application, by acquiring and utilizing tactile sensing data in real time, precise control of the multi-fingered dexterous hand is achieved, and finally stable and efficient grasping and following of the cable is achieved. The problems of adaptability of the manipulator to the wire pulling task and low operating efficiency are solved.

[0120] Reference Figure 7 , the electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0121] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0122] The memory 504 is used to store various types of data to support the operation of the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, multimedia, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0123] The power supply component 506 provides power to the various components of the electronic device 500. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0124] The multimedia component 508 includes an interface that provides an output interface between the electronic device 500 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0125] The audio component 510 is used to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the electronic device 500 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0126] The input / output I / O interface 512 provides an interface between the processing component 502 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0127] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, and the sensor assembly 514 can also detect the position change of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0128] The communication component 516 is used to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0129] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the methods provided in the embodiments of the present application.

[0130] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by a processor 520 of an electronic device 500 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0131] Figure 8 FIG. 6 is a block diagram of an electronic device 600 according to another embodiment of the present invention. For example, the electronic device 600 may be provided as a server.

[0132] Reference Figure 8 , the electronic device 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632, for storing instructions that can be executed by the processing component 622, such as an application. The application stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute instructions to perform the method provided in the embodiment of the present application.

[0133] The electronic device 600 may also include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0134] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0135] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A robot autonomous threading method based on tactile sensing multi-finger dexterous hand, characterized in that: include: Acquiring mechanical input data collected from the fingertips of a tactile sensing multi-fingered dexterous hand; The mechanical input data is generated by the gripping of the cable by the fingertips during the process of stringing the cable by the tactile sensing multi-finger dexterous hand; Generating position and posture estimation data of the cable according to the mechanical input data; The pose estimation data is used to characterize the spatial pose of the cable when the cable is grasped in the tactile sensing multi-finger dexterous hand; Generate expected dynamics data of the tactile-sensing multi-finger dexterous hand according to the posture estimation data and / or the mechanical input data; the expected dynamics data is used to characterize the expected change of the motion trajectory of the tactile-sensing multi-finger dexterous hand during the grasping process; According to the expected dynamics data, a control instruction for the tactile sensing multi-finger dexterous hand is generated to control the process of threading by the tactile sensing multi-finger dexterous hand.

2. The method according to claim 1, characterized in that The step of acquiring mechanical input data collected from the fingertips of the tactile sensing multi-finger dexterous hand comprises: Determine the mechanical sampling distribution of each fingertip by arranging a plurality of tactile sensors at each fingertip; Decomposing the mechanical sampling distribution into two mechanical sampling sub-distributions, so that the two mechanical sampling sub-distributions constitute a two-component Gaussian mixture model; each of the mechanical sampling sub-distributions has a corresponding distribution weight parameter; The mechanical sampling sub-distribution corresponding to the smaller distribution weight parameter of the two mechanical sampling sub-distributions is determined as the mechanical input data of the fingertip corresponding to the mechanical sampling sub-distribution.

3. The method according to claim 1, characterized in that Generating the estimated position and posture data of the cable according to the mechanical input data includes: Generate contact surface simulation data between the finger tip and the cable according to the mechanical input data; the contact surface simulation data is used to characterize the force value at each contact point between the finger tip and the cable; The contact surface simulation data is used as a first boundary value, and the minimum value of the total variation formed by the distance between all the contact points and the cable and the radius of the cable in the contact surface simulation data is used as a first target value to construct a posture optimization model of the cable; Solve the pose optimization model to obtain the pose estimation data.

4. The method according to claim 1, characterized in that The expected dynamics data includes expected data of angle change of the clamping angle of the cable by the tactile sensing multi-finger dexterous hand, and generating the expected dynamics data of the tactile sensing multi-finger dexterous hand according to the posture estimation data and / or the mechanical input data includes: In the case where the mechanical input data is characterized by the friction force between the finger tip and the cable in the tangential direction exceeding a preset tangential force threshold, the angle change expected data is determined to be reduced; the tangential direction is the movement direction of the tactile sensing multi-finger dexterous hand; When the mechanical input data is characterized by that the area of ​​the contact region between the finger tip and the cable is smaller than a preset contact area threshold, the angle change expected data is determined to be increased.

5. The method according to claim 1, characterized in that The expected dynamics data includes expected data on the change in the palm direction of the tactile sensing multi-finger dexterous hand. The generating the expected dynamics data of the tactile sensing multi-finger dexterous hand according to the posture estimation data and / or the mechanical input data includes: When the posture estimation data characterizes that the grasping angle formed by the cable and the tactile-sensing multi-finger dexterous hand does not match a preset grasping angle threshold, the expected data of the palm direction change is determined as a rotation in a target rotation direction; the target rotation direction is the minimum rotation direction that makes the grasping angle match the grasping angle threshold.

6. The method according to claim 1, characterized in that The step of generating a control instruction for the tactile sensing multi-finger dexterous hand according to the dynamics expected data to control the process of threading by the tactile sensing multi-finger dexterous hand comprises: Taking the running trajectory limit data of the tactile sensing multi-finger dexterous hand as the second boundary value, and taking the actual dynamic state data of the tactile sensing multi-finger dexterous hand and the dynamic expected data, and the dynamic state error data of the tactile sensing multi-finger dexterous hand determined as the second target value, constructing the joint motion optimization model of the tactile sensing multi-finger dexterous hand; Solve the joint motion optimization model to obtain the control instruction.

7. The method according to claim 1, characterized in that The dynamic expected data includes expected position data for characterizing the spatial position of the tactile sensing multi-finger dexterous hand. The generating of control instructions for the tactile sensing multi-finger dexterous hand according to the dynamic expected data to control the process of threading by the tactile sensing multi-finger dexterous hand includes: Bringing the expected position data into a dynamic balance model of the tactile sensing multi-finger dexterous hand; the dynamic balance model is used to characterize the relationship between the applied force value of each joint of the tactile sensing multi-finger dexterous hand and the travel trajectory when grasping the thread through the tactile sensing multi-finger dexterous hand; Solving the dynamic balance model to determine the mechanical output value of each joint; According to the mechanical output value of each joint, the joint corresponding to each mechanical output value is rotated to control the grasping of the cable.

8. A robot autonomous thread-winding system based on tactile sensing multi-finger dexterous hand, characterized in that: include: A tactile sensing multi-finger dexterous hand and a controller having tactile sensors disposed at the fingertips; The controller is used to obtain mechanical input data collected from the fingertips of the tactile sensing multi-finger dexterous hand, and generate posture estimation data of the cable according to the mechanical input data, and generate dynamic expected data of the tactile sensing multi-finger dexterous hand according to the posture estimation data and / or the mechanical input data, and generate control instructions for the tactile sensing multi-finger dexterous hand according to the dynamic expected data, so as to control the process of threading the cable by the tactile sensing multi-finger dexterous hand; the mechanical input data is generated by the grip of the cable by the fingertips during the process of threading the cable by the tactile sensing multi-finger dexterous hand; The posture estimation data is used to characterize the spatial posture of the cable when the cable is grasped in the tactile sensing multi-finger dexterous hand; the dynamic expectation data is used to characterize the expected change in the motion trajectory of the tactile sensing multi-finger dexterous hand during the grasping process.

9. An electronic device, characterized in that: include: A processor, a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 7.

Images