Method for force-position hybrid control of dexterous hand with tactile-visual fusion, robot and medium

Abstract

The application provides a dexterous hand force-position hybrid control method based on tactile-visual fusion, a robot and a medium, and belongs to the technical field of robot control. The method comprises the following steps: determining a desired grasping position qd and a desired grasping force Fd of a target to be operated; calculating a position difference eq based on an actual touch position qs of the dexterous hand and the desired grasping position qd, and calculating an error force eF based on an actual grasping force Fs and the desired grasping force Fd; determining a basic PID control signal up based on the position difference eq, and determining a correction signal uF based on the error force eF; acquiring visual feature data of the actual touch position qs, calculating a force feedback weight wF of the correction signal uF based on the visual feature data; determining a hybrid control signal uh based on the force feedback weight wF, the correction signal uF and the basic PID control signal up; and controlling the dexterous hand to grasp the target to be operated based on the hybrid control signal uh. The application can improve the control precision of the dexterous hand.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a tactile and visual fusion dexterous hand force-position hybrid control method, robot, and medium. Background Technology

[0002] As the core actuator for achieving precise grasping and complex operations, the dexterous hand's control performance directly determines the accuracy, safety, and reliability of task completion. It has enormous application potential in high-demand fields such as industrial precision assembly (e.g., single-handed mobile phone operation), medical surgery (e.g., laparoscopic robots), and home services. With the continuous expansion of application scenarios, dexterous hands need to handle diverse grasping objects (e.g., fragile items, irregularly shaped parts) and complex interactive environments. Especially in the field of medical surgery, it can assist surgeons or perform accurate and efficient surgical procedures independently. This places dual demands on its control strategy: "high-precision position tracking" and "highly compliant contact force adjustment."

[0003] Existing finger movement control methods for dexterous hands are mainly divided into two categories. One is position-based control, which uses the angle of the finger joint or the position of the finger tip as the control target. It can achieve precise tracking of the angle of the finger joint or the position of the finger tip, but it is prone to damage to the object or the object slipping when in contact with the object, and lacks grasping stability and flexibility. The other is force-based control, which uses the contact force as the core control target. It maintains the contact force within a preset range by adjusting the driving force in real time. It can ensure the safety and flexibility of the contact process, but it cannot perform precise motion trajectory tracking and is difficult to achieve complex and dexterous operations.

[0004] Some improved methods attempt to combine position control and force control, such as those disclosed in patents CN120516743A and CN114474073A. Although these methods improve the accuracy compared to single-dimensional control, the control process is still relatively complex. There is coupling interference between force adjustment and position adjustment, making it difficult to ensure the accuracy of both in dynamic working conditions. This makes it unsuitable for the increasingly complex environment that demands higher control efficiency and accuracy. Summary of the Invention

[0005] The purpose of this application is to provide a novel tactile-visual fusion dexterous hand force-position hybrid control method, robot, and medium to solve at least one of the above-mentioned technical problems.

[0006] To achieve the above objectives, in a first aspect, this application proposes a tactile-visual fusion-based dexterous hand force-position hybrid control method, the method comprising:

[0007] Determine the target to be operated on, and based on the target to be operated on, determine the desired grasping position qd and the desired grasping force Fd;

[0008] Control the dexterous hand to move toward the desired grasping position qd, and obtain the actual touch position qs and actual grasping force Fs of the dexterous hand in real time;

[0009] The position difference eq is calculated based on the actual touch position qs and the expected gripping position qd, and the error force eF is calculated based on the actual gripping force Fs and the expected gripping force Fd.

[0010] The basic PID control signal up is determined based on the position difference eq, and the correction signal uF is determined based on the error force eF;

[0011] Obtain visual feature data of the actual touch position qs, and calculate the force feedback weight wF of the correction signal uF based on the visual feature data;

[0012] The hybrid control signal uh is determined based on the force feedback weight wF, the correction signal uF, and the basic PID control signal up.

[0013] The dexterous hand is controlled to grasp the target to be operated based on the hybrid control signal uh.

[0014] Optionally, determining the hybrid control signal uh based on the force feedback weight wF, the correction signal uF, and the basic PID control signal up includes:

[0015] The weighted correction signal is obtained by multiplying the force feedback weight wF and the correction signal uF.

[0016] The weighted correction signal and the basic PID control signal up are superimposed to obtain the hybrid control signal uh.

[0017] Optionally, the step of calculating the force feedback weight wF based on the visual feature data to correct the signal uF includes:

[0018] The basic weight w0 of the corrected signal uF is calculated based on the visual feature data.

[0019] The corresponding weighting coefficient w1 is calculated based on the error force eF and the preset contact force threshold.

[0020] The force feedback weight wF is calculated based on the basic weight w0 and the weight coefficient w1.

[0021] Optionally, the visual feature data includes the surface deformation rate, local sliding velocity, and contact distance at the dexterous hand grasping position; the calculation of the basic weight w0 of the correction signal uF based on the visual feature data includes:

[0022] The visual state parameter value et is obtained by weighted summing of the surface deformation rate, local slip velocity and contact distance under the current visual cycle.

[0023] The weight correction value for the current visual cycle is calculated based on the quantized value of the dexterous hand's response sensitivity be, the visual feedback gain ke, and the visual state parameter value et. w;

[0024] Based on the weight value w0' from the previous visual cycle and the weight correction value from the current visual cycle. w calculates the base weight w0 for the current period.

[0025] Optionally, determining the correction signal uF based on the error force eF includes: obtaining a preset force feedback coefficient Kf, and multiplying the force feedback coefficient Kf and the error force eF to obtain the correction signal uF.

[0026] Optionally, the real-time acquisition of the actual touch position qs and actual grasping force Fs of the dexterous hand includes:

[0027] Acquire multiple raw crawling forces within a single window period;

[0028] Calculate the rate of change for each original gripping force within the window period;

[0029] The rate of change is suppressed by a preset limiting function, and each original gripping force is corrected according to the suppressed rate of change to obtain the corresponding corrected gripping force.

[0030] The actual grasping force is obtained by performing a moving average filter on the modified grasping force corresponding to each original grasping force within the window period.

[0031] Optionally, the real-time acquisition of the actual touch position qs and actual grasping force Fs of the dexterous hand includes:

[0032] Obtain the original touch position qs1 of the dexterous hand;

[0033] Obtain the position error value qs0 determined based on visual correction;

[0034] The actual touch position qs is calculated based on the position error value qs0 and the original touch position qs1.

[0035] Optionally, the method further includes: when the absolute value of the position difference eq is less than the position difference threshold and the absolute value of the error force eF is also less than the error force threshold, stopping the update of the hybrid control signal uh and continuing to capture the target to be operated according to the latest updated hybrid control signal uh; otherwise, continuing to execute the determination of the basic PID control signal up based on the position difference eq and the determination of the correction signal uF based on the error force eF.

[0036] In a second aspect, this application provides a computer-readable storage medium storing executable instructions that, when executed by a processor, cause the processor to perform the method described in any embodiment of this application.

[0037] A third aspect of this application provides a robot, comprising:

[0038] The robot itself;

[0039] robotic arm;

[0040] A vision module, configured on the robot body or robotic arm, is used to capture images of the environment;

[0041] One or more processors are used to control the robotic arm to grasp objects;

[0042] A memory for storing one or more programs that, when executed by one or more processors, cause the one or more processors to perform the method as described in any embodiment of this application.

[0043] The tactile-visual fusion dexterous hand force-position hybrid control method, robot, and medium in this application introduces a force control influence weight based on visual features. This control method can be divided into a dual closed-loop structure. In the outer loop, the vision and tactile fusion correction stage (low-frequency operation) is responsible for updating the initial values ​​of the desired force and control weights. In the inner loop, the force-position hybrid PID control stage (high-frequency operation) is responsible for calculating the position difference and force error in real time and outputting the final hybrid control signal uh. This control method can identify instability signs in advance, rather than relying solely on post-event error feedback, enabling a tighter coupling between force adjustment and position adjustment, achieving dynamic coordination of position and force. It can automatically adjust the degree of finger closure during grasping, adapting to target grasping under complex dynamic working conditions and meeting the requirements for control efficiency and control accuracy in complex environments. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation on the scope of this application.

[0045] Figure 1 This is a flowchart illustrating a tactile-visual fusion dexterity hand force-position hybrid control method in one embodiment;

[0046] Figure 2 This is a schematic diagram of the process for calculating the force feedback weight wF of the correction signal uF based on the visual feature data in one embodiment.

[0047] Figure 3 This is a schematic diagram of the process for obtaining the actual touch position qs and actual grasping force Fs of a dexterous hand in real time in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] All terms used in this application (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0050] For example, the terms "first," "second," etc., used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0051] For example, the terms "comprising" or "including" used in this application indicate the presence of features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0052] like Figure 1 As shown, this application provides a tactile-visual fusion dexterity hand force-position hybrid control method, the method comprising the following steps:

[0053] Step 110: Determine the target to be operated on, and determine the desired grasping position qd and desired grasping force Fd based on the target to be operated on.

[0054] In this embodiment, the grasping task requirements are quantified into calculable and executable control parameters through attribute analysis and kinematic transformation of the target object. The target object can be any object suitable for dexterous hand manipulation, and the grasping operation can specifically be any suitable action such as grasping, holding, lifting, or carrying. The physical properties of the target object (such as material, size, surface condition, mechanical properties, or resistance to deformation, etc.) are the core basis for determining the control parameters, serving as the input prerequisite for the control logic and directly determining the desired grasping position qd and the desired grasping force Fd. The robot can scan the target object using a vision module, identify the target object, determine its physical properties, and then determine the desired grasping position qd and the desired grasping force Fd based on the target object and its physical properties.

[0055] The desired grasping position qd is the grasping coordinate of the dexterous hand joint at the target angle. At this coordinate, a suitable / reference position (target contact point) can be grasped onto the target. This parameter can be calculated by combining the desired fingertip position and / or the target motion angle of the dexterous hand joint (represented in Cartesian space) with a kinematic model of the dexterous hand's fingers. For example, the calculated MCP joint angle corresponding to the target contact point—the side wall of the glass—is 42°. At this angle, the fingertip can fit against the glass wall without touching the rim or bottom. Based on this joint angle, the desired grasping position can be determined.

[0056] The expected grasping force Fd is the ideal contact force that the dexterous hand needs to maintain when it comes into contact with the target, and it needs to match the target's resistance to external forces.

[0057] Taking a thin-walled glass cup as an example, its physical properties can be identified as follows: cup wall thickness 1.2mm, maximum compressive stress 5MPa (corresponding to maximum bearing force 8N), smooth surface (friction coefficient μ=0.25), outer diameter 60mm, and height 150mm. Based on this, to avoid touching the rim or bottom of the cup during grasping, the upper-middle part of the glass cup's side wall (Cartesian coordinates X=180mm, Y=90mm, Z=220mm, with the center of the dexterous hand base as the origin) is selected as the fingertip target contact point. Through inverse kinematics calculation, the target angle corresponding to this contact point at the MCP joint (metacarpophalangeal joint) is qd=42°, thus obtaining the desired grasping position qd. For the desired grasping force Fd, at this target contact point, the desired grasping force Fd=5N is determined.

[0058] Specifically, the target to be manipulated in the environment can first be identified using a vision module (such as a monocular RGB camera or a structured light sensor). This, combined with a position detection module, allows for the acquisition and localization of the target's attributes, clarifying its physical characteristics (material, size, resistance to external forces) and spatial location. Subsequently, the host planning module, considering the target's physical attributes and the grasping task requirements, determines the target's coordinates (qd and Fd). For example, the target contact point of the fingertip can be defined in Cartesian space (e.g., with the dexterous hand base as the origin) (e.g., the upper-middle part of the glass sidewall, coordinates (180, 90, 220)). For instance, a monocular RGB camera can be used, employing any suitable pre-set target detection algorithm (e.g., the YOLOv5 lightweight target detection algorithm) to identify the target and output its bounding box coordinates in the image. Then, combining the object distance within the camera's calibration and the target's physical characteristics, the coordinates of the fingertip target contact point in Cartesian space can be calculated.

[0059] Based on the kinematic model of a single finger of a dexterous hand, the target angle of the corresponding MCP joint (metacarpophalangeal joint) is obtained through inverse kinematics calculation, and the desired grasping position qd is obtained.

[0060] For the expected grasping force Fd, the robot can determine the expected grasping force Fd by reserving an appropriate safety margin (such as 30%) based on the physical characteristics of the target to be operated, such as its resistance to deformation, so as to ensure that the target to be operated is neither slipped nor damaged during grasping.

[0061] Step 120: Control the dexterous hand to move towards the desired grasping position qd, and obtain the actual touch position qs and actual grasping force Fs of the dexterous hand in real time.

[0062] In this embodiment, the robot's main planning module can generate movement planning information for the dexterous hand based on the current environment and the real-time position of the dexterous hand, and send this movement planning information to the finger mechanical transmission system of the dexterous hand. The transmission system controls the dexterous hand to move towards the desired grasping position qd based on this movement planning information. The movement planning information can be updated in real time. Specifically, during the movement of the dexterous hand, dual detection channels are activated simultaneously: one channel collects qs at high frequency through a position sensing module, and the other channel collects the raw contact force signal through a force sensor. After filtering, amplitude limiting, and other preprocessing, Fs is obtained. Data synchronization is ensured during the acquisition process to avoid timing misalignment during error calculation.

[0063] The actual touch position qs is the real-time three-dimensional coordinates of the dexterous hand and / or the real-time motion angle of the dexterous hand joints (joint space feedback parameters). For example, specifically, it is the real-time three-dimensional coordinates of the fingertip joints of the dexterous hand. These position coordinates can be acquired by a position detection module and / or a vision module. For example, the real-time angle q of the finger's MCP joint can be acquired by an absolute magnetic encoder or a variable resistance potentiometer, and the actual touch position qs can be calculated based on this real-time angle q. The absolute magnetic encoder or variable resistance potentiometer can be installed on the output shaft of the MCP joint to output a digital signal of the joint angle in real time. The controller can read the data according to a preset acquisition frequency (e.g., once every 1ms) to obtain qs.

[0064] The actual grasping force Fs refers to the force value when a dexterous fingertip actually contacts a target, which can be measured by a tactile detection module. This module includes a tactile sensor (such as a 3D force sensor) integrated into the fingertip for real-time measurement of the contact force. The tactile sensor can take various forms, such as single-point, array, or multi-dimensional force acquisition. Specifically, the acquired contact force data (raw signal) can be preprocessed, for example, by performing dimensionality reduction to obtain a single force value. Due to the influence of motor vibration, the raw signal acquired by the tactile sensor fluctuates within a certain range. The actual grasping force Fs can be obtained by performing derivative limiting and moving average filtering on the acquired raw signal.

[0065] Step 130: Calculate the position difference eq based on the actual touch position qs and the expected gripping position qd, and calculate the error force eF based on the actual gripping force Fs and the expected gripping force Fd.

[0066] In this embodiment, the position difference eq reflects the degree of deviation between the actual joint position and the target position. The position difference eq has a certain directionality, and the calculation formula is eq=qd. qs, its sign reflects the deviation direction. Taking a certain direction (such as the direction the dexterous hand faces the target contact point) as the reference direction, if the calculated eq > 0, it means the fingertip has not yet reached the target position and needs to continue moving towards qd; if eq < 0, it means the fingertip has exceeded the target position and needs to be finely adjusted in the opposite direction. The position difference eq is the core basis for the PID controller to adjust the joint position. Taking the quantification of the position difference into units consistent with the joint angle (in this embodiment, "degrees") as an example, if qd = 42° and qs = 36°, then eq = 42°. 36° = 6°, which means that the fingertip needs to move 6° towards the target position.

[0067] The error force eF reflects the deviation between the actual contact force and the target force, and it also has a certain directionality. Its calculation formula is eF=Fd Fs represents the direction of the positive or negative force deviation. Taking a certain direction (such as the direction of the dexterous hand toward the target contact point) as the reference direction, if eF>0, it means that the contact force is insufficient and the thrust needs to be increased; if eF<0, it means that the contact force is too large and the thrust needs to be reduced.

[0068] Step 140: Determine the basic PID control signal up based on the position difference eq, and determine the correction signal uF based on the error force eF.

[0069] In this embodiment, the basic PID control signal up is a position control master signal generated by a PID (proportional-integral-derivative) algorithm based on the position difference eq. Its core function is to drive the dexterous hand joint to move towards qd, thereby reducing eq. The correction signal uF is a force control auxiliary signal generated based on the error force eF. Its core function is to compensate for the rigidity defects of up, dynamically adjust the contact force, and avoid force overload or underload caused by pure position control.

[0070] Two types of signals can be generated using hierarchical control logic: a basic PID control signal up and a correction signal uF. The basic PID control signal up uses a positional PID algorithm, combined with preset proportional, integral, and derivative parameters. Through the synergistic effect of the proportional term rapidly reducing deviation, the integral term eliminating static error, and the derivative term suppressing overshoot, it generates the dominant position control signal. The correction signal uF can use a proportional control algorithm (simple structure, fast response, suitable for real-time force signal compensation). The correction sensitivity is adjusted by the force feedback gain coefficient Kf to generate a force control auxiliary signal. The generation processes of both signals are executed in parallel, ensuring that the calculation is completed within the control cycle. The output signals of both types can be voltage / current signals.

[0071] For the basic PID control signal up, a positional PID algorithm is used. Taking eq obtained in step 130 as input, and combining it with preset PID parameters such as proportional coefficient Kp, integral coefficient Ki, and derivative coefficient Kd, the basic control signal up is calculated. The values ​​of Kp, Ki, and Kd can be set based on practical experience. The proportional term reflects sensitivity; the larger the value, the stronger the influence of the position difference eq on up. The integral term eliminates static error, and the derivative term suppresses overshoot. For example, Kp = 2.8, Ki = 0.12, and Kd = 0.06 can be set.

[0072] In one embodiment, a preset force feedback coefficient Kf is obtained, and the force feedback coefficient Kf and the error force eF are multiplied together to obtain a correction signal uF.

[0073] For the correction signal uF, a proportional force control algorithm can be used. Taking eF obtained in step 130 as input, it is multiplied by a preset force feedback gain coefficient Kf (used to adjust the sensitivity of force correction) to obtain the force correction signal uF. Its function is to compensate for the rigidity defects of position control and avoid excessive or insufficient contact force. That is, uF = Kf × eF. Similarly, the value of Kf can be set according to empirical values, such as Kf = 3.2.

[0074] Step 150: Obtain visual feature data of the actual touch position qs, and calculate the force feedback weight wF of the correction signal uF based on the visual feature data.

[0075] In this embodiment, visual feature data refers to the image feature parameters of the target to be operated, collected by the vision module, providing an environmental perception basis for adjusting the force feedback weight wF. It can include image feature parameters of the target itself, as well as image feature parameters of the fingertip in contact with the target. Visual feature data can include one or more of the following: surface deformation rate ΔSt / S0, local sliding velocity vs, and contact distance dc. ΔSt / S0 represents the ratio of the difference between the real-time contact area St and the initial area S0 of the target to S0, reflecting the degree of deformation of the target; the local sliding velocity vs can be calculated using optical flow to determine the relative motion velocity of the contact point, and the collected vs reflects the degree of sliding of the target; the contact distance dc can be calculated using structured light ranging to determine the vertical distance between the fingertip and the cup wall, and the collected dc is obtained.

[0076] The force feedback weight wF is a dynamic coefficient used to balance the ratio of the effects of up and uF. Its magnitude reflects the influence weight of the force correction signal. The larger wF is, the greater the contribution of uF to the hybrid control signal (the higher the proportion of force control); the smaller wF is, the stronger the dominant role of up (the higher the proportion of position control).

[0077] The system uses a vision module to sense the contact state and calculates the force feedback weight wF, enabling adaptive switching between position control and force control. When contact is stable, wF is reduced to enhance the positional accuracy of up; when signs of instability such as slippage or excessive deformation occur, wF is increased to strengthen the force compensation effect of uF and improve gripping stability.

[0078] In one implementation, the three types of visual feature data mentioned above can be extracted from the images acquired by the vision module, and the visual state parameters can be obtained by weighted summation. Then, based on the deviation between the visual state parameters and the preset threshold, the weight correction amount is calculated. Finally, the force feedback weight wF in the current cycle is updated in combination with the force feedback weight calculated in the previous cycle. The whole process is executed in parallel with the control cycle, for example, wF can be updated every 300ms.

[0079] Step 160: Determine the hybrid control signal uh based on the force feedback weight wF, the correction signal uF, and the basic PID control signal up.

[0080] The hybrid control signal uh refers to the final drive signal obtained by fusing up and wF with uF. Optionally, the correction signal and force feedback weight can be multiplied first to obtain a weighted correction signal, and then the weighted correction signal can be fused with the basic PID control signal up to obtain the hybrid control signal uh.

[0081] In one embodiment, the force feedback weight wF and the correction signal uF are multiplied to obtain a weighted correction signal; the weighted correction signal and the basic PID control signal up are superimposed to obtain the hybrid control signal uh.

[0082] In this embodiment, the correction signal uF is scaled by a force feedback weight wF to match the intensity of the correction signal with the current contact state (determined by visual feature data). That is, when the contact is stable, the influence of uF is reduced to avoid interfering with the position control accuracy; when the contact is unstable, the influence of uF is amplified to quickly compensate for force deviation, thereby achieving adaptive adjustment of the force correction effect.

[0083] The mixed control signal uh can be calculated using the following formula:

[0084] .

[0085] Step 170: Control the dexterous hand to grasp the target to be operated based on the hybrid control signal uh.

[0086] In this embodiment, a hybrid control signal uh drives the actuator (such as a motor or joint) to move. Combined with real-time feedback qs and Fs, uh is iteratively adjusted until the dexterous hand and the target to be manipulated reach a precise and stable grasping state, achieving a smooth grasping without damage or slippage. Specifically, the drive execution module receives the PWM signal corresponding to uh and drives the motor to rotate the joint in a preset direction and speed. Simultaneously, the vision module, position detection module, and tactile detection module continuously collect qs and Fs. The controller iteratively executes steps 120-160, updating uh once every preset period. When the absolute values ​​of eq and eF are both less than the corresponding preset thresholds, the grasping is considered stable, uh updates are stopped, and the current signal is maintained to preserve the grasping state; otherwise, iterative adjustments continue.

[0087] The essential function of the mixed control signal uh can be approximately regarded as the current applied to the DC motor, that is, driving the motor to rotate forward or backward. The system automatically adjusts the control mode weight by comparing the relationship between Fs and Fd. When the tactile sensor detects that the contact force gradually approaches Fd, the force correction signal uF gradually decreases, making the finger movement tend to be stable; when the contact force exceeds Fd, uF takes a negative value, driving the finger to move slightly in the reverse direction to prevent over-squeezing.

[0088] Specifically, when Fs < Fd: The finger has not reached the target contact force, indicating that the current grasp is too light or not yet in contact. At this time, the system should increase the force output, that is, increase the position control output (continue to close the finger). The force feedback correction signal uF is positive and is superimposed on the position control output, making the finger continue to approach the object. This is equivalent to increasing the thrust during the contact establishment stage until the tactile signal reaches the target level.

[0089] When Fs ≈ Fd, the contact force has reached the desired level, and at this time, it should enter a balanced state. The force feedback term approaches 0, and the controller maintains a small position adjustment to maintain a constant contact and prevent sliding or squeezing caused by external force disturbances.

[0090] When Fs > Fd, it indicates that the current finger force is too large and exceeds the expected threshold. The correction signal uF of the force feedback term is negative, and the acting direction is to reduce the drive output. The controller automatically relaxes the finger to bring the force back to the safe range.

[0091] In one embodiment, the method further includes: when the absolute value of the position difference eq is less than the position difference threshold, and the absolute value of the error force eF is also less than the error force threshold, stop updating the mixed control signal uh, and maintain grasping the target to be operated according to the latest updated mixed control signal uh; otherwise, continue to execute determining the basic PID control signal up based on the position difference eq and determining the correction signal uF based on the error force eF.

[0092] The tactile and visual fusion-based dexterous hand force-position hybrid control method in this application determines the basic PID control signal up based on the position difference eq, determines the correction signal uF based on the error force eF, and then calculates the force feedback weight wF of the correction signal uF based on visual feature data. Based on the force feedback weight wF, the correction signal uF, and the basic PID control signal up, a hybrid control signal uh is determined. Compared to traditional position control + force control, this application introduces a visual feature-based adjustment of the force control influence weight. The proposed control method is a hierarchical dual-closed-loop structure. In the outer loop, the visual... The haptic feedback correction stage (low-frequency operation) is responsible for updating the initial values ​​of the desired force and control weights. In the inner loop, the force-position hybrid PID control stage (high-frequency operation) is responsible for real-time calculation of the position difference and force error, outputting the final hybrid control signal uh. This control method can identify instability signs in advance, rather than relying solely on post-event error feedback, allowing for tighter coupling of force and position adjustments. This enables dynamic coordination of position and force, automatically adjusting the degree of finger closure during grasping, adapting to complex dynamic conditions, and meeting the requirements for control efficiency and accuracy in complex environments. Furthermore, this application improves grasping compliance and safety, avoiding crush damage to fragile objects. The control structure is simple, computationally low, and can be directly implemented in embedded systems. Additionally, this application achieves force feedback hybrid control without complex modeling, making it highly feasible in engineering.

[0093] In one embodiment, such as Figure 2 As shown, the force feedback weight wF of the correction signal uF is calculated based on the visual feature data, including:

[0094] Step 210: Calculate the basic weight w0 of the corrected signal uF based on the visual feature data.

[0095] In this embodiment, the base weight refers to the weight benchmark value determined by visual feature data, and its core function is to quantify the stability of the contact state. The more stable the contact (e.g., no slippage, slight deformation), the smaller w0 (the proportion of weakening force control); the more unstable the contact (e.g., slippage tendency, excessive deformation), the larger w0 (the proportion of strengthening force control). The base weight can be updated in real time based on the visual feature data, and further, the base weight used in the current (current visual cycle) can be updated based on the base weight calculated in the previous (previous visual cycle) (or the weight value under the previous visual cycle hereinafter).

[0096] The basic weight w0 can be calculated based on the surface deformation rate, local slip velocity, and contact distance. For example, the visual state parameter value et can be calculated based on the surface deformation rate, local slip velocity, and contact distance, and then a corresponding coefficient is set for the visual state parameter value et. The product of the coefficient can be used as the basic weight w0, or on the basis of multiplying by the coefficient, the weight value in the previous visual cycle (i.e., the basic weight calculated in the previous visual cycle) is further added to obtain the basic weight in the current visual cycle.

[0097] Step 220: Calculate the corresponding weight coefficient w1 according to the error force eF and a preset contact force threshold.

[0098] The weight coefficient w1 refers to an adjustment coefficient jointly determined by the error force eF and the contact force threshold. Its core function is to quantify the compensation requirement for force deviation. The contact force threshold is used to smoothly transition the control mode.

[0099] In one embodiment, the contact force threshold may include a contact force upper limit value F1 and a contact force lower limit value F2. Among them, the magnitudes of F1 and F2 can be set according to the target to be operated and the actual situation of the robot, and F1 < F2. The weight coefficient w1 can be calculated according to the formula w1 = (eF - F1) / (F2 - F1).

[0100] Step 230: Calculate the force feedback weight wF according to the basic weight w0 and the weight coefficient w1.

[0101] Optionally, the basic weight w0 and the weight coefficient w1 can be multiplied, and the product is used as the force feedback weight wF. In one embodiment, when eF < F1, it can be considered that the dexterous hand has not yet contacted the target to be operated, and wF = 0 can be directly set; when F1 ≤ eF < F2, this is the contact establishment stage, and as the value of eF increases, the corresponding w1 also increases, and wF = w0 × w1; when eF > F2, this is the stable contact stage, and wF = w0 can be set.

[0102] By periodically updating wF, a hierarchical control mechanism of low-frequency visual perception - high-frequency tactile execution can be achieved.

[0103] In one embodiment, step 210 includes: performing weighted summation on the surface deformation rate, local slip velocity, and contact distance in the current visual cycle to obtain the visual state parameter value et; calculating the weight correction value w in the current visual cycle according to the quantization value be of the response sensitivity of the dexterous hand, the visual feedback gain ke, and the visual state parameter value et; calculating the basic weight w0 in the current cycle based on the weight value w0' in the previous visual cycle and the weight correction value w in the current visual cycle.

[0104] The visual state parameter et refers to a unified quantization value obtained by weighted summation of the normalized surface deformation rate, local slip velocity, and contact distance in a visual cycle. It is the weighted fusion result of visual feature data and is used to uniformly quantify contact stability. Its core function is to convert multi-dimensional visual features into a single index for facilitating subsequent correction value calculation. When its value is small, it indicates stable contact and no obvious slip; the larger the value of et, the more unstable the contact state, with a tendency of sliding or deformation, and the force control ratio should be increased.

[0105] For example, the normalized surface deformation rate ΔSt / S0 = 0.03, the local slip velocity vs = 0.07, and the contact distance dc = 0.28. The sum of the weighting coefficients of the three can be 1. For example, they are a1 = 0.4, a2 = 0.3, and a3 = 0.3 respectively. Then et = a1×(ΔSt / S0)+a2×vs+a3×dc = 0.4×0.03 + 0.3×0.07 + 0.3×0.28 = 0.117.

[0106] The weight correction value Δw refers to the incremental value used to iteratively update the base weight. Its core function is to dynamically adjust w0 according to the deviation between the current contact state and the stable state. The visual feedback gain ke refers to the coefficient that controls the amplitude of the weight correction value. The larger ke is, the larger Δw is, and the more sensitive the adjustment of w0 is; the smaller ke is, the smoother the adjustment is.

[0107] The response sensitivity quantization value be refers to the coefficient that amplifies the deviation between et and the visual stability threshold e0. The larger be is, the larger Δw is under the same deviation, and the more sensitive the response to changes in the contact state. The visual stability threshold e0 refers to the reference value for determining the stable contact state. If et > e0, it indicates that the grasping state is unstable, and w0 should be increased to make the hybrid control tend to force control; if et < e0, it indicates stable contact, and w0 should be decreased to maintain the dominance of position control.

[0108] In one embodiment, the weight correction value Δw can be calculated according to the formula Δw = ke×tanh(be×(et - e0)). According to the formula w0 = w0’ + Δw, the base weight w0 in the current visual cycle is calculated. When entering the next visual cycle, the currently used w0 is used as w0’, and a new w0 is calculated again according to the above process.

[0109] In this embodiment, w0 is periodically adjusted to gradually affect the mixing ratio in the force feedback weight function.

[0110] In one embodiment, the real-time acquisition of the actual touch position qs and actual gripping force Fs of the dexterous hand includes: acquiring multiple raw gripping forces within a window period; calculating the corresponding rate of change for each raw gripping force within the window period; suppressing the rate of change through a preset limiting function; correcting each raw gripping force according to the suppressed rate of change to obtain a corresponding corrected gripping force; and performing a moving average filter based on the corrected gripping force corresponding to each raw gripping force within the window period to obtain the actual gripping force Fs.

[0111] In this embodiment, the raw grasping force refers to the grasping force data collected by the tactile detection module before it has undergone rate of change suppression and filtering. Its core function is to provide the raw data source for force measurement. The raw grasping force may contain errors, which mainly originate from sensor noise, electromagnetic interference, mechanical vibration, etc. Based on this, the raw grasping force can also be subjected to moving average filtering and derivative limiting processing to make the actual grasping force obtained after processing more accurate.

[0112] The amount of data within the window period can be any suitable quantity, such as N. That is, the original crawling power within the window period can be the N most recently crawled original crawling powers. Let Fr(t) represent the original crawling power at the current time t. Then, at time t, the amount of data within the corresponding window period can be represented as {Fr(t-N+1), Fr(t-N+2), Fr(t-N+3)……Fr(t)}.

[0113] Let Fr(t) be the rate of change Fr(t) = (Fr(t) - Fr(t-1)) / t, t represents the acquisition time interval between two adjacent raw gripping forces.

[0114] The limiting function of Fr(t) can be expressed as: Fr(t)'=limit( Fr(t), Fr1, Fr2), this function is expressed as, when Fr(t)< When Fr1, Fr(t)'= Fr1; when Fr(t)> When Fr2, Fr(t)'= Fr2; when Fr1< Fr(t)< When Fr2, Fr(t)'= Fr(t). Wherein... Fr1 and Fr2 can be set according to the actual situation. By setting the amplitude limiting function, the oscillation of the control system caused by abnormal sudden changes in the tactile signal can be effectively avoided.

[0115] After calculating the rate of change of each original gripping force within the window period based on the limiting function, the corrected gripping force of that original gripping force can be calculated accordingly. For example, the corrected gripping force of Fr(t) is Fc(t) = Fc(t-1) + Fr(t)'.

[0116] By applying a moving average filter to the original grasping force within the window period at the current time t, the actual grasping force Fs(t) at the current time t can be obtained. .

[0117] In one embodiment, such as Figure 3 As shown, the actual touch position qs and actual grasping force Fs of the dexterous hand are acquired in real time, including:

[0118] Step 310: Obtain the original touch position qs1 of the dexterous hand.

[0119] Step 320: Obtain the position error value qs0 determined based on visual correction.

[0120] Step 330: Calculate the actual touch position qs based on the position error value qs0 and the original touch position qs1.

[0121] Alternatively, qs = qs1 + qs0.

[0122] In this embodiment, the robot can pre-construct a simulated task environment specifically for calculating the position error value qs. This simulated task environment includes the target object to be grasped, which contains reference points suitable for dexterous hand perception. A reference point is a specific point selected in the initial task scenario model for tactile calibration, possessing distinct tactile characteristics. These characteristics ensure that when the dexterous hand contacts the reference point, the point and area of ​​force application do not change with the direction of the dexterous hand's touch, facilitating accurate identification by the dexterous hand through tactile sensors. For example, the reference point may be a corner of the target object or a clearly convex point.

[0123] In this simulation environment, the reference coordinates of a reference point are pre-calibrated. The dexterous hand can be controlled to move towards the reference point through a combination of a vision module, a position detection module, and a tactile detection module. The movement direction of the dexterous hand is adjusted based on the tactile features perceived by the force-bearing points and the position of the dexterous hand identified by the vision and position detection modules, continuously bringing it closer to the reference point. When the tactile features perceived by the tactile detection module remain unchanged during the movement, it indicates that the current force-bearing point of the dexterous hand has made contact with the reference point, and the actual coordinates of the dexterous hand at this moment are recorded. The actual coordinates represent the real-time three-dimensional coordinates of the force-bearing point of the dexterous hand. Calculating the difference between these actual coordinates and the reference coordinates yields the position error value qs0.

[0124] This embodiment combines visual and tactile perception, enabling calibration based on tactile feedback, thereby improving the robot's positioning accuracy.

[0125] In one embodiment, a computer-readable storage medium is provided having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the above method embodiments.

[0126] In one embodiment, a robot is provided, comprising: a robot body; a robotic arm; a vision module configured on the robot body or robotic arm for capturing environmental images; one or more processors for controlling the robotic arm to grasp objects; and a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to perform the method described in any embodiment of this application.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0128] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, any of the embodiments or implementations claimed above can be used in any combination. The information disclosed in this background section is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for dexterous hand force-position hybrid control integrating tactile and visual perception, characterized in that, The method includes: Determine the target to be operated on, and based on the target to be operated on, determine the desired grasping position qd and the desired grasping force Fd; Control the dexterous hand to move toward the desired grasping position qd, and obtain the actual touch position qs and actual grasping force Fs of the dexterous hand in real time; The position difference eq is calculated based on the actual touch position qs and the expected gripping position qd, and the error force eF is calculated based on the actual gripping force Fs and the expected gripping force Fd. The basic PID control signal up is determined based on the position difference eq, and the correction signal uF is determined based on the error force eF; The visual feature data of the actual touch position qs is obtained, and the surface deformation rate, local sliding velocity, and contact distance under the current visual cycle are weighted and summed to obtain the visual state parameter value et. The weight correction value under the current visual cycle is calculated based on the quantized value be of the dexterous hand's response sensitivity, the visual feedback gain ke, and the visual state parameter value et. w is based on the weight value w0' from the previous visual cycle and the weight correction value from the current visual cycle. w calculates the basic weight w0 under the current cycle. The visual feature data includes the surface deformation rate, local sliding speed and contact distance of the dexterous hand grasping position. The surface deformation rate is used to reflect the degree of deformation of the target to be operated. The corresponding weight coefficient w1 is calculated based on the error force eF and the preset contact force threshold, and the force feedback weight wF is calculated based on the basic weight w0 and the weight coefficient w1. The hybrid control signal uh is determined based on the force feedback weight wF, the correction signal uF, and the basic PID control signal up. The basic PID control signal up is the position control dominant signal generated by the PID algorithm based on the position difference eq. The correction signal uF is the force control auxiliary signal generated based on the error force eF. The force feedback weight wF is a dynamic coefficient used to balance the ratio of the action of up and uF. The dexterous hand is controlled to grasp the target to be operated based on the hybrid control signal uh.

2. The method according to claim 1, characterized in that, The determination of the hybrid control signal uh based on the force feedback weight wF, the correction signal uF, and the basic PID control signal up includes: The weighted correction signal is obtained by multiplying the force feedback weight wF and the correction signal uF. The weighted correction signal and the basic PID control signal up are superimposed to obtain the hybrid control signal uh.

3. The method according to claim 1, characterized in that, The determination of the correction signal uF based on the error force eF includes: Obtain the preset force feedback coefficient Kf, and multiply the force feedback coefficient Kf and the error force eF to obtain the correction signal uF.

4. The method according to claim 1, characterized in that, The real-time acquisition of the actual touch position qs and actual grasping force Fs of the dexterous hand includes: Acquire multiple raw crawling forces within a single window period; Calculate the rate of change for each original gripping force within the window period; The rate of change is suppressed by a preset limiting function, and each original gripping force is corrected according to the suppressed rate of change to obtain the corresponding corrected gripping force. The actual grasping force is obtained by performing a moving average filter on the modified grasping force corresponding to each original grasping force within the window period.

5. The method according to any one of claims 1 to 4, characterized in that, The real-time acquisition of the actual touch position qs and actual grasping force Fs of the dexterous hand includes: Obtain the original touch position qs1 of the dexterous hand; Obtain the position error value qs0 determined based on visual correction; The actual touch position qs is calculated based on the position error value qs0 and the original touch position qs1.

6. The method according to any one of claims 1 to 4, characterized in that, The method further includes: When the absolute value of the position difference eq is less than the position difference threshold and the absolute value of the error force eF is also less than the error force threshold, the update of the hybrid control signal uh is stopped, and the target to be operated is captured according to the latest updated hybrid control signal uh. Otherwise, the basic PID control signal up is determined based on the position difference eq, and the correction signal uF is determined based on the error force eF.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores executable instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 6.

8. A robot, characterized in that, include: The robot itself; robotic arm; A vision module, configured on the robot body or robotic arm, is used to capture images of the environment; One or more processors are used to control the robotic arm to grasp objects; A memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to perform the method as described in any one of claims 1 to 6.

Images