A robotic dual-hand haptically guided shaft-hole assembly control method and apparatus

Abstract

The application provides a robot double-hand tactile guidance shaft hole assembly control method and device. The method provided by the application comprises: integrating a motor direct drive roller fingertip module with a robot dexterous hand; the robot grasps a hole part through the modified dexterous hand, collects initial pose information of the hand object, performs an intra-hand reorientation operation by using a hierarchical control architecture, and adjusts the hole part to a shaft hole assembly adaptive pose matched with an insertion direction of a shaft part; the robot controls a calibrated tactile gripper to grasp the shaft part, and collects force / torque sensing data in a shaft hole contact process in real time; a factor graph optimization model is used to jointly estimate a contact state and a six-degree-of-freedom relative pose of the shaft relative to the hole in the shaft hole assembly process in real time, and outputs a current optimal contact state and an optimal relative pose; adaptive adjustment of assembly control parameters is performed based on the optimal contact state, a force / torque control instruction adapted to the current contact state is output, and the robot is driven to perform an assembly action until the shaft hole assembly is completed.

Description

Technical Field

[0001] This application relates to the field of robot shaft hole assembly technology, and in particular to a robot hand-held tactile guidance shaft hole assembly control method and device. Background Technology

[0002] Shaft and hole assembly is a fundamental and critical process in the industrial field, directly determining the quality of the final product. Traditional manual assembly is labor-intensive, has low precision, poor consistency, and lacks safety in special working environments. Humanoid dexterous hands, with multiple degrees of freedom and high flexibility, can simulate human hand operations, achieving precise part grasping and posture adjustment, becoming an important way to improve the automation level and quality of shaft and hole assembly.

[0003] Currently, there are two main approaches to robotic shaft-hole assembly in the industrial field: one is a rigid assembly method based on traditional industrial robotic arms. This method uses a preset fixed motion trajectory to drive the robotic arm to move the shaft part towards the hole part and complete the insertion. During the assembly process, vision sensors are used for coarse positioning of the shaft and hole, and in some scenarios, simple force sensors are used for collision detection. The other approach is a flexible assembly method based on humanoid dexterous hands. Existing technologies mostly use dexterous hands with conventional fingertip structures. Visual guidance is used to complete part grasping and posture adjustment, and basic force control strategies are combined to achieve shaft-hole docking and insertion. Some solutions introduce simple contact state judgment logic to help optimize the assembly process. Although existing robotic shaft-hole assembly methods have replaced manual operation to a certain extent, there are still many problems and shortcomings that make it difficult to meet the requirements of high-precision and high-robustness assembly: traditional rigid assembly is prone to hard collisions and jamming; conventional dexterous hands lack active rolling degrees of freedom, have low operability, and are prone to grasping instability; contact state recognition accuracy is low, control parameters are fixed, and they cannot adapt to dynamic contact conditions; the hand-mounted redirection and assembly are not well connected, resulting in poor assembly efficiency and robustness.

[0004] Therefore, there is an urgent need for a method to achieve high precision, high robustness, and high efficiency in shaft-hole assembly. Summary of the Invention

[0005] In view of this, this application provides a robot two-hand tactile guided shaft hole assembly control method and device to achieve high precision, high robustness and high efficiency in shaft hole assembly.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] The first aspect of this application provides a method for assembling and controlling tactile guided shaft holes for robotic hands, the method comprising:

[0008] The open-source Leaphand-v1 robot integrates a direct-drive motor, roller, and fingertip module to build the hardware foundation for the collaborative operation of the dexterous hand and tactile gripper. The sensors and actuators are initialized and calibrated.

[0009] The robot uses its modified dexterous hand to grasp the hole part, uses a visual sensing system to collect the initial pose information of the hand object, and uses a vision-guided hierarchical control architecture to perform the hand redirection operation, adjusting the hole part to a shaft-hole assembly adaptation pose that matches the insertion direction of the shaft part.

[0010] The robot-controlled calibrated tactile gripper grasps the shaft part, drives the dexterous hand and tactile gripper to complete the two-hand coordinated docking and enter the shaft hole assembly stage, and collects force / torque sensing data in real time during the shaft hole contact process;

[0011] Based on real-time force / torque sensing data, a factor graph optimization model integrating multiple factors is used to jointly estimate the contact state and the six-degree-of-freedom relative pose of the shaft to the hole during the shaft-hole assembly process in real time, and solve and output the current optimal contact state and optimal relative pose.

[0012] Based on the optimal contact state and optimal relative pose, the assembly control parameters are dynamically adjusted using a contact state-driven adaptive impedance control strategy. Force / torque control commands adapted to the current contact state are output to drive the robot to perform assembly actions in a coordinated manner with both hands until the shaft and hole assembly is completed.

[0013] The second aspect of this application provides a robot two-hand tactile guide shaft hole assembly control device, the device including an integration module, an adjustment module, a data acquisition module and an output module;

[0014] The integrated module is used to build the hardware foundation for the collaborative operation of the dexterous hand and the tactile gripper by integrating the motor direct drive roller fingertip module of the open-source Leaphand-v1 robot, and to initialize and calibrate each sensor and actuator.

[0015] The adjustment module is used by the robot to grasp the hole part with the modified dexterous hand, collect the initial pose information of the hand object using the visual sensing system, and perform the hand repositioning operation with the visual guidance hierarchical control architecture to adjust the hole part to the shaft hole assembly adaptation pose that matches the insertion direction of the shaft part.

[0016] The acquisition module is used to control the calibrated tactile gripper of the robot to grasp the shaft part, drive the dexterous hand and the tactile gripper to complete the two-hand coordinated docking and enter the shaft hole assembly stage, and collect force / torque sensing data in real time during the shaft hole contact process.

[0017] The output module is used to perform joint real-time estimation of the contact state and the six-degree-of-freedom relative pose of the shaft to the hole during the shaft-hole assembly process based on the real-time collected force / torque sensing data and a factor graph optimization model that integrates multiple factors, and solves and outputs the current optimal contact state and optimal relative pose.

[0018] The output module is also used to dynamically adjust the assembly control parameters according to the optimal contact state and the optimal relative pose using a contact state-driven adaptive impedance control strategy, and output force / torque control commands adapted to the current contact state to drive the robot's two hands to perform assembly actions in a coordinated manner until the shaft and hole assembly is completed.

[0019] The robotic two-handed tactile-guided shaft hole assembly control method and device provided in this application first integrates a motor-driven roller fingertip module into the dexterous hand and completes hardware calibration, which expands the degree of freedom of finger movement and improves operability. At the same time, it transforms the traditional tangential friction constraint into a rolling friction constraint to reduce the risk of slippage, providing hardware support for stable and precise in-hand operation. Second, it adopts a vision-guided hierarchical control architecture to perform in-hand repositioning of hole parts, which can adjust the hole parts to the assembly adaptation pose in advance, eliminate the interference of initial pose error on subsequent docking, and reduce the alignment difficulty and jamming probability in the shaft hole assembly stage. Third, it is based on force / force The moment sensing data employs a factor graph optimization model that integrates multiple factors, enabling joint real-time estimation of contact state and six-degree-of-freedom relative pose. This balances continuous pose accuracy with the rationality of discrete contact states, providing reliable and comprehensive state input for subsequent control. Finally, based on the optimal state, a contact state-driven adaptive impedance control strategy is adopted, which can dynamically match assembly parameters and output appropriate force / torque commands to achieve compliant assembly at different contact stages, effectively avoiding impact and jamming. The overall solution forms a complete shaft-hole assembly control scheme from hardware to software, and from preprocessing to closed-loop control, significantly improving assembly accuracy, stability, and success rate. Attached Figure Description

[0020] Figure 1 A flowchart of the robot's two-handed tactile guide shaft hole assembly control method provided in Embodiment 1 of this application;

[0021] Figure 2 This is a schematic diagram of the robot's two-handed tactile guide shaft hole assembly control device provided in Embodiment 2 of this application. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0023] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0025] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0026] Figure 1 This is a flowchart illustrating the robot's two-handed tactile guide shaft hole assembly control method provided in Embodiment 1 of this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0027] S101. The open-source Leaphand-v1 robot integrates a direct-drive motor-driven roller fingertip module to build the hardware foundation for the collaborative operation of the dexterous hand and tactile gripper, and initializes and calibrates each sensor and actuator.

[0028] The motor-driven roller fingertip module adopts a modular structure design and is embedded into the fingertips of the open-source Leaphand-v1 dexterous hand, replacing the original 3D-printed fingertip structure without interfering with the original gripping function of the dexterous hand. The motor-driven roller fingertip module includes left and right support structures, bearings, encoders, drive servos and servo brackets. The main structure of the module is made of PETG material through 3D printing, and the internal gears are CNC machined. The drive servos are connected to the rollers through plug-in buckles. The circuit part integrates a multi-layer flexible PCB and is integrated with the tactile array. The total width of the roller assembly of the motor-driven roller fingertip module does not exceed 40mm, and the roller diameter is set to 25mm, which is adapted to the overall structural size of the dexterous hand. The alignment error during the module assembly process is less than 0.2mm, and the total degree of freedom of the modified dexterous hand is increased to 20 degrees of freedom.

[0029] Specifically, the Leaphand-v1 dexterous hand is an open-source five-finger humanoid robot dexterous hand, which is the basic hardware platform of this application. It is a multi-degree-of-freedom humanoid manipulator with multi-joint drive and multi-finger collaborative gripping capabilities, enabling it to grasp conventional objects. This application uses this dexterous hand as a modification object, adding roller fingertip modules to form the Leaphand-RT dexterous hand, which serves as the core actuator for repositioning the hand and gripping hole parts. The tactile gripper is the end effector used to grip shaft parts in this application, integrating force / torque sensors or tactile arrays, possessing force sensing and compliant control capabilities. Its main function is to stably grip shaft parts and collect contact force and contact torque information between the shaft and hole in real time during shaft-hole assembly. The two have a clear division of labor and work together to complete shaft-hole assembly: the modified Leaphand-RT dexterous hand (right hand) is responsible for gripping hole parts and completing hand posture repositioning, while the tactile gripper (left hand) is responsible for gripping shaft parts and providing the tactile / force sensing required for assembly. The dexterous hand provides precise and stable positioning of the hole parts, while the tactile gripper provides positioning and contact sensing for the shaft parts. Under the unified scheduling of the control system, the two work together to complete the assembly of the shaft and hole.

[0030] Furthermore, the motor-driven roller fingertip module is a modular end effector designed in this application to improve the operational performance of the dexterous hand, replacing the original rigid fingertip of the Leaphand-v1. It consists of a drive servo motor, roller, bearing, encoder, support structure, and flexible circuitry. The servo motor directly drives the roller rotation, adding an independent degree of freedom for rolling to the dexterous hand, improving finger operability and contact stability. Its modular embedded design does not damage the original dexterous hand structure or interfere with the original gripping function. The sensors include vision sensors (such as industrial cameras, used for hand and object pose detection during the hand redirection phase), tactile / force sensors (integrated in the tactile gripper and roller fingertip module, used to collect shaft hole contact force / torque), joint position sensors / encoders (distributed within the dexterous hand joints and roller fingertip module, used to acquire joint angles and roller rotation angles), and actuator-related feedback sensors (including current / position feedback sensors for the servo motor and motor, used for closed-loop control of the actuator).

[0031] In practical implementation, the original 3D-printed rigid fingertip structure of the five fingers of the open-source Leaphand-v1 dexterous hand was disassembled. A motor-driven roller fingertip module was embedded into the fingertips of the dexterous hand via a modular interface. The module drives the servo motor and completes the electrical connection with the original control circuit of the dexterous hand. The encoder signal line is connected to the main control unit of the dexterous hand, ensuring that the rotational degree of freedom of the roller and the original joint degree of freedom of the dexterous hand are independently controllable and can exchange data. The modified Leaphand-RT dexterous hand and tactile gripper were respectively installed at the ends of the robot's two arms. A communication link and motion coordination protocol were established between the two through the robot control system. The relative base coordinate system of the dexterous hand and the tactile gripper was calibrated, the initial pose relationship of the two-handed collaborative operation was determined, and the hardware physical foundation for the two-handed collaborative operation was completed. Furthermore, industrial cameras and other visual sensing devices are activated to acquire images from the calibration board to complete intrinsic and extrinsic parameter calibration, establish a reference coordinate system for hand-object pose recognition, and verify pose detection accuracy. Zero-point calibration is performed on the force / torque sensors integrated into the tactile gripper and roller fingertip modules, and a standard force value is applied to complete range calibration. Data acquisition frequency and noise filtering parameters are configured. The dexterous hand's joints and the rollers of the roller fingertip modules are driven to complete their full stroke, and the deviation between the encoder feedback value and the actual position is recorded to complete zero-point compensation and linearity calibration. Reference value calibration is performed on the current / position feedback sensors of the dexterous hand's joint motors and roller drive servos to verify the consistency between the feedback signal and the actual execution state. Finally, the joints and rollers of the Leaphand-RT dexterous hand were driven to complete the idle stroke, and the movement limit positions of the joints / rollers were recorded. The soft limit parameters of the movement were set. The no-load and loaded motion tests of the gripping drive mechanism of the tactile gripper were carried out to calibrate the correspondence between the gripping force and the drive current, and the gripping force control reference value was set. The motion response delay of the dexterous hand and the tactile gripper was verified, and the frequency of control command issuance was adjusted to ensure that the execution accuracy and response speed of the actuator motion command meet the requirements. The zero-point reset operation of all actuators was completed to return the dexterous hand and the tactile gripper to the initial reference posture of the two-handed coordinated operation.

[0032] Optionally, after establishing the hardware foundation for the collaborative operation of the dexterous hand and tactile gripper using the integrated motor direct-drive roller fingertip module of the open-source Leaphand-v1 dexterous hand, the method further includes: establishing a kinematic model for a single finger of the dexterous hand that includes the rotational degree of freedom of the roller; incorporating the roller rotation angle, roller radius, and rolling tangential unit vector into the modeling system of the fingertip pose and Jacobian matrix; verifying through matrix expansion analysis that the newly added rolling degree of freedom column vector is independent of the original matrix column space; establishing a non-holonomic rolling contact model between the roller fingertip and the object; comparing the original friction constraint form of the fingertip; verifying that the roller fingertip transforms the tangential friction constraint into a rolling friction torque constraint, and that the rolling friction coefficient is less than the static friction coefficient; and simultaneously limiting the roller motion form through the no-slip rolling condition.

[0033] Specifically, a kinematic model of a single finger of a dexterous hand is established, incorporating the rotational degree of freedom of a roller into the model. The specific definitions of the roller rotation angle, roller radius, and rolling tangential unit vector are clarified, and these three are integrated into the calculation logic of the fingertip pose and the Jacobian matrix modeling system, completing the extended Jacobian matrix construction that includes the roller rotational degree of freedom. Column space analysis is performed on the extended Jacobian matrix to extract the column vectors corresponding to the newly added roller rolling degree of freedom. Vector correlation analysis verifies that these column vectors are independent of the column vectors corresponding to the original joint degrees of freedom in the Jacobian matrix, i.e., independent of the original matrix column space. A nonholonomic rolling contact model between the roller fingertip and an object is established, clarifying the force relationship at the contact point between the roller and the object, defining the calculation logic of the rolling friction torque, and determining the constraints of the model. A contact friction model between the original rigid fingertip and an object is constructed, clarifying the tangential friction constraint form of the original fingertip, and comparing it with the contact model of the roller fingertip. Standard parameters for rolling friction coefficient and static friction coefficient are introduced. Through parameter comparison, it is verified that the rolling friction coefficient is less than (far less than) the static friction coefficient. Simultaneously, the specific implementation logic of transforming the traditional tangential friction constraint of the roller fingertip into a rolling friction torque constraint is clarified. A condition for slip-free rolling is set and incorporated into the contact model between the roller fingertip and the object, limiting the roller's motion and ensuring no relative slippage at the contact point between the roller and the object during movement.

[0034] For example, in one embodiment, the original fingertip scheme (taking LeapHand used in this application as an example) has a single fingertip pose. Joint variables Jacobian matrix The model is as follows:

[0035] ;

[0036] in, Add angular velocity to the linear velocity of the fingertip; The Jacobian matrix of the original single finger; This is the original joint angular velocity vector of a single finger, containing... , , , Four joint angular velocity components; This is the submatrix in the Jacobian matrix that is related to the terminal linear velocity; This is the submatrix in the Jacobian matrix that is related to the terminal angular velocity; This is the joint angle vector of the original single finger.

[0037] Introducing the roller fingertip adds one degree of freedom to the system, using variables... Indicates the scroll wheel rotation angle. Expanded single fingertip pose. Joint variables Jacobian matrix The model is as follows:

[0038] ;

[0039] in, To introduce the roller fingertip, the linear velocity plus angular velocity of the fingertip of a single finger; This is the expanded Jacobian matrix; For the expanded joint variables; Add the roller angular velocity vector to the extended joint, including ; The rotation angle of the roller; Where is the radius of the roller; It is a unit vector along the rolling tangent; , Submatrices representing the linear and angular velocities at the end of the rollers, respectively, for their rotational degrees of freedom; This is the rotated part of the Jacobian matrix; This represents the linear velocity component of the Jacobian matrix.

[0040] Due to the newly added column vector Independent of the column space of the original matrix, according to the Gram-Schmidt orthogonalization principle, the new manipulability index satisfies:

[0041] ;

[0042] in, To measure the operability of a single finger after the introduction of the roller fingertip; This is the original single-finger operability index; To extend the Jacobian matrix.

[0043] The above derivation shows that after introducing the fingertip roller degree of freedom, the operability index of a single finger will be improved, the reach of the finger can be expanded, the fingertip end posture adjustment ability can be optimized, thereby optimizing the grasping and manipulation effect.

[0044] Furthermore, a key characteristic of roller fingertips is their ability to roll on object surfaces, thereby reducing the frictional limitations of traditional fingertips. To quantitatively analyze this characteristic, a nonholonomic rolling contact model of roller fingertips is established.

[0045] For traditional, dexterous fingertips, the tangential frictional force at the point of contact is constrained by the friction cone:

[0046] ;

[0047] in, This is the vector of the tangential frictional force at the point of contact. It is a collection of friction cones; The normal force at the point of contact; It is the static friction coefficient.

[0048] When the fingertip slips during operation, it can easily exceed the upper limit of the cone, causing gripping instability. This application's design introduces a slip-free rolling condition for the roller fingertip. The roller fingertip is not constrained by the traditional friction cone, but rather by the rolling friction torque. control:

[0049] ;

[0050] in, This refers to the rolling friction torque; The normal force at the point of contact; Where is the radius of the roller; The rolling friction coefficient is typically much smaller than the static friction coefficient. ,Right now:

[0051] ;

[0052] in, This refers to the rolling friction torque; This is the vector of the tangential frictional force at the point of contact. Where is the radius of the roller.

[0053] The method provided in this embodiment integrates a motor-driven roller fingertip module into the original Leaphand-v1 dexterous hand. From a kinematic perspective, it constructs an extended Jacobian matrix including the rotational degree of freedom of the roller to verify that the newly added degree of freedom is independent of the original joint degree of freedom column space, thus improving finger operability. At the same time, from the perspective of contact mechanics, it establishes a non-holonomic rolling contact model, transforming the traditional tangential friction constraint into a rolling friction torque constraint and verifying that the rolling friction coefficient is much smaller than the static friction coefficient. It adopts a non-slip rolling condition to limit the motion form, so that the modified dexterous hand has better posture adjustment capability and contact stability when performing hand-to-hand redirection operations on hole parts. It can accurately adjust the hole part to the assembly adaptation posture that matches the insertion direction of the shaft part and maintain a stable clamping state without slippage or loosening. This provides a reliable and accurate hole part posture basis for subsequent tactile grippers to hold shaft parts for two-handed collaborative shaft-hole assembly, while reducing friction interference and slippage risk during the contact process, improving the compliance, accuracy and success rate of the two-handed collaborative docking and assembly process, and thus ensuring the stable and efficient completion of the entire tactile guided shaft-hole assembly process.

[0054] S102. The robot uses its modified dexterous hand to grasp the hole part, uses a visual sensing system to collect the initial pose information of the hand object, and uses a vision-guided hierarchical control architecture to perform a repositioning operation within the hand, adjusting the hole part to a shaft-hole assembly adaptation pose that matches the insertion direction of the shaft part.

[0055] The hierarchical control architecture includes an offline learning module and an online control module.

[0056] Specifically, the initial pose information of the hand refers to the initial pose of the dexterous hand end in the world coordinate system and the initial position and orientation of the hole part relative to the dexterous hand or the world coordinate system, which are collected and calculated by the vision sensing system after the robot's modified dexterous hand grasps the hole part. It includes position coordinates and orientation angle information and is the initial input data for performing the in-hand redirection operation.

[0057] The hierarchical control architecture refers to the two-layer control structure used in this application for performing in-hand redirection, which is divided into an offline learning module and an online control module. The offline learning module is used to train multiple types of in-hand operation motion primitives using reinforcement learning algorithms and construct a reward function that integrates pose error, grasping stability coefficient, motion energy consumption parameters and terminal reward. The online control module is used to arrange motion primitive sequences based on the pose deviation obtained by the visual sensing system and combined with a large language model to drive the dexterous hand to adjust the pose of the hole parts. The two modules work together to complete the training and online execution of in-hand redirection.

[0058] In specific implementation, the offline learning module uses a deep deterministic policy gradient algorithm to train multiple types of hand-operation motion primitives. During training, a state space including finger joint angles, end-effector pose, current and target object poses are defined, as well as a motion space including end-effector linear velocity and angular velocity. A reward function incorporating pose error, grasping stability coefficient, and motion energy consumption parameters is designed to provide feedback optimization for the motion primitive training process until the training of each motion primitive converges. The observation submodule of the online control module acquires hand-object pose information through visual sensing and uses a pose estimation algorithm to obtain error data between the current pose of the hole part and the target assembly pose. The orchestration submodule of the online control module, combining a large language model and shaft-hole assembly task instructions, selects and orchestrates the optimal motion primitive sequence from the trained motion primitives based on the error data. This drives the dexterous hand roller fingertip and finger joints to move in coordination, executing the pose adjustment action of the hole part according to the optimal motion primitive sequence until the hole part pose error is less than a preset pose error threshold, adjusting the hole part to a shaft-hole assembly adaptation pose that matches the shaft part's insertion direction.

[0059] Optionally, a reward function integrating pose error, grasping stability coefficient, and motion energy consumption parameter is designed, including: defining the pose error of the hole component during the in-hand redirection process; decomposing the hole component pose error into position error and attitude error; normalizing the position error and attitude error respectively; assigning a weight coefficient to the normalized attitude error and weighting and summing it with the normalized position error to obtain a comprehensive hole component pose error value; setting the grasping stability coefficient and motion energy consumption parameter; wherein, the grasping stability coefficient is assigned a value based on the contact state between the dexterous hand and the hole component, and the value range is 0. Up to 1, the motion energy consumption parameter is the L2 norm of the dexterity hand joint motion; by configuring a first weighting coefficient to penalize the comprehensive hole part pose error value, configuring a second weighting coefficient to reward the grasping stability coefficient, and configuring a third weighting coefficient to penalize the motion energy consumption parameter, the three are weighted and calculated to obtain the basic reward; when the comprehensive hole part pose error value is less than the preset pose error threshold, it is determined that the pose adjustment has reached the target requirement and a fixed terminal reward is given; if it has not reached the target, the terminal reward is 0; the basic reward and the terminal reward are summed to obtain the reward function.

[0060] Specifically, the training framework for the deep deterministic policy gradient algorithm is determined, defining the state space and action space: the state space includes the dexterous hand's finger joint angles, fingertip pose, current pose of the hole part, and target assembly pose of the hole part; the action space includes the dexterous hand's end-effector linear velocity and end-effector angular velocity. A reward function is designed that integrates pose error, grasping stability coefficient, and motion energy consumption parameters: the hole part's pose error is decomposed into position error and attitude angle error, normalized, and weighted to obtain a comprehensive hole part pose error value, with penalty weights configured; a grasping stability coefficient in the 0-1 interval is set based on the contact state between the dexterous hand and the hole part, with reward weights configured; the L2 norm of the dexterous hand's joint motion is calculated as a motion energy consumption parameter, with penalty weights configured; simultaneously, a terminal reward is set, with a preset pose error threshold. When the comprehensive hole part pose error value is lower than the preset pose error threshold, a fixed success reward is given; otherwise, the terminal reward is 0; the base reward and the terminal reward are summed to obtain the total reward function. Based on the aforementioned state space, action space, and reward function, the deep deterministic policy gradient algorithm training process is initiated. The parameters of the policy network and evaluation network are iteratively updated to drive the dexterous hand to simulate and execute in-hand motion primitives such as rotation around the X-axis, rotation around the Y-axis, rotation around the Z-axis, translation along the Z-axis, and fingertip rolling adjustments. The training convergence index of each motion primitive is monitored in real time. When the index reaches the preset convergence condition (e.g., the reward value fluctuation is less than a set threshold for multiple consecutive rounds), training is stopped, and the trained motion primitive model and parameters are saved.

[0061] Furthermore, the visual sensing system is activated to collect hand-object image data after the modified dexterous hand grasps the hole part. The observation submodule calls the pre-calibrated intrinsic and extrinsic parameters of the visual sensor and uses a pose estimation algorithm to calculate the current pose (position coordinates + attitude angle) of the hole part and the preset target pose for shaft-hole assembly. The error data between the current pose and the target pose of the hole part is calculated, including the position error vector and the attitude angle error value. After standardizing the error data, it is input into the orchestration submodule. The orchestration submodule loads the trained motion primitive model, combines it with the pre-loaded shaft-hole assembly task instructions and the large language model, and selects suitable motion primitives from the motion primitive library according to the type and magnitude of the error data. It then orchestrates the optimal motion primitive sequence according to execution priority and action connection logic. The orchestrated motion primitive sequence is converted into control instructions for the dexterous hand joints and roller fingertips, driving the dexterous hand finger joint movements and roller fingertip rotation movements to be executed in coordination, adjusting the hole part pose in real time. The visual sensing system continuously collects the hand object's pose data, and the observation submodule calculates the pose error of the hole part in real time. If the error does not reach the preset pose error threshold, the motion element sequence adjustment action is repeated. If the error reaches the preset pose error threshold, the pose adjustment is stopped immediately. At this time, the hole part has been adjusted to the shaft hole assembly adaptation pose that matches the insertion direction of the shaft part.

[0062] For example, in one embodiment, the entire reinforcement learning task definition consists of three parts: a state space, an action space, and a reward function. The system state and action space are defined as follows: and :

[0063]

[0064] ;

[0065] in, This refers to the system status; For the finger joint angle, This is the end pose. and These represent the current pose of the object and the pose of the target, respectively. For action space; The terminal linear velocity, This represents the terminal angular velocity.

[0066] The positional error of a hole part is defined as follows:

[0067] ;

[0068] in, For the positional error of the hole part; and The normalized scaling factor. These are the attitude error weighting coefficients; For positional error, This represents the attitude error.

[0069] In summary, the basic reward function is set as follows:

[0070] ;

[0071] in, Basic rewards; The normalization weighting coefficient is used to balance the relationship between accuracy, stability and energy consumption. It is usually set according to experience. For example, the normalization weighting coefficient can be set to 0.3, 0.3, or 0.4. For the positional error of the hole part; To capture the stability coefficient; This is the action space.

[0072] When the pose error of the hole part reaches the preset pose error threshold, a reward is given to the terminal:

[0073] ;

[0074] in, For terminal rewards; For the positional error of the hole part; This is the preset pose error threshold.

[0075] The total reward function is:

[0076] ;

[0077] in, For the total reward function; Basic rewards; For terminal rewards.

[0078] S103. The robot-controlled calibrated tactile gripper grasps the shaft part, drives the dexterous hand and tactile gripper to complete the two-handed collaborative docking and enter the shaft hole assembly stage, and collects force / torque sensing data in real time during the shaft hole contact process.

[0079] Specifically, force / torque sensing data refers to the six-dimensional physical quantities of the contact interaction between the shaft and the hole, collected in real time by force / torque sensors integrated into the tactile gripper or roller fingertip module during the shaft-hole assembly process. These quantities include three-dimensional force components and three-dimensional torque components. The three-dimensional force components include contact forces along the three orthogonal directions X, Y, and Z (corresponding to axial, radial, and lateral forces, respectively), while the three-dimensional torque components include contact torques around the three orthogonal axes X, Y, and Z (corresponding to torques in the roll, pitch, and yaw directions, respectively).

[0080] In practice, the robot control system sends a gripping command to the calibrated tactile gripper. The tactile gripper closes the actuator according to the preset gripping force parameters to achieve stable gripping of the shaft part. The control system plans the collaborative motion trajectory of the dexterous hand and the tactile gripper based on the assembly adaptation posture of the hole part held by the dexterous hand. The control system drives the dexterous hand and the tactile gripper to move synchronously according to the trajectory, so that the shaft part and the hole part gradually approach each other and complete the initial docking, entering the shaft-hole assembly stage. During the shaft-hole assembly process, the force / torque sensor built into the tactile gripper collects the three-dimensional force components and three-dimensional torque components generated by the contact between the shaft and the hole in real time, and continuously outputs the force / torque sensing data according to the preset acquisition frequency and uploads it to the robot control system.

[0081] S104. Based on real-time force / torque sensing data, a factor graph optimization model integrating multiple factors is used to jointly estimate the contact state and the six-degree-of-freedom relative pose of the shaft to the hole during the shaft-hole assembly process in real time, and solve and output the current optimal contact state and optimal relative pose.

[0082] Specifically, the factor graph optimization model uses a factor graph as its data structure, with motion factors, observation factors, and state transition factors as constraint nodes, and the shaft-hole assembly mixed state as variable nodes. It optimizes the state estimation model by maximizing the posterior probability. Contact state refers to the contact form between the shaft and hole during assembly, specifically including hole-searching state, single-point contact state, two-point contact state, and surface contact state. The six-degree-of-freedom relative pose refers to the three axial position components and three orthaxial attitude components of the shaft part relative to the hole part in three-dimensional space, used to describe the relative position and attitude relationship between the shaft and hole. Optimal means that the combination of contact state and six-degree-of-freedom relative pose obtained in the factor graph optimization model minimizes the sum of the constraint residuals of all factors and is consistent with the actual assembly physical process.

[0083] In specific implementation, a factor graph optimization model integrating motion factors, observation factors, and state transition factors is constructed. The shaft-hole assembly state is decomposed into a six-degree-of-freedom continuous pose of the shaft relative to the hole and a discrete contact configuration between the shaft and the hole, serving as a mixed state for joint estimation by the model. Real-time force / torque sensing data, robot end-effector control input, and initial assembly pose information are input into the model to complete the calculation and constraint of each factor. Among them, the motion factor calculates the deviation between the actual pose change and the expected pose change of the end-effector control input based on the motion law of the continuous pose constrained by the robot end-effector control input. The observation factor first predefines a contact manifold library containing single-point, two-point, and surface contacts, and each contact... The manifold is constructed based on the force decomposition, force-torque balance, and friction constraints of the corresponding contact configuration. The minimum distance from the actual force / torque sensing data to each contact manifold is then calculated as the matching deviation between the observed value and the contact manifold. The state transition factor limits the reasonable evolution sequence of the discrete contact configuration according to the actual process rules of shaft-hole assembly. The deviation values ​​of each factor are incorporated into the model optimization objective. The optimal estimation of the mixed state is performed by solving the maximum a posteriori probability. Among all combinations of continuous poses and discrete contact configurations, the optimal combination with the highest matching degree with each factor constraint is selected. The optimal contact state of the current shaft-hole assembly and the optimal six-degree-of-freedom relative pose of the shaft to the hole corresponding to the optimal combination are output.

[0084] Optionally, the calculation process of the observation factor includes: constructing corresponding contact manifolds for single-point, two-point, and surface contact configurations in the shaft-hole assembly process. During construction, the total contact force is first decomposed into normal force and tangential force. Then, the theoretical contact torque is determined based on the correspondence between the contact point position and the contact force. At the same time, the range of tangential force values ​​is limited by friction constraints. The feasible force / torque set under each contact configuration is determined by combining the torque balance condition and friction constraints, forming a contact manifold library. The real-time collected force / torque sensing data is matched with the contact manifolds corresponding to each contact configuration in the contact manifold library to calculate the minimum distance from the actual sensing data to each contact manifold. The minimum distance is the matching residual between the sensing data and the corresponding contact manifold. In the calculation of the matching residual, a force / torque measurement noise weighting matrix is ​​introduced to differentiate the weights of measurement noise in different dimensions. The matching residuals corresponding to each contact configuration are used as observation factors.

[0085] Specifically, the overall framework of the factor graph optimization model is constructed, and the variable nodes and factor nodes of the model are defined. The variable nodes are the shaft-hole assembly hybrid state, and the factor nodes are three types of constraint factors: motion factors, observation factors, and state transition factors. The shaft-hole assembly hybrid state is defined and decomposed into two types of core variables: first, the six-degree-of-freedom continuous pose of the shaft relative to the hole, including position components along the three orthogonal directions X, Y, and Z and attitude components around the three orthogonal axes; second, the discrete contact configuration between the shaft and the hole, covering four types: hole search state, single-point contact state, two-point contact state, and surface contact state. The input data required by the model is organized, including: real-time collected shaft-hole contact force / torque sensing data (three-dimensional force components, three-dimensional torque components), robot end-effector control input data (joint control commands and motion speed commands of the dexterous hand and tactile gripper), and initial assembly pose information (initial pose of shaft parts, assembly adaptation pose of hole parts, and initial relative pose of both hands working together). The input data is preprocessed, and the force / torque sensing data is noise filtered to remove abnormal data. The control input data and initial pose data are standardized to unify the data coordinate system and units. All preprocessed data are then synchronously imported into the factor graph optimization model.

[0086] Furthermore, the robot end-effector control input data is extracted, and the expected six-DOF pose change of the axis relative to the hole is derived based on the control input commands. The actual pose change of the axis relative to the hole is collected through the vision sensing system and joint encoder, and the deviation between the actual pose change and the expected pose change is calculated, which is used as the constraint term of the motion factor. Observation factor calculation and constraint are performed: a contact manifold library is pre-constructed, containing contact manifolds corresponding to single-point contact, two-point contact, and surface contact configurations. When constructing each contact manifold, the total contact force is first decomposed into normal and tangential directions. The theoretical contact torque is derived based on the correspondence between the contact point position and the contact force. Combined with the torque balance condition and friction constraints, the feasible force / torque set under each contact configuration is determined, forming the corresponding contact manifold. The real-time preprocessed force / torque sensing data is matched with each contact manifold in the contact manifold library to calculate the minimum distance from the actual sensing data to each contact manifold. This distance is the matching deviation between the observed value and the corresponding contact manifold, which is used as the constraint term of the observation factor. The state transition factor is calculated and constrained: Based on the actual technological processes of shaft-hole assembly, a reasonable evolution sequence for discrete contact configurations is defined (hole-searching state → single-point contact state → two-point contact state → surface contact state). State transition rules are set to prohibit illogical contact state transitions (such as directly jumping from the hole-searching state to the surface contact state). According to this evolution sequence and transition rules, constraint terms for the state transition factor are constructed to limit the range of variation of the discrete contact configuration. Finally, the deviation values ​​of the motion factor, the matching deviation value of the observation factor, and the constraint deviation value of the state transition factor are weighted and summed to form the total deviation objective function, with the optimization objective being to minimize the total deviation. A maximum a posteriori probability algorithm is used to solve the model optimization objective, traversing all possible combinations of the shaft-to-hole six-degree-of-freedom continuous pose and the discrete contact configuration of the shaft and hole, calculating the total deviation value corresponding to each combination. The combination with the smallest total deviation value that conforms to the actual physical laws of shaft-hole assembly is selected; this combination is the optimal solution for the mixed state, corresponding to the optimal contact state of the current shaft-hole assembly and the optimal six-degree-of-freedom relative pose of the shaft-to-hole.

[0087] For example, in one embodiment, the six-degree-of-freedom continuous pose of the shaft relative to the hole can be represented as follows: The discrete contact configuration between the shaft and the hole can be represented as The robot's control input is speed, denoted as . At each time step, the tactile sensor provides a multimodal observation vector. Includes: contact force / torque: Measurements are taken in the sensor coordinate system.

[0088] The goal of the factor graph optimization model is to optimize the model at a given initial pose. In this case, determine an optimal control sequence. including continuous pose and discrete contact configuration This is to drive the system to complete the assembly while minimizing contact forces and avoiding failure states such as jamming. In other words:

[0089] ;

[0090] in, This is the optimal control sequence; The optimal relative pose for six degrees of freedom; This represents the optimal contact state. The six-degree-of-freedom continuous pose of the shaft relative to the hole; This is a discrete contact configuration between the shaft and the hole; This is the multimodal observation vector.

[0091] To address the aforementioned mixed-state estimation problem, a factor graph optimization model was constructed. This model aims to jointly estimate the state sequence by solving for the maximum a posteriori probability (MAP). and contact configuration sequence Its optimization objective can be expressed as:

[0092] ;

[0093] in, This is the optimal control sequence; The optimal relative pose for six degrees of freedom; This represents the optimal contact state. The six-degree-of-freedom continuous pose of the shaft relative to the hole; This is a discrete contact configuration between the shaft and the hole; This is the multimodal observation vector.

[0094] This posterior probability can be decomposed into a product of a series of factors:

[0095] ;

[0096] in, The six-degree-of-freedom continuous pose of the shaft relative to the hole; This is a discrete contact configuration between the shaft and the hole; This represents the multimodal observation vector; The initial state prior probability; For time step; for The six-degree-of-freedom continuous pose of the time axis relative to the hole; for The six-degree-of-freedom continuous pose of the time axis relative to the hole; for Time-sensitive robot end-effector control input; for Force / torque sensing data at any given time; for Discrete contact configurations at any given time; for Discrete contact configurations at any given time; For motion factors; For observation factors; This is the state transition factor.

[0097] The residual of the motion factor is defined as:

[0098] ;

[0099] in, The residuals of the motion factor; for The six-degree-of-freedom continuous pose of the time axis relative to the hole; for The six-degree-of-freedom continuous pose of the time axis relative to the hole; for Time-sensitive robot end-effector control input; For time step.

[0100] The residual of the observed factor is defined as:

[0101] ;

[0102] in, The residuals of the observed factors; for The six-degree-of-freedom continuous pose of the time axis relative to the hole; for Discrete contact configurations at any given time; for Force / torque sensing data at any given time; This represents the theoretically feasible force / torque vector. It is a contact manifold.

[0103] Taking single-point contact as an example, let the position of the contact point in the object coordinate system be... The contact normal is normal force Tangential force for:

[0104] ;

[0105] ;

[0106] in, Normal force; For the total contact force; For contact normal; It is a tangential force.

[0107] The theoretical torque and friction cone constraint under stable contact can be expressed as:

[0108] ;

[0109] ;

[0110] in, The theoretical torque; This represents the position of the contact point in the object's coordinate system. Normal force; It is a tangential force; Let be the coefficient of friction. From this, the single-point contact manifold can be obtained:

[0111] ;

[0112] in, It is a single-point contact manifold; The six-degree-of-freedom continuous pose of the shaft relative to the hole; It is a force / torque vector; This represents the position of the contact point in the object's coordinate system. Total contact force; Normal force; It is a tangential force; is the coefficient of friction.

[0113] Other contact configurations (such as two-point or surface contact) can be obtained by superimposing the force balance equations at the contact points.

[0114] Assuming two independent contact points exist , Then, the manifold of two-point contact (line contact) can be represented as:

[0115] ;

[0116] in, It is a two-point contact manifold; The six-degree-of-freedom continuous pose of the shaft relative to the hole; For force / torque combination, For the total three-dimensional contact force, This represents the total three-dimensional torque; , Let be the position vectors of two independent contact points; , These are the three-dimensional contact force vectors acting on the two contact points respectively; It is a tangential force; For the first The static friction coefficient at each contact point; For the first Normal force at each contact point.

[0117] A surface contact manifold can be represented as:

[0118] ;

[0119] in, It is a surface contact manifold; The six-degree-of-freedom continuous pose of the shaft relative to the hole; For force / torque combination, For the total three-dimensional contact force, The total three-dimensional moment is obtained by integrating the normal distributed force of the stable contact surface, which satisfies the friction cone constraint.

[0120] Among them, the contact area is considered to be a continuous surface. The resultant force and resultant moment are:

[0121] ;

[0122] ;

[0123] in, For the total three-dimensional contact force, This represents the total three-dimensional torque; This is a continuous contact area; Let the position vector of a point within the contact area be denoted as . This is the distributed force density vector; Let be the area of ​​the infinitesimal element within the contact region.

[0124] S105. Based on the optimal contact state and optimal relative pose, the assembly control parameters are dynamically adjusted using an adaptive impedance control strategy driven by the contact state. Force / torque control commands adapted to the current contact state are output to drive the robot to perform assembly actions in a coordinated manner with both hands until the shaft and hole assembly is completed.

[0125] Specifically, the adaptive impedance control strategy refers to using the optimal contact state and optimal relative pose as driving inputs, and dynamically adjusting impedance parameters such as desired stiffness, desired damping, and desired inertia according to different hole-searching states, single-point contact states, two-point contact states, and surface contact states. This transforms the position control of the robot's end effector into force / torque compliant control, and outputs force / torque control commands in real time that are adapted to the current contact state. This allows the robot's hands to maintain the desired contact force characteristics during shaft-hole assembly, avoiding jamming and impact, and ultimately completing the shaft-hole insertion smoothly.

[0126] In practical implementation, an adaptive control framework with impedance control as its core is built. The core adjustable parameters of the control framework are defined as the target pose and stiffness matrix, while the damping matrix is ​​set as a fixed adaptation parameter. The optimal relative pose output by the factor graph optimization model is incorporated into the control framework as the current actual pose. According to the type of optimal contact state, the target pose and stiffness matrix are dynamically configured, with different contact states corresponding to matching combinations of target pose and stiffness matrix parameters. The configured target pose, stiffness matrix, and current actual pose are substituted into the impedance control logic. The pose deviation and velocity deviation are combined with the stiffness matrix and damping matrix respectively to perform calculations, thereby obtaining the robot end effector command force / torque adapted to the current contact state.

[0127] Optionally, the contact state types include hole-searching state, single-point contact state, two-point contact state, and surface contact state. Different contact states correspond to matching target pose and stiffness matrix parameter combinations, including: for hole-searching state, configuring a stiffness matrix with all dimensions below a first threshold and a spiral search-type dynamic target pose, using the spiral search-type dynamic target pose to plan the search trajectory for shaft-hole docking; for single-point contact state, configuring a stiffness matrix with the normal direction above a second threshold and the tangential direction below a first threshold and an attitude calibration target pose, using the attitude calibration target pose to plan the centering trajectory within the axial hole; for two-point contact state, configuring a stiffness matrix with the tangential direction above a second threshold and the normal direction below a first threshold and an axis-centered target pose, using the axis-centered target pose to plan the assembly trajectory for the contact state to evolve into surface contact; for surface contact state, configuring a stiffness matrix with all dimensions between the first threshold and the second threshold and an axis-fully-inserted target pose, using the axis-fully-inserted target pose to plan the shaft's pressing and insertion trajectory, and the target pose supports the shaft's correction action; the first threshold is less than the second threshold.

[0128] Specifically, an adaptive control framework centered on impedance control is constructed. The framework's inputs are defined as the optimal contact state and optimal relative pose output from the factor graph optimization model, and its outputs are the robot's end-effector command force / torque. The core adjustable parameters of the control framework are defined as the target pose and stiffness matrix, while the damping matrix is ​​set as a fixed adaptation parameter (pre-configured as a 6×6 diagonal matrix value based on the shaft-hole assembly process characteristics). The optimal six-DOF relative pose of the shaft to the hole, output from the factor graph optimization model, is used as the current actual pose and connected to the state input port of the adaptive control framework. Predefined parameter configuration tables for different contact states: Hole search state: Configure a wide-range exploratory target pose (covering the possible pose range of shaft-hole docking) and a low-stiffness matrix (reducing the risk of contact force impact); Single-point contact state: Configure a small-range fine-tuning target pose (converging towards the coaxial direction of the shaft and hole) and a medium-stiffness matrix (ensuring pose adjustment accuracy); Two-point contact state: Configure a precise alignment target pose (completely coaxial between the shaft and hole) and a medium-high stiffness matrix (improving attitude correction capability); Surface contact state: Configure an insertion-propulsion target pose (propelling along the shaft-hole axial direction) and a high-stiffness matrix (ensuring insertion force stability). The optimal contact state output by the factor graph optimization model is read, and the corresponding target pose and stiffness matrix parameter combination is matched from the parameter configuration table according to the state type.

[0129] Furthermore, the difference between the configured target pose and the current actual pose is calculated to obtain a six-degree-of-freedom pose deviation vector (including position deviation and attitude deviation). The current actual six-degree-of-freedom velocity of the robot end effector is collected and the difference between it and the expected velocity corresponding to the target pose is calculated to obtain a six-degree-of-freedom velocity deviation vector. The pose deviation vector is multiplied by the dynamically configured stiffness matrix to obtain the stiffness force / torque components; the velocity deviation vector is multiplied by the fixed damping matrix to obtain the damping force / torque components; the two components are summed to obtain the robot end effector command force / torque adapted to the current contact state. The command force / torque is converted into joint control commands for the robot's two hands (dexterous hand + tactile gripper) to drive the actuator to perform the assembly action; the above parameter configuration and calculation process is continuously cyclical until the shaft part is completely inserted into the hole part, and the shaft-hole assembly is completed.

[0130] For example, in one embodiment, the adaptive impedance control strategy can be expressed as:

[0131] ;

[0132] in, To adapt the robot's end effector command force / torque to the current contact state; Here is the stiffness matrix; Here is the damping matrix; The target pose; This is the actual pose; For target speed; This refers to the actual speed.

[0133] The method provided in this embodiment, firstly, integrates a motor-driven roller fingertip module into the open-source Leaphand-v1 dexterous hand and completes hardware calibration. On one hand, by adding a roller rotational degree of freedom, the Jacobian matrix dimension of a single finger is expanded, verifying that the added degree of freedom is independent of the original joint degree of freedom column space, significantly improving finger operability and end-effector posture adjustment capabilities, and expanding the finger's reachable area. On the other hand, the traditional tangential friction cone constraint of the fingertip is transformed into a rolling friction torque constraint. By utilizing a rolling friction coefficient much smaller than the static friction coefficient and a slip-free rolling condition, the friction restriction and slip instability risk during the contact process are greatly reduced, providing a reliable hardware foundation for subsequent stable grasping, precise intra-hand redirection, and compliant assembly, fundamentally improving the dexterous hand's operational compliance and posture adjustment capabilities.

[0134] Secondly, a vision-guided hierarchical control architecture is adopted to perform in-hand repositioning of hole parts. In the offline stage, multiple types of in-hand operation motion primitives are trained through a deep deterministic strategy gradient algorithm to construct a reward function that integrates pose error, grasping stability coefficient, motion energy consumption parameters, and terminal rewards. In the online stage, the visual sensing system collects the initial pose information of the hand and object, and the pose error is calculated through a pose estimation algorithm. Combined with a large language model, the optimal motion primitive sequence is arranged to drive the dexterous hand's rollers, fingertips, and joints to move in coordination. This pre-adjusts the hole parts to an assembly-fit pose that matches the insertion direction of the shaft parts, effectively eliminating the interference of initial pose deviation on subsequent shaft-hole docking, reducing the alignment difficulty and jamming probability in the assembly stage, and providing accurate initial pose assurance for subsequent two-handed collaborative docking and assembly, significantly improving the smoothness and efficiency of the assembly process.

[0135] Thirdly, based on real-time acquired force / torque sensing data, a factor graph optimization model integrating motion factors, observation factors, and state transition factors is adopted to decompose the shaft-hole assembly state into a mixed state of six-degree-of-freedom continuous pose and discrete contact configuration. The motion factors constrain the pose motion law, the observation factors match the force / torque sensing data through a predefined single-point / two-point / surface contact manifold library, and the state transition factors limit the reasonable evolution sequence of the contact configuration. By maximizing the posterior probability, the joint real-time estimation of the mixed state is achieved. This not only ensures the accuracy of the six-degree-of-freedom relative pose of the shaft to the hole, but also accurately identifies discrete contact states such as hole searching, single-point contact, two-point contact, and surface contact. This provides a comprehensive, reliable, and real-time state input for subsequent adaptive impedance control, solving the limitation of traditional state estimation that only focuses on pose and ignores contact morphology, and improving the completeness and robustness of state perception.

[0136] Fourthly, based on the optimal contact state and optimal relative pose, an adaptive impedance control strategy driven by the contact state is adopted. An adaptive framework with impedance control as the core is built, with the stiffness matrix and target pose set as core adjustable parameters and the damping matrix set as fixed adaptation parameters. The stiffness matrix and target pose are dynamically configured for different contact states: low stiffness and exploratory target pose are used in the hole-finding stage to reduce impact; medium-high stiffness and fine-tuning target pose are used in the single-point / two-point contact stage to improve alignment accuracy; and high stiffness and propulsion target pose are used in the surface contact stage to ensure stable insertion force. By combining the pose deviation and velocity deviation with the stiffness matrix and damping matrix respectively, force / torque control commands adapted to the current contact state are output, realizing compliant control in different assembly stages. This effectively avoids impact, jamming, and overload problems in the shaft-hole docking process, and finally completes the shaft-hole insertion smoothly, comprehensively improving the success rate and safety of assembly.

[0137] Corresponding to the aforementioned embodiment of a robot two-handed tactile guide shaft hole assembly control method, this application also provides an embodiment of a robot two-handed tactile guide shaft hole assembly control device.

[0138] Figure 2 This is a schematic diagram of the robot's two-handed tactile guide shaft hole assembly control device provided in Embodiment 2 of this application. Please refer to... Figure 2 The device provided in this embodiment includes an integration module 210, an adjustment module 220, a data acquisition module 230, and an output module 240.

[0139] The integrated module 210 is used to build the hardware foundation for the collaborative operation of the dexterous hand and the tactile gripper by integrating the motor direct drive roller fingertip module of the open-source Leaphand-v1 robot, and to initialize and calibrate each sensor and actuator.

[0140] The adjustment module 220 is used by the robot to grasp the hole part with the modified dexterous hand, collect the initial pose information of the hand object using the visual sensing system, and perform the hand repositioning operation using a vision-guided hierarchical control architecture to adjust the hole part to the shaft hole assembly adaptation pose that matches the insertion direction of the shaft part.

[0141] The acquisition module 230 is used to control the calibrated tactile gripper of the robot to grasp the shaft part, drive the dexterous hand and the tactile gripper to complete the two-hand coordinated docking and enter the shaft hole assembly stage, and collect force / torque sensing data in real time during the shaft hole contact process.

[0142] The output module 240 is used to perform joint real-time estimation of the contact state and the six-degree-of-freedom relative pose of the shaft relative to the hole during the shaft-hole assembly process based on the real-time collected force / torque sensing data and using a factor graph optimization model that integrates multiple factors, and solves and outputs the current optimal contact state and optimal relative pose.

[0143] The output module 240 is also used to dynamically adjust the assembly control parameters according to the optimal contact state and the optimal relative pose using a contact state-driven adaptive impedance control strategy, and output force / torque control commands adapted to the current contact state to drive the robot to perform assembly actions in a coordinated manner until the shaft hole assembly is completed.

[0144] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0145] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0146] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0147] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for assembling and controlling tactile guided shaft holes for robotic hands, characterized in that, The method includes: The open-source Leaphand-v1 robot integrates a direct-drive motor roller fingertip module to build the hardware foundation for the collaborative operation of the dexterous hand and tactile gripper, and initializes and calibrates various sensors and actuators. The robot uses its modified dexterous hand to grasp the hole part, uses a visual sensing system to collect the initial pose information of the hand object, and uses a vision-guided hierarchical control architecture to perform the hand repositioning operation, adjusting the hole part to a shaft-hole assembly adaptation pose that matches the insertion direction of the shaft part. The robot controls the calibrated tactile gripper to grasp the shaft part, drives the dexterous hand and the tactile gripper to complete the two-hand coordinated docking and enter the shaft hole assembly stage, and collects force / torque sensing data in real time during the shaft hole contact process; Based on real-time force / torque sensing data, a factor graph optimization model integrating multiple factors is used to jointly estimate the contact state and the six-degree-of-freedom relative pose of the shaft to the hole during the shaft-hole assembly process in real time, and solve and output the current optimal contact state and optimal relative pose. Based on the optimal contact state and optimal relative pose, the assembly control parameters are dynamically adjusted using an adaptive impedance control strategy driven by the contact state. Force / torque control commands adapted to the current contact state are output to drive the robot to perform assembly actions in a coordinated manner with both hands until the shaft and hole assembly is completed. The step of dynamically adjusting assembly control parameters using a contact state-driven adaptive impedance control strategy based on the optimal contact state and optimal relative pose, and outputting force / torque control commands adapted to the current contact state, includes: An adaptive control framework with impedance control as its core is constructed. The core adjustable parameters of the control framework are defined as the target pose and stiffness matrix, and the damping matrix is ​​set as a fixed adaptation parameter. The optimal relative pose output by the factor graph optimization model is incorporated into the control framework as the current actual pose. Based on the type of optimal contact state, the target pose and stiffness matrix are dynamically configured, and different contact states correspond to matching combinations of target pose and stiffness matrix parameters. Substitute the configured target pose, stiffness matrix, and current actual pose into the impedance control logic. Then, by combining the pose deviation and velocity deviation with the stiffness matrix and damping matrix respectively, the robot end-effector command force / torque adapted to the current contact state is obtained. The types of contact states include hole-searching state, single-point contact state, two-point contact state, and surface contact state. Each contact state corresponds to a matching combination of target pose and stiffness matrix parameters, including: For the hole search state, configure a stiffness matrix with all dimensions below the first threshold and a spiral search dynamic target pose, and plan the search trajectory for shaft hole docking using the spiral search dynamic target pose; For single-point contact, configure a stiffness matrix with a normal direction higher than the second threshold and a tangential direction lower than the first threshold, and an attitude calibration target pose, and plan the centering trajectory in the axial hole using the attitude calibration target pose; For two-point contact states, a stiffness matrix with tangential direction higher than the second threshold and normal direction lower than the first threshold is configured, along with an axis-centered target pose. The assembly trajectory of the contact state evolving into surface contact is planned using the axis-centered target pose. For the surface contact state, a stiffness matrix with all dimensions between a first threshold and a second threshold is configured, and a target pose with full axis insertion is used to plan the downward insertion trajectory of the axis, and the target pose supports the correction action of the axis; the first threshold is less than the second threshold.

2. The method according to claim 1, characterized in that, The hierarchical control architecture includes an offline learning module and an online control module. It employs a vision-guided hierarchical control architecture to perform manual redirection operations, adjusting the hole part to a shaft-hole assembly adaptation pose that matches the insertion direction of the shaft part. This includes: The offline learning module employs a deep deterministic policy gradient algorithm to train multiple types of hand-operation motion primitives. During training, a state space is defined that includes finger joint angles, end-effector pose, current and target poses of the object, and an action space that includes end-effector linear velocity and angular velocity. At the same time, a reward function that integrates pose error, grasping stability coefficient, and motion energy consumption parameters is designed to provide feedback optimization for the motion primitive training process until the training of each motion primitive converges. The observation submodule of the online control module acquires the hand object pose information through visual sensing, and uses a pose estimation algorithm to obtain the error data between the current pose of the hole part and the target assembly pose. The arrangement submodule of the online control module combines the large language model and the shaft-hole assembly task instructions, selects and arranges the optimal motion primitive sequence from the trained motion primitives according to the error data, drives the dexterous hand roller fingertip and finger joint to move in coordination, and executes the pose adjustment action of the hole part according to the optimal motion primitive sequence until the pose error of the hole part is less than the preset pose error threshold, and adjusts the hole part to the shaft-hole assembly adaptation pose that matches the insertion direction of the shaft part.

3. The method according to claim 2, characterized in that, Design a reward function that integrates pose error, grasping stability coefficient, and motion energy consumption parameters, including: Define the hole part pose error during the hand reorientation process, decompose the hole part pose error into position error and attitude error, normalize the position error and attitude error respectively, assign a weight coefficient to the normalized attitude error and sum it with the normalized position error to obtain the comprehensive hole part pose error value. Set a grasping stability coefficient and a motion energy consumption parameter; wherein, the grasping stability coefficient is assigned a value based on the contact state between the dexterous hand and the hole part, and the value range is 0 to 1, and the motion energy consumption parameter is the L2 norm of the dexterous hand joint motion amount; The basic reward is obtained by weighting the comprehensive hole part pose error value with a first weighting coefficient for penalty, the grasping stability coefficient with a second weighting coefficient for reward, and the motion energy consumption parameter with a third weighting coefficient for penalty. When the overall hole part pose error value is less than the preset pose error threshold, it is determined that the pose adjustment has met the target requirements and a fixed terminal reward is given; otherwise, the terminal reward is 0. The reward function is obtained by summing the basic reward and the terminal reward.

4. The method according to claim 1, characterized in that, The solution outputs the current optimal contact state and optimal relative pose, including: A factor graph optimization model integrating motion factors, observation factors, and state transition factors is constructed. The shaft-hole assembly state is decomposed into a six-degree-of-freedom continuous pose of the shaft relative to the hole and a discrete contact configuration between the shaft and the hole, which are used as a mixed state for joint estimation by the model. The model is input with real-time force / torque sensing data, robot end-effector control input, and initial assembly pose information. Each factor is then calculated and constrained. Specifically, the motion factor calculates the deviation between the actual pose change and the expected pose change based on the continuous pose constraints imposed by the robot end-effector control input. The observation factor predefines a library of contact manifolds containing single-point, two-point, and surface contacts. Each contact manifold is constructed based on the force decomposition, force-torque balance, and friction constraints of its corresponding contact configuration. The minimum distance from the actual force / torque sensing data to each contact manifold is then calculated as the matching deviation between the observed value and the contact manifold. The state transition factor, based on the actual process rules of shaft-hole assembly, limits the reasonable evolution sequence of discrete contact configurations. The deviation values ​​of each factor are incorporated into the model optimization objective. The optimal estimate of the mixed state is obtained by solving the maximum a posteriori probability. Among all combinations of continuous poses and discrete contact configurations, the optimal combination with the highest matching degree with each factor constraint is selected. The optimal combination is output as the current optimal contact state of the shaft-hole assembly and the optimal six-degree-of-freedom relative pose of the shaft to the hole.

5. The method according to claim 4, characterized in that, The calculation process of the observed factor includes: For single-point, two-point, and surface contact configurations in the shaft-hole assembly process, corresponding contact manifolds are constructed. During construction, the total contact force is first decomposed into normal force and tangential force. Then, the theoretical contact torque is determined based on the correspondence between the contact point position and the contact force. At the same time, the range of tangential force is limited by friction constraints. The feasible force / torque set under each contact configuration is determined by combining the torque balance condition and friction constraints, forming a contact manifold library. The real-time force / torque sensing data is matched with the contact manifolds corresponding to each contact configuration in the contact manifold library to calculate the minimum distance from the actual sensing data to each contact manifold. The minimum distance is the matching residual between the sensing data and the corresponding contact manifold. In the process of calculating the matching residual, a force / torque measurement noise weighting matrix is ​​introduced to allocate differentiated weights to the measurement noise in different dimensions. The matching residuals corresponding to each contact configuration are used as observation factors.

6. The method according to claim 1, characterized in that, After establishing the hardware foundation for collaborative operation of the dexterous hand and tactile gripper using the integrated motor direct-drive roller fingertip module of the open-source Leaphand-v1 robot, the method further includes: A kinematic model including the rotational degree of freedom of a roller is established for a single finger of a dexterous hand. The roller rotation angle, roller radius, and rolling tangential unit vector are incorporated into the modeling system of the finger end pose and Jacobian matrix. The newly added rolling degree of freedom column vector is verified to be independent of the original matrix column space through matrix expansion analysis. A non-holistic rolling contact model between the roller fingertip and the object is established. By comparing the original friction constraint form of the fingertip, it is verified that the roller fingertip transforms the tangential friction constraint into a rolling friction torque constraint, and the rolling friction coefficient is less than the static friction coefficient. At the same time, the motion form of the roller is limited by the no-slip rolling condition.

7. The method according to claim 1, characterized in that, The motor-driven roller fingertip module adopts a modular structure design and is embedded into the fingertips of the open-source Leaphand-v1 dexterous hand, replacing the original 3D-printed fingertip structure without interfering with the original gripping function of the dexterous hand. The motor-driven roller fingertip module includes left and right support structures, bearings, encoders, drive servos and servo brackets. The main structure of the module is made of PETG material through 3D printing, and the internal gears are CNC machined. The drive servos are connected to the rollers through plug-in buckles. The circuit part integrates a multi-layer flexible PCB and is integrated with the tactile array. The total width of the roller assembly of the motor-driven roller fingertip module does not exceed 40mm, and the roller diameter is set to 25mm, which is adapted to the overall structural size of the dexterous hand. The alignment error during the module assembly process is less than 0.2mm, and the total degree of freedom of the modified dexterous hand is increased to 20 degrees of freedom.

8. A robotic two-hand tactile guide shaft hole assembly control device, characterized in that, The device includes an integration module, an adjustment module, a data acquisition module, and an output module; The integrated module is used to build the hardware foundation for the collaborative operation of the dexterous hand and tactile gripper by integrating the motor direct drive roller fingertip module of the open-source Leaphand-v1 robot, and to initialize and calibrate each sensor and actuator. The adjustment module is used by the robot to grasp the hole part with the modified dexterous hand, collect the initial pose information of the hand object using the visual sensing system, and perform the hand repositioning operation with the visual guidance hierarchical control architecture to adjust the hole part to the shaft hole assembly adaptation pose that matches the insertion direction of the shaft part. The acquisition module is used to control the calibrated tactile gripper of the robot to grasp the shaft part, drive the dexterous hand and the tactile gripper to complete the two-hand coordinated docking and enter the shaft hole assembly stage, and collect force / torque sensing data in real time during the shaft hole contact process. The output module is used to perform joint real-time estimation of the contact state and the six-degree-of-freedom relative pose of the shaft to the hole during the shaft-hole assembly process based on the real-time collected force / torque sensing data and a factor graph optimization model that integrates multiple factors, and solves and outputs the current optimal contact state and optimal relative pose. The output module is also used to dynamically adjust the assembly control parameters according to the optimal contact state and the optimal relative pose using a contact state-driven adaptive impedance control strategy, and output force / torque control commands adapted to the current contact state to drive the robot to perform assembly actions in a coordinated manner until the shaft hole assembly is completed. The step of dynamically adjusting assembly control parameters using a contact state-driven adaptive impedance control strategy based on the optimal contact state and optimal relative pose, and outputting force / torque control commands adapted to the current contact state, includes: An adaptive control framework with impedance control as its core is constructed. The core adjustable parameters of the control framework are defined as the target pose and stiffness matrix, and the damping matrix is ​​set as a fixed adaptation parameter. The optimal relative pose output by the factor graph optimization model is incorporated into the control framework as the current actual pose. Based on the type of optimal contact state, the target pose and stiffness matrix are dynamically configured, and different contact states correspond to matching combinations of target pose and stiffness matrix parameters. Substitute the configured target pose, stiffness matrix, and current actual pose into the impedance control logic. Then, by combining the pose deviation and velocity deviation with the stiffness matrix and damping matrix respectively, the robot end-effector command force / torque adapted to the current contact state is obtained. The types of contact states include hole-searching state, single-point contact state, two-point contact state, and surface contact state. Each contact state corresponds to a matching combination of target pose and stiffness matrix parameters, including: For the hole search state, configure a stiffness matrix with all dimensions below the first threshold and a spiral search dynamic target pose, and plan the search trajectory for shaft hole docking using the spiral search dynamic target pose; For single-point contact, configure a stiffness matrix with a normal direction higher than the second threshold and a tangential direction lower than the first threshold, and an attitude calibration target pose, and plan the centering trajectory in the axial hole using the attitude calibration target pose; For two-point contact states, a stiffness matrix with tangential direction higher than the second threshold and normal direction lower than the first threshold is configured, along with an axis-centered target pose. The assembly trajectory of the contact state evolving into surface contact is planned using the axis-centered target pose. For the surface contact state, a stiffness matrix with all dimensions between a first threshold and a second threshold is configured, and a target pose with full axis insertion is used to plan the downward insertion trajectory of the axis, and the target pose supports the correction action of the axis; the first threshold is less than the second threshold.

Images