Vision-touch based hand control methods, electronic devices, storage media, and software products

Abstract

This application provides a hand control method, electronic device, storage medium, and program product based on visual-touch integration. The method includes: acquiring tactile sensing data and joint state data of the hand; determining the curvature information of the fingertip surface based on the joint state data; performing deformation compensation on the tactile sensing data based on the curvature information to generate a compensated tactile tensor; acquiring visual observation data of the hand; encoding the visual observation data into a visual feature sequence; encoding the compensated tactile tensor into a tactile feature sequence; performing cross-modal attention fusion using the tactile feature sequence as a query and the visual feature sequence as a key and value to generate cross-modal alignment features; generating an operation state vector based on the cross-modal alignment features; and generating control commands based on the sliding velocity and contact force information in the operation state vector to drive the hand to perform a grasping action.

Description

Technical Field

[0001] This application relates to the field of robot perception and control, and in particular to a gripper control method, electronic device, storage medium and program product based on vision-touch combination. Background Technology

[0002] In industrial automation, service robots, and medical assistance, dexterous hands typically use a combination of visual perception and tactile feedback to perform grasping, assembly, and contact operations.

[0003] However, existing solutions often rely on rule switching, fixed weight fusion, or end-to-end feature fusion, which makes it difficult to take into account factors such as close-range occlusion, tactile deformation, modal asynchrony, and material changes. This can easily lead to state estimation bias, increased response latency, and insufficient cross-scene adaptability, thereby affecting grasping stability and control accuracy.

[0004] Therefore, how to improve the consistency, real-time performance, and environmental adaptability of multimodal perception and control in visual-touch collaborative operation has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a gripper control method, electronic device, storage medium, and program product based on visual-touch integration, in order to improve the consistency of multimodal information, real-time response, and scene adaptability, and enhance the grasping accuracy and stability.

[0006] In a first aspect, embodiments of this application provide a hand control method based on a combination of vision and touch, including:

[0007] Acquire tactile sensing data and joint state data of the hand, determine the curvature information of the fingertip surface based on the joint state data, and perform deformation compensation on the tactile sensing data based on the curvature information to generate the compensated tactile tensor.

[0008] Visual observation data of the hand is acquired, the visual observation data is encoded into a visual feature sequence, the compensated tactile tensor is encoded into a tactile feature sequence, and cross-modal attention fusion is performed using the tactile feature sequence as the query and the visual feature sequence as the key and value to generate cross-modal aligned features.

[0009] An operation state vector is generated based on cross-modal alignment features, and control commands are generated based on sliding velocity and contact force information to drive the gripper to perform grasping actions. The operation state vector contains sliding velocity and contact force information.

[0010] In one possible embodiment, the curvature information of the fingertip surface is determined based on joint state data, and deformation compensation is performed on the tactile sensing data based on the curvature information to generate a compensated tactile tensor, including:

[0011] Based on joint state data, the posture of the finger segments involved in contact is calculated using forward kinematics to obtain the curvature information of the fingertip surface;

[0012] A warping map of the tactile mesh is generated based on the curvature information. The warping map is used to record the sampling coordinates of the compensated tactile mesh position in the original tactile array.

[0013] The tactile sensing data is resampled by bilinear interpolation based on the warp mapping to generate a compensated tactile tensor.

[0014] In one possible embodiment, it also includes:

[0015] The pressure center is calculated based on the compensated tactile tensor, and the slip velocity is calculated based on the difference between the pressure center at the current moment and the previous moment, as well as the time interval.

[0016] In one possible embodiment, visual observation data is encoded into a visual feature sequence, the compensated tactile tensor is encoded into a tactile feature sequence, and cross-modal attention fusion is performed using the tactile feature sequence as a query and the visual feature sequence as a key and value to generate cross-modal aligned features, including:

[0017] Visual observation data is encoded into visual feature sequences, and the compensated tactile tensor is encoded into tactile feature sequences.

[0018] Using the tactile feature sequence as the query matrix and the visual feature sequence as the key and value matrices, the product of the query matrix and the key matrix is ​​calculated, and after normalization, it is multiplied by the value matrix to obtain the cross-modal alignment features.

[0019] In one possible embodiment, the tactile feature sequence is used as the query matrix, and the visual feature sequence is used as the key and value matrices. The product of the query matrix and the key matrix is ​​calculated, normalized, and then multiplied by the value matrix to obtain cross-modal alignment features, including:

[0020] The tactile feature sequence is multiplied by the stored projection weight matrix to obtain the tactile query matrix;

[0021] Multiply the visual feature sequence with the stored projection weight matrix to obtain the visual key matrix and the visual value matrix;

[0022] The product of the tactile query matrix and the visual key matrix is ​​calculated, normalized by scale and normalization exponential function, and then multiplied by the visual value matrix to obtain the cross-modal alignment feature.

[0023] In one possible embodiment, generating an operational state vector based on cross-modal alignment features includes:

[0024] The cross-modal alignment features are decoded by a decoder to generate an intermediate representation containing physical meaning;

[0025] The three-dimensional position and three-dimensional orientation of the target in the palm coordinate system are extracted from the intermediate representation;

[0026] Force information along the normal and tangential directions of the fingertip at the contact point is extracted from the intermediate representation. The force information is determined by the tactile normal pressure integral and the tangential pressure integral.

[0027] The fingertip curvature, sliding velocity, and friction coefficient estimates at the current moment are extracted from the intermediate representation. The sliding velocity is determined by the pressure center displacement rate of the compensated tactile tensor, and the friction coefficient estimates are determined by the ratio of normal force to tangential force.

[0028] The contact stiffness estimate, rebound time constant, and state confidence are extracted from the intermediate representation. The state confidence is determined by a combination of visual clarity, tactile signal-to-noise ratio, and cross-modal matching weights.

[0029] The analytically obtained state components are combined into a unified operational state vector.

[0030] In one possible embodiment, control commands are generated based on the sliding speed and contact force information to drive the gripper to perform a grasping action, including:

[0031] Determine whether the triggering condition is met based on the slip velocity and state confidence in the operation state vector;

[0032] When the sliding speed is greater than the preset speed threshold or the state confidence is less than the preset confidence threshold, the control parameters are adjusted to obtain the adjusted control parameters, which include grip force, stiffness matrix and damping matrix.

[0033] The desired trajectory is updated based on the adjusted control parameters to obtain the updated desired trajectory;

[0034] Based on the adjusted control parameters and the updated desired trajectory, joint torque commands are generated to drive the gripper to perform the grasping action.

[0035] In one possible embodiment, joint torque commands are generated based on the adjusted grip force, stiffness matrix, damping matrix, and desired trajectory, including:

[0036] Get the current joint angle and current joint velocity of the gripper;

[0037] Obtain the desired joint angle and desired joint velocity based on the desired trajectory;

[0038] The joint torque command is calculated based on the difference between the current joint angle and the desired joint angle, the difference between the current joint velocity and the desired joint velocity, the stiffness matrix, the damping matrix, and the contact force information.

[0039] In one possible embodiment, the tactile sensing data includes normal pressure data and tangential pressure data; the visual observation data includes color images and depth images; and the joint state data includes at least one of joint angles, joint velocities, and joint torques.

[0040] Secondly, embodiments of this application provide a hand control device based on a combination of vision and touch, comprising:

[0041] The acquisition module is used to acquire tactile sensing data and joint state data of the hand, determine the curvature information of the fingertip surface based on the joint state data, and perform deformation compensation on the tactile sensing data based on the curvature information to generate the compensated tactile tensor.

[0042] The fusion module is used to acquire visual observation data of the hand, encode the visual observation data into a visual feature sequence, encode the compensated tactile tensor into a tactile feature sequence, and perform cross-modal attention fusion with the tactile feature sequence as the query and the visual feature sequence as the key and value to generate cross-modal aligned features.

[0043] The drive module is used to generate an operation state vector based on cross-modal alignment features and to generate control commands based on sliding speed and contact force information to drive the gripper to perform grasping actions. The operation state vector includes sliding speed and contact force information.

[0044] Thirdly, embodiments of this application provide a hand control device based on a combination of visual and tactile feedback, including: a memory and a processor;

[0045] The memory stores computer-executed instructions;

[0046] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0047] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0048] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0049] The vision-touch combined hand control method, electronic device, storage medium, and program product provided in this application acquire tactile sensing data and joint state data of the hand, and combine the fingertip surface curvature information determined by the joint state to perform deformation compensation on the tactile sensing data, which can reduce the impact of hand posture changes and contact deformation on tactile representation; by encoding visual observation data and compensated tactile data separately, and performing cross-modal attention fusion with tactile features as queries and visual features as keys and values, the alignment degree of vision-touch information in spatiotemporal semantics can be improved; furthermore, by generating an operation state vector containing sliding speed and contact force information based on the cross-modal alignment features, and generating control commands based on this vector to drive the hand to perform grasping actions, the consistency of state estimation and control accuracy during the grasping process can be improved, and the grasping stability, real-time response capability, and cross-scene environment adaptability can be enhanced. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0051] Figure 1 The flowchart of the hand control based on vision-touch combination provided in this application Figure 1 ;

[0052] Figure 2 Flowchart of the vision-touch combined gripper control method provided in this application Figure 2 ;

[0053] Figure 3 A schematic diagram of the hand control device based on visual-touch integration provided in this application;

[0054] Figure 4 A schematic diagram of the hand control device based on visual-touch combination provided in this application.

[0055] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0056] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0057] Dexterous hand vision-touch collaborative control technology belongs to the field of robot perception and operation control, and is applicable to scenarios such as industrial automation grasping and assembly, multi-item operation of service robots, and medical assisted contact operations. Related systems typically include a visual observation unit for acquiring the appearance and spatial position of the target, a tactile sensing unit installed on the fingertips of the gripper, and a joint state acquisition unit for providing feedback on finger posture.

[0058] Existing dexterous hand solutions typically rely primarily on visual information to estimate the object's position and orientation when approaching it. After contact occurs, tactile information is then combined to determine the contact force, contact location, or whether slippage has occurred, thereby generating a grasping motion or adjusting the grasping force. Other solutions directly stitch visual and tactile features together or fuse them according to preset weights to achieve state recognition and control output.

[0059] However, visual observation is prone to distortion, defocusing, or localized loss when grasping at close range, encountering enhanced occlusion, or experiencing surface reflection. Simultaneously, pressure and bending of the fingertips deform tactile sensory data; without compensation related to joint posture, discrepancies can easily arise between the perceived tactile contact location and the actual contact state. Furthermore, visual and tactile senses originate from different sources and have different timeframes, making it difficult for fixed fusion methods to simultaneously account for localized contact changes and overall pose perception, leading to unstable state estimation.

[0060] The aforementioned defects can further lead to lag in slip recognition, inaccurate contact force adjustment, and over-gripping or slippage during the grasping process. This is especially true when there are significant changes in materials or dynamic changes in the operating environment, making it even more difficult to maintain grasping stability and control precision. Therefore, improving information consistency, real-time response, and scene adaptability in visual-touch collaborative operation has become a key issue that needs to be addressed.

[0061] In view of this, a hand control method based on visual-tactile integration is provided. After acquiring hand tactile sensing data, joint state data, and visual observation data, the curvature information of the fingertip surface is first determined based on the joint state data, and deformation compensation is performed on the tactile sensing data accordingly to generate a compensated tactile tensor. Then, the visual observation data is encoded into a visual feature sequence, and the compensated tactile tensor is encoded into a tactile feature sequence. Cross-modal alignment features are generated by cross-modal attention fusion with the tactile feature sequence as the query and the visual feature sequence as the key and value.

[0062] Based on this, an operational state vector containing sliding velocity and contact force information is generated according to cross-modal alignment features. Control commands are then generated based on the sliding velocity and contact force information to drive the gripper to perform grasping actions. This method can be applied to a gripper control architecture consisting of a visual observation unit, a fingertip tactile array, a joint state acquisition unit, and a control processing unit to improve the consistency of multimodal perception and control and the stability of grasping.

[0063] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0064] Figure 1 The flowchart of the hand control based on vision-touch combination provided in this application Figure 1 ,like Figure 1 As shown, the method includes:

[0065] S101: Acquire tactile sensing data and joint state data of the hand, determine the curvature information of the fingertip surface based on the joint state data, and perform deformation compensation on the tactile sensing data based on the curvature information to generate the compensated tactile tensor.

[0066] In this application, tactile sensing data is used to characterize the tactile information of the contact area between the hand and the target object; joint state data is used to characterize the motion state of each finger joint at the current moment; the curvature information of the fingertip surface is used to characterize the bending geometry formed by the fingertip under joint drive; deformation compensation is used to correct the deviation between the tactile sensing data and the actual contact state under bending state; and the compensated tactile tensor is a unified tactile representation formed after compensation processing. In specific implementation, the execution entity can be a processor set in the hand control processing unit, which is communicatively connected to both the tactile sensing unit and the joint state acquisition unit. In one possible embodiment, the processor acquires the tactile data of each fingertip and the joint state data at the corresponding moment to form input data that can be processed jointly within the same control cycle.

[0067] In the process of determining and compensating for curvature information, the processor determines the curvature information of the current fingertip surface based on joint state data, and performs compensation processing on the tactile sensing data based on this curvature information, thereby generating a compensated tactile tensor. This compensated tactile tensor can be organized in a tensor form of a predetermined dimension for subsequent participation in visual fusion and contact state analysis. This processing flow transforms the tactile distribution under different bending postures into a unified reference representation, which can then directly participate in visual fusion and contact state analysis. Based on the above analysis, this step establishes a processing link from joint state to curvature estimation and then to tactile compensation around the tactile deformation error caused by fingertip bending, making the tactile data more consistent in representing the real contact state and pressure distribution, thus reducing the interference of bending posture changes on subsequent slip determination and contact force estimation. It should be understood that the above example is only for demonstration and not a limitation.

[0068] S102: Obtain visual observation data of the hand, encode the visual observation data into a visual feature sequence, encode the compensated tactile tensor into a tactile feature sequence, and perform cross-modal attention fusion using the tactile feature sequence as the query and the visual feature sequence as the key and value to generate cross-modal aligned features.

[0069] In this application, visual observation data is used to provide information on the appearance, location, and spatial structure of the target object and its surrounding environment. The visual feature sequence is a sequence representation formed by encoding the visual observation data, and the tactile feature sequence is a sequence representation formed by encoding the compensated tactile tensor. Cross-modal attention fusion is used to establish a correspondence between local tactile contact information and global visual scene information. Cross-modal alignment features are the joint representation after fusion processing. In specific implementations, the visual observation unit can be located in front of the gripper, at the wrist, or at the end of the robotic arm. In one possible embodiment, the processor acquires visual observation data containing the target object during the current control cycle, preprocesses the visual observation data, and then encodes it. For the compensated tactile tensor, the processor encodes it to extract the corresponding tactile feature sequence. Subsequently, the processor performs cross-modal attention fusion on the visual feature sequence and the tactile feature sequence, using the tactile feature sequence as the query and the visual feature sequence as the key and value, to generate cross-modal alignment features.

[0070] In an exemplary implementation, the above fusion process establishes a correspondence between the contact area and the visual area of ​​the target object, thereby reducing the influence of irrelevant background features and constraining the local tactile interpretation results using visual information. Based on the above analysis, this step encodes both vision and touch, then uses a touch-dominant attention mechanism to achieve modal alignment. This allows the current contact state to directly retrieve relevant regional features from global visual information, thus forming a cross-modal consistent representation for grasping control, thereby improving the stability and scene adaptability of subsequent operation state vector generation. It should be understood that the above example is merely illustrative and not limiting.

[0071] S103: Generate an operation state vector based on cross-modal alignment features, and generate control commands based on sliding speed and contact force information to drive the gripper to perform grasping actions.

[0072] The operation state vector includes information on sliding velocity and contact force.

[0073] In this application, the operation state vector is used to convert the fused perception results into a state representation that the controller can directly call. The sliding velocity characterizes the sliding change of the object relative to the fingertip, the contact force information characterizes the contact force state between the hand and the object, and the control command drives the actuator to output corresponding joint or torque actions. The grasping action corresponds to the execution processes such as hand closing, holding, adjusting, or re-grasping. In specific implementation, the processor generates the operation state vector based on cross-modal alignment features and obtains the estimated sliding velocity and contact force values ​​from it for use in control decisions.

[0074] During the control command generation process, the processor establishes a closed-loop control law for grasping based on the operation state vector. Specifically, when the sliding speed exceeds a preset threshold and the contact force is below the stable grasping range, the processor increases the grip force reference value of the relevant fingers; when the contact force has reached the upper limit range but significant sliding is still detected, the processor adjusts the posture of the relevant fingers; when the contact force suddenly increases and the sliding speed approaches zero, the processor reduces some control outputs to suppress excessive squeezing. The control command can be output to the gripper actuator in the form of one or a combination of joint position increment, joint speed increment, joint current reference value, or joint torque reference value, and the actuator then acts on each motor to execute the grasping action. In an exemplary implementation, the processor generates an adjustment term based on the sliding speed component and an adjustment term based on the contact force error, and superimposes the two onto the basic grasping trajectory to form the final control command. If the controlled object is a multi-finger dexterous hand, the processor can also generate local control components for each finger involved in the contact and coordinate their distribution in the palm coordinate system to keep the overall grasping posture stable.

[0075] Based on the above analysis, this step further maps the cross-modal alignment features into a unified operational state vector containing sliding velocity and contact force information. Control commands are then directly output based on this operational state vector, ensuring continuous updates of the sensing results and the executed actions within the same closed-loop link. Since sliding velocity reflects the dynamic changes in contact stability, and contact force information reflects the mechanical state of the grasping constraint, their joint participation in control allows the gripper to promptly increase or redistribute gripping force when approaching instability and suppress over-grip when contact is too strong, thus ensuring the grasping action remains consistent with the current surface state of the target object. It should be understood that the above example is merely illustrative and not limiting.

[0076] Based on the above analysis, this application provides a hand control method based on visual-tactile integration, including acquiring tactile sensing data and joint state data of the hand; determining the curvature information of the fingertip surface based on the joint state data; and performing deformation compensation on the tactile sensing data based on the curvature information to generate a compensated tactile tensor; acquiring visual observation data of the hand; encoding the visual observation data into a visual feature sequence; encoding the compensated tactile tensor into a tactile feature sequence; and performing cross-modal attention fusion using the tactile feature sequence as a query and the visual feature sequence as a key and value to generate cross-modal alignment features; generating an operation state vector based on the cross-modal alignment features; and generating control commands based on the sliding speed and contact force information to drive the hand to perform grasping actions, wherein the operation state vector includes sliding speed and contact force information. In this application, by introducing joint-state-driven fingertip curvature estimation into the tactile compensation link, and using the compensated tactile representation as the query end in cross-modal attention, a directional alignment relationship is established with the visual global representation facing the contact area. The fusion result is then parsed into sliding velocity and contact force information, and control commands are generated in a closed loop. This allows the hand to maintain high information consistency and control responsiveness even under conditions of visual distortion, partial occlusion, target surface reflection, and tactile shift caused by fingertip bending. Correspondingly, the application uses a unified operational state vector to drive grasping control, ensuring continuity and specificity in sliding recognition, contact force adjustment, and posture correction during the grasping process.

[0077] The vision-touch combined hand control method provided in this application acquires tactile sensing data and joint state data of the hand, and combines the fingertip surface curvature information determined by the joint state to perform deformation compensation on the tactile sensing data, which can reduce the impact of hand posture changes and contact deformation on tactile representation. By encoding visual observation data and compensated tactile data separately, and performing cross-modal attention fusion with tactile features as queries and visual features as keys and values, the alignment degree of vision-touch information in spatiotemporal semantics can be improved. Furthermore, by generating an operation state vector containing sliding speed and contact force information based on the cross-modal alignment features, and generating control commands based on this vector to drive the hand to perform grasping actions, the consistency of state estimation and control accuracy during the grasping process can be improved, and the grasping stability, real-time response capability and cross-scene environment adaptability can be enhanced.

[0078] Figure 2 Flowchart of the vision-touch combined gripper control method provided in this application Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the embodiments, a hand control method based on vision-touch combination is described in detail. The method includes:

[0079] S301. Based on the joint state data, calculate the finger segment posture involved in the contact through positive kinematics to obtain the curvature information of the fingertip surface; generate a tactile mesh warp map based on the curvature information, and use the warp map to record the sampling coordinates corresponding to the position of the compensated tactile mesh in the original tactile array; perform bilinear interpolation resampling on the tactile sensing data based on the warp map to generate the compensated tactile tensor.

[0080] In this scheme, after receiving joint state data, the controller first establishes a finger kinematic model based on the joint angles and link lengths, and obtains the finger segment postures involved in contact through forward kinematics. Then, it calculates curvature information by combining the finger segment postures with the fingertip geometric parameters, which can be preset by the assembly structure of the fingertip silicone layer and the array substrate. Based on the curvature information, a tactile mesh warp mapping table with the same size as the tactile array is generated, and each sampling point in the compensated tactile mesh is mapped to a continuous coordinate position in the original array. Subsequently, for each compensated mesh point, its corresponding four-neighbor sampling value in the original array is read, and the resampling result is calculated according to the bilinear interpolation relationship, outputting the compensated tactile tensor. If a change in joint state causes a curvature update, the tactile mesh warp mapping is reconstructed and resampled, thereby maintaining the consistency between the tactile data and the actual bending shape of the fingertip.

[0081] Through the above processing, the sampling offset of the tactile array under finger bending and pressure conditions can be mapped back to a unified reference coordinate. The compensated tactile tensor has a more consistent spatial correspondence with the real contact area, thus providing a stable input for subsequent tactile feature encoding and cross-modal attention fusion, and improving the accuracy of contact position determination and slip perception.

[0082] For example, the pressure center is calculated based on the compensated tactile tensor, and the slip velocity is calculated based on the difference between the pressure center at the current moment and the previous moment and the time interval.

[0083] In practical implementation, after receiving the compensated tactile tensor, the gripper control processing unit first extracts the normal pressure data and can combine it with tangential pressure data to jointly estimate the pressure center. When the tactile array outputs only a single pressure component, the pressure center can also be calculated based solely on that component. The pressure center can be represented as two-dimensional plane coordinates, denoted as the pressure center coordinates at the current moment and the pressure center coordinates at the previous moment. The gripper control processing unit calculates the displacement based on the coordinate difference between the two coordinates and divides this displacement by the time interval between two samplings to obtain the sliding velocity vector or its scalar amplitude. The time interval can be directly obtained from the sampling period, which can be configured to a fixed value to maintain consistency with the tactile data sampling frequency. In practical applications, this time interval can also be determined by the system timestamp difference.

[0084] In a specific configuration, the pressure center calculation module can use a weighted centroid method to integrate or discretely sum the pressure values ​​in the tactile mesh. The weights are determined by the contact pressure corresponding to each mesh cell, thus outputting the pressure center in continuous coordinate form. The slip velocity calculation module further performs differential calculations on the pressure centers at continuous time points and can apply low-pass filtering or moving average to the differential results to reduce the impact of tactile sampling noise on slip determination. In practical applications, other models of this module can also be selected, and this application does not limit this selection.

[0085] In this way, the compensated tactile tensor can be converted into a pressure center parameter with spatial significance. The temporal change of the pressure center can then directly reflect the offset trend of the contact area, ensuring that the sliding speed matches the actual contact sliding state. Based on this sliding speed participating in the generation of subsequent control commands, the gripper can adjust its grip strength and fingertip posture in a timely manner, thereby improving the accuracy of state perception and the consistency of control response during the grasping process.

[0086] S302. Encode the visual observation data into a visual feature sequence, and encode the compensated tactile tensor into a tactile feature sequence. Use the tactile feature sequence as the query matrix and the visual feature sequence as the key matrix and value matrix. Calculate the product of the query matrix and the key matrix, and multiply it with the value matrix after normalization to obtain the cross-modal alignment features.

[0087] During the calculation, a correlation operation is performed between the query matrix composed of tactile feature sequences and the key matrix composed of visual feature sequences to obtain the matching weights between each tactile position and each visual position. These matching weights are then normalized so that the sum of the weights at different positions is a predetermined constant. Subsequently, the normalized weights are multiplied by the value matrix formed by the visual feature sequences and summed to obtain the cross-modal alignment feature corresponding to the current tactile contact state. This cross-modal alignment feature can characterize the consistency between the tactile contact area and the visual target area, thus providing a unified feature basis for the construction of subsequent operation state vectors.

[0088] In practical implementation, the visual feature sequence can be formed by encoding and stitching together color and depth images separately. The compensated tactile tensor can be composed of multi-channel sampled values ​​from the tactile array. Before entering cross-modal attention fusion, the two can be scaled through a normalization layer. The attention calculation used can be performed sequentially in the processor by a matrix multiplication unit, a normalization unit, and a weighted summation unit, or it can be implemented by a dedicated accelerator with tensor operation capabilities. In practical applications, other models of this component can also be selected, and this application does not limit this.

[0089] In this way, the tactile feature sequence can actively retrieve visual region information corresponding to the contact state and aggregate it into cross-modal aligned features, so that visual and tactile information can be associated in the same representation space, providing consistent input for the extraction of sliding speed and contact force information, thereby improving the stability of the operation state vector and the grasping control accuracy during visual-tactile collaborative control.

[0090] For example, the tactile feature sequence is multiplied by the stored projection weight matrix to obtain the tactile query matrix; the visual feature sequence is multiplied by the stored projection weight matrix to obtain the visual key matrix and the visual value matrix; the product of the tactile query matrix and the visual key matrix is ​​calculated, normalized by scale normalization and normalization exponential function, and then multiplied by the visual value matrix to obtain the cross-modal alignment feature.

[0091] In practical implementation, the hand control processing unit first expands the compensated tactile tensor along the time dimension and inputs it into the tactile encoding network to obtain a tactile feature sequence. This sequence is then linearly mapped to a pre-stored tactile projection weight matrix to form a tactile query matrix. Visual observation data is extracted by the visual encoding network to obtain a visual feature sequence, which is then multiplied by the pre-stored visual projection weight matrix to generate a visual key matrix and a visual value matrix. The projection weight matrix can be composed of fully connected layer parameters, with the parameter dimensions matching the channel dimensions of the tactile and visual feature sequences to ensure that the matrix multiplication results meet the dimensionality requirements of subsequent attention calculations. In practical applications, the projection layer can also employ an equivalent linear transformation structure; this application does not impose any limitations on this approach.

[0092] After completing matrix projection, the hand control processing unit calculates the product of the tactile query matrix and the visual key matrix, and then normalizes the product result according to the feature dimension. Scale normalization can be achieved by scaling with the square root of the feature dimension as the denominator to suppress gradient instability caused by excessively large dot product values. Subsequently, a normalization exponential function is applied to the scaled result to normalize the weights corresponding to each visual position into a probability distribution. This weight is then multiplied by the visual value matrix to obtain the cross-modal alignment feature. This cross-modal alignment feature preserves the visual region responses related to tactile contact and injects the visual context into the tactile representation in a weighted form, thus providing a unified feature basis for the subsequent generation of operation state vectors.

[0093] When this structure is in operation, it uses tactile information as a query to guide the filtering and aggregation of visual features. This gives higher weight to visual regions related to contact location, contact boundary, and target surface morphology, and aggregates their content information into cross-modal aligned features. As a result, the local contact state on the tactile side can establish a correspondence with the overall appearance information on the visual side, thereby outputting a fused representation consistent with the contact scene.

[0094] With this implementation, tactile and visual senses are aligned within the same attention space. Projection mapping and normalized exponential function calculation make weight allocation more stable, and the output features can simultaneously reflect local contact changes and visual environment information, thereby improving the consistency of cross-modal attention fusion results and the accuracy of subsequent control calculations.

[0095] In one optional embodiment, the cross-modal alignment features are decoded by a decoder to generate an intermediate representation containing physical meaning; the three-dimensional position and three-dimensional attitude of the target in the palm coordinate system are parsed from the intermediate representation; the force information along the normal and tangential directions of the fingertip at the contact point is parsed from the intermediate representation, and the force information is determined by the tactile normal pressure integral and the tangential pressure integral; the fingertip curvature, sliding velocity, and friction coefficient estimate at the current moment are parsed from the intermediate representation, wherein the sliding velocity is determined by the pressure center displacement rate of the compensated tactile tensor, and the friction coefficient estimate is determined by the ratio of normal force to tangential force; the contact stiffness estimate, rebound time constant, and state confidence are parsed from the intermediate representation, and the state confidence is determined by a combination of visual sharpness, tactile signal-to-noise ratio, and cross-modal matching weights; and the parsed state components are combined into a unified operational state vector.

[0096] In its implementation, the decoder can employ a multi-layer fully connected network, a recurrent network, or a lightweight Transformer structure. Its input consists of cross-modal alignment features, and its output comprises multiple feature channels corresponding one-to-one with physical quantities. The pose channel in the intermediate representation obtains the target's position vector and attitude parameters in the palm coordinate system through coordinate regression. The force channel maps the normal pressure integral and tangential pressure integral to form normal and tangential forces, respectively. The fingertip curvature channel corresponds to the compensated tactile tensor and is used to characterize the current geometric state of the pressure surface. The pressure center displacement rate channel is obtained by dividing the change in the pressure center in the tactile tensor over consecutive time steps by the time interval. The friction coefficient estimate is calculated from the ratio of the normal force to the tangential force. The state confidence channel integrates visual clarity, tactile signal-to-noise ratio, and cross-modal matching weights to output the confidence level of the current fusion result, which, together with the contact stiffness estimate and rebound time constant, constitutes the operational state vector. The channels can be combined using a normalized and concatenated method to output a unified operational state vector for use by subsequent control modules.

[0097] This structure unifies pose, contact force, sliding motion, and material response into a single state space by decoding cross-modal alignment features into intermediate representations with explicit physical meaning, thereby forming an operational state vector that can be directly used for control decisions. Since state confidence incorporates visual clarity, tactile signal-to-noise ratio, and cross-modal matching weights, the control unit can match subsequent adjustment intensity based on state reliability.

[0098] By adopting this method, the operation state vector simultaneously contains geometric, mechanical, and reliability information, resulting in a more complete representation. Each component has an interpretable physical meaning, which can support contact judgment, slip recognition, and force adjustment during the grasping process, thereby improving the consistency of state estimation and the stability of control output.

[0099] S303. Determine whether the triggering condition is met based on the sliding speed and state confidence in the operation state vector; when the sliding speed is greater than the preset speed threshold or the state confidence is less than the preset confidence threshold, adjust the control parameters to obtain the adjusted control parameters, which include gripping force, stiffness matrix, and damping matrix; update the desired trajectory based on the adjusted control parameters to obtain the updated desired trajectory; generate joint torque commands based on the adjusted control parameters and the updated desired trajectory to drive the gripper to perform the grasping action.

[0100] In this implementation, after receiving the operation state vector, the control unit first extracts the sliding velocity and state confidence, and compares them with preset thresholds. When the sliding velocity exceeds the threshold or the state confidence is lower than the threshold, the control unit performs incremental compensation on the gripping force based on the current contact state, and retunes the stiffness matrix and damping matrix to match the adjusted control parameters with the current grasping stability. Subsequently, the control unit corrects the target motion description of the gripper based on the adjusted control parameters, updates the original desired trajectory to an updated desired trajectory adapted to the current contact state, and calculates the torque distribution required for each joint accordingly. After generating joint torque commands, the commands are output to the actuator to drive the gripper to complete the grasping action.

[0101] The aforementioned joint torque command incorporates the current contact state, parameter correction results, and trajectory error into the control calculation, enabling the gripper to promptly enter an adaptive adjustment state when slippage increases or state confidence is insufficient. Consequently, control parameters are updated according to changes in the grasping state, and the desired trajectory is corrected synchronously, ensuring that the output torque remains consistent with the actual contact requirements, thereby improving control stability and motion consistency during the grasping process.

[0102] For example, the current joint angle and current joint velocity of the gripper are obtained; the desired joint angle and desired joint velocity are obtained according to the desired trajectory; and the joint torque command is calculated based on the difference between the current joint angle and the desired joint angle, the difference between the current joint velocity and the desired joint velocity, the stiffness matrix, the damping matrix, and the contact force information.

[0103] In its implementation, the controller first reads the current joint angle and velocity from the joint encoder and velocity estimation module, and inputs the desired trajectory into the kinematic mapping module to obtain the desired joint angle and velocity corresponding to each joint. Subsequently, the controller calculates the joint angle error and joint velocity error respectively, and combines the error vector with the stiffness matrix, damping matrix, and contact force information to generate the target torque components for each joint. The stiffness matrix is ​​used to adjust the gain of the angle error, the damping matrix is ​​used to suppress the velocity error, and the contact force information is used to correct the torque output after contact with an object, ensuring that the joint torque command balances trajectory tracking and contact stability.

[0104] The joint torque command uses the desired trajectory as a reference and is calculated in real time based on the current joint state and contact force information. This allows the gripper to output corresponding driving torques according to posture deviations and contact states during movement, thereby achieving continuous tracking of joint movement and compliant adjustment during the grasping contact process. By introducing stiffness, damping, and contact force terms into the torque calculation, the joint control output can be promptly corrected according to changes in actual contact.

[0105] By adopting this method, the joint torque command can directly reflect the current motion error and contact state, improve the consistency of the gripper's tracking of the target trajectory, and make the torque adjustment during the contact phase more stable, thereby improving the control accuracy and stability during the grasping process.

[0106] For example, the tactile sensing data includes normal pressure data and tangential pressure data; the visual observation data includes color images and depth images; and the joint state data includes at least one of joint angle, joint velocity, and joint torque.

[0107] Tactile sensing data can be acquired by a tactile array deployed on the surface of the fingertips. The sensing units of the tactile array are arranged in a matrix, and each unit outputs corresponding normal and tangential pressure data. Visual observation data can be acquired by a camera mounted on the outside of the gripper or the robot's end effector. Color and depth images can be obtained by a binocular camera, an RGB (Red Green Blue)-D (Red Green Blue-Depth) camera, or a combination of a color camera and a depth sensor. Joint state data can be acquired by an encoder, a driver feedback unit, or a torque sensing unit, and transmitted in real time to the controller via a bus. For joint angles, joint velocities, and joint torques in the joint state data, the controller can read them synchronously according to the sampling period, or select at least one of them as state inputs according to the specific hardware interface. In practical applications, other models of sensors and acquisition units can also be selected, and this application does not limit this.

[0108] During operation, the system uses normal and tangential pressure data as raw tactile inputs, color and depth images as raw visual inputs, and at least one of joint angles, joint velocities, and joint torques as mechanical state inputs. This supports fingertip curvature compensation, tactile tensor construction, and cross-modal alignment calculations. Since joint states reflect the posture changes of the fingertip after pressure, tactile pressure reflects contact distribution, and visual data provides target appearance and spatial reference, these three types of data can form a complementary relationship within a unified control cycle and further participate in the generation of sliding speed, contact force, and control commands.

[0109] By adopting the above data composition method, tactile input, visual input, and joint state input can respectively correspond to the three types of information sources: contact, appearance, and motion, providing a clear data foundation for subsequent compensation, encoding, and fusion processes. Since tactile data is subdivided into normal pressure data and tangential pressure data, visual data into color images and depth images, and joint state data into joint angles, joint velocities, and joint torques, the system's representation of contact changes, spatial position, and posture changes is more complete, thereby improving the consistency of grasping state estimation and the stability of control output.

[0110] The present invention will be further illustrated below with reference to a specific embodiment. This embodiment takes the process of a dexterous hand with 16 independent degrees of freedom grasping an unknown object as an example to describe in detail the complete implementation process of the hand control method based on vision-touch combination.

[0111] In this embodiment, before performing the grasping task, the dexterous hand needs to stably grasp an unknown object (such as a smooth plastic bottle weighing approximately 0.3 kg) on ​​the worktable. The dexterous hand system is equipped with a palm RGB-D camera (mounted above the palm, with the optical axis at a 15° angle to the palm normal) to acquire visual observation data of the object; a fingertip tactile array (such as SynTouch BioTac, with a resolution of 64×64, fitting against the fingertips of the index, middle, and thumb) to sense the pressure distribution during contact; a joint encoder (with a resolution of at least 14 bits) to collect angle information of each joint; a processor (such as an NVIDIA Jetson AGX Orin) to perform data processing and control decisions; and a low-level impedance controller (such as a Maxon EPOS4 series servo driver with a force control loop frequency of 500 Hz) to drive the motor to perform the grasping action.

[0112] Step 1: Synchronous acquisition and preprocessing of multi-source data;

[0113] First, acquire tactile sensor data of the hand (i.e., normal and tangential pressure data output by the fingertip tactile array). Where t represents the sampling time, c=1 indicates that only the normal pressure channel is included, and c=3 indicates that three channels are included: normal pressure, tangential x-pressure, and tangential y-pressure (the pressure unit is Newtons) and joint state data (i.e., the joint angles output by the joint encoder). Where n is the number of dexterous hand joints, in radians), and simultaneously acquire visual observation data of the hand (i.e., color depth frames output by the palm RGB-D camera). ,in and These represent the image height and width, respectively. The four channels represent the RGB three-channel and the depth channel, with the depth unit being meters.

[0114] Specifically, during the initialization phase, the processor controls the dexterous hand to perform micro-motion scanning, recording the positions of contact points on the haptic array in the haptic array's physical coordinate system {T} and the positions of the same contact points in the camera point cloud in the palm's RGB-D camera coordinate system {C}. This allows the processor to calculate the calibrated extrinsic parameter TTC between the haptic array and the camera. This extrinsic parameter is stored in memory for subsequent contact area trimming.

[0115] During operation, to align the three types of data in time, the FPGA or microcontroller generates a unified hardware synchronization pulse, simultaneously triggering camera exposure, haptic sampling, and encoder latching; each data entry also records a hardware timestamp based on the IEEE 1588 precision time protocol. The processor uses the haptic sampling time as a reference to perform linear interpolation on adjacent RGB-D frames and joint angles to form synchronized data frames. The synchronization error is controlled to be no more than 5 milliseconds.

[0116] By using hardware synchronization pulses and linear interpolation, visual, tactile, and joint state data are unified to the same time base, solving the problem of mismatch between image changes and pressure changes caused by time misalignment when slippage and contact occur at the millisecond level. This provides a time-consistent input for subsequent cross-modal fusion.

[0117] Step 2, fingertip deformation compensation and slippage detection:

[0118] After data synchronization is completed, the curvature information of the fingertip surface is determined based on the joint state data, and the tactile sensing data is deformed and compensated based on the curvature information to generate the compensated tactile tensor.

[0119] Specifically, the processor calculates the finger segment postures involved in the contact based on joint state data using forward kinematics (i.e., based on joint angles). Calculate the pose of each finger segment in space to obtain the curvature information of the fingertip surface. Where m is the number of fingertips or finger segments involved in the compensation. The unit is The value indicates the degree to which the fingertip bends at the current joint angle.

[0120] The processor generates a haptic mesh warp mapping based on curvature information. The warp mapping is used to record the sampling coordinates of the compensated haptic mesh position in the original haptic array. Here, i and j are the row and column indices of the compensated haptic mesh. and The compensated position is in the original tactile array The sampling coordinates to be read are in pixels. This warp map is not a physical quantity to be estimated, but a resampling table calculated by the processor based on the current fingertip curvature and pre-stored haptic array geometry parameters.

[0121] The processor performs bilinear interpolation resampling on the tactile sensing data based on the warp mapping to generate a compensated tactile tensor:

[0122]

[0123] in X is the compensated tactile tensor, where l is the pressure channel number, p and q are the row and column indices of the original tactile array sampling points, and φ(·) is the bilinear interpolation weight. After the above processing, X t The same grid position in the image is closer to the actual contact position of the fingertip surface than the original reading position after bending deformation.

[0124] Furthermore, the processor calculates the pressure center based on the compensated tactile tensor. (Unit: mm), and calculate the slip velocity based on the difference between the pressure center at the current moment and the previous moment, and the time interval:

[0125]

[0126] in The time interval between adjacent haptic frames, in seconds. The value represents the tangential slip velocity, expressed in mm / s. For local slip, the processor can also compare... Compared to the previous frame The local texture displacement is used to obtain pixel-by-pixel tactile optical flow, which is used to determine which contact area experiences micro-slip first.

[0127] By calculating the fingertip curvature using forward kinematics and generating a warping map, the "pressure position offset caused by fingertip bending" and "actual contact slippage" can be distinguished, effectively eliminating the positioning offset caused by tactile skin deformation. At the same time, the slippage trend is detected in real time based on the pressure center displacement rate, providing timely feedback for subsequent control.

[0128] Step 3, Cross-modal attention fusion:

[0129] After obtaining the compensated tactile tensor, the visual observation data is encoded into a sequence of visual features. The compensated tactile tensor is encoded into a tactile feature sequence. Cross-modal attention fusion is performed using tactile feature sequences as queries and visual feature sequences as keys and values ​​to generate cross-modal aligned features.

[0130] Specifically, the processor first uses the calibrated extrinsic parameters TTC and RGB-D depth point cloud as priors, and then processes the compensated haptic tensor... The corresponding tactile contact area is projected into the camera's field of view, and only the visual candidate areas that may be touched are cropped to avoid sending the entire image into the fusion model.

[0131] The visual encoder encodes candidate visual regions into sequences of visual features. The haptic encoder will compensate the haptic tensor Encoded as a tactile feature sequence ;in It represents the number of feature blocks within the visual candidate region. This represents the number of haptic mesh feature blocks, with d=128 being the dimension of each feature block. Here, a feature block is also called a token, which means "a digital description of a local area".

[0132] Using tactile feature sequences as query matrices Using visual feature sequences as key matrices Sum matrix The product of the query matrix and the key matrix is ​​calculated, normalized, and then multiplied by the value matrix to obtain the cross-modal alignment feature.

[0133] Specifically, the processor uses the projection matrices WQ, WK, and WV stored in memory after training to calculate:

[0134]

[0135] in, It is a tactile query matrix, used to represent "what the current tactile contact wants to find in the visual field"; It is a visual key matrix, used to represent "the features that can be matched for each visual region"; It is the visual value matrix, used to provide visual information to be fused after a successful match; WQ, WK and WV are all d×d in size, which are network weights obtained offline and called at fixed times during the runtime, not physical parameters that are resolved at runtime.

[0136] Then, the tactile feature sequence is multiplied by the stored projection weight matrix to obtain the tactile query matrix; the visual feature sequence is multiplied by the stored projection weight matrix to obtain the visual key matrix and the visual value matrix; the product of the tactile query matrix and the visual key matrix is ​​calculated, normalized by scale and normalization exponential function, and then multiplied by the visual value matrix to obtain the cross-modal alignment features:

[0137]

[0138] in It is a cross-modal alignment feature, indicating that each tactile contact block has been fused with information from the visual candidate region most relevant to it; This is used to avoid excessively large dot product values. Output the matching weight of each visual region to the current haptic block.

[0139] By using a cross-modal attention mechanism that uses touch as the query and vision as the key and value, the tactile contact area actively seeks the corresponding visual area. The global pose and 3D contour provided by vision, and the local contact force and sliding trend of tactile feedback are mutually corrected in a unified feature space, which solves the problem that visual images and finger pressure are difficult to correspond in space.

[0140] Step 4: Generate unified operation status:

[0141] After obtaining the cross-modal alignment features, an operational state vector is generated based on the cross-modal alignment features.

[0142] Specifically, the processor decodes the cross-modal aligned features using a decoder to generate an intermediate representation containing physical meaning. Instead of outputting a hidden vector without physical meaning, the decoder outputs a 15-dimensional operational state vector. .

[0143] The processor parses the target's three-dimensional position in the palm coordinate system from the intermediate representation. (Unit: meters; primarily estimated from RGB-D point cloud and contact constraints) and 3D pose. (Using ZYX Euler angles, in radians).

[0144] The processor parses the force information at the contact point along the normal and tangential directions of the fingertip from the intermediate representation. This force information is determined by the tactile normal pressure integral and the tangential pressure integral. It is along the fingertip coordinate system The force in the contact normal direction (in N, obtained by integrating the tactile normal pressure). and It is along Forces in two tangential directions (in N, estimated from three-channel tactile data or changes in the center of pressure).

[0145] The processor parses the fingertip curvature at the current moment from the intermediate representation. (Unit is) (from forward kinematics calculations) and sliding velocity (Unit: mm / s, derived from pressure center displacement rate) and the estimated friction coefficient μ (dimensionless, determined by the ratio of normal force to tangential force).

[0146] The processor parses the contact stiffness estimate kc (in N / mm), rebound time constant τᵣ (in ms), and state confidence ρ∈[0,1] (dimensionless) from the intermediate representation. The state confidence is determined by a combination of visual sharpness, tactile signal-to-noise ratio, and cross-modal matching weights. Among them, kc and τᵣ together serve as material surrogate quantities to determine whether the object is hard, soft, slippery, or easily damaged.

[0147] The processor combines the parsed state components into a unified operational state vector:

[0148]

[0149] Processor judgment Greater than 0.05 mm / s or When the value is less than 0.85, the control parameter adjustment signal is triggered.

[0150] The decoder outputs 15-dimensional operational states with clear physical meaning, rather than uninterpretable black-box features, enabling the output of the fusion model to be directly used by the controller, thus solving the problem of "how the output of the fusion model can be directly used by the controller".

[0151] Step 5: Layered Smoothing Control Execution:

[0152] After generating the operation state vector, control commands are generated based on the sliding speed and contact force information to drive the gripper to perform the grasping action.

[0153] Specifically, the high-level policy network reads the operation state vector s at 30Hz and outputs the expected joint trajectory of the fingertip. ,in It is the joint angle that a dexterous hand is expected to achieve, measured in radians.

[0154] The low-level impedance controller determines whether the triggering condition is met based on the slip velocity and state confidence in the operating state vector. When the slip velocity... If the speed exceeds the preset speed threshold of 0.05 mm / s or the state confidence level is exceeded... When the confidence level is less than the preset confidence threshold of 0.85, the control parameters are adjusted to obtain the adjusted control parameters, which include grip strength, stiffness matrix, and damping matrix.

[0155] The desired trajectory is updated based on the adjusted control parameters to obtain the updated desired trajectory. Then, based on the adjusted control parameters and the updated desired trajectory, joint torque commands are generated to drive the gripper to perform the grasping action.

[0156] Specifically, the low-level impedance controller acquires the current joint angle of the gripper at 500Hz. and current joint velocity Obtain the desired joint angle based on the desired trajectory. and expected joint velocity d; Calculate the joint torque command based on the difference between the current joint angle and the desired joint angle, the difference between the current joint velocity and the desired joint velocity, the stiffness matrix, the damping matrix, and the contact force information:

[0157]

[0158] It should be noted that: , corresponding to joint angular velocity, describes how quickly the joint angle changes over time, and the standard unit is radians per second (rad / s).

[0159] in, It is the joint torque command output to the servo driver, in N·m; Here is the stiffness matrix. The damping matrix is ​​used, and both are adjusted online based on the material surrogate quantities (μ, kc, τᵣ) and the slip velocity in the operation state vector; It is the terminal Jacobian matrix calculated from the kinematic model of the dexterous hand; It is a vector of contact force and torque, which can be obtained by integrating the tactile pressure distribution.

[0160] If a slip or low-confidence event has been triggered, the processor prioritizes updating K, B, and the grip strength target, and then updates the higher-level trajectory. This allows the haptic fast branch to suppress slip first, and the cross-modal branch to correct the target pose, ultimately forming a closed loop of perception and control.

[0161] Through a hierarchical control architecture, the high-level policy network is responsible for trajectory planning, and the low-level impedance controller is responsible for torque execution. The two are connected by a 15-dimensional operational state vector. When the slip speed or confidence triggers the condition, the haptic fast branch responds first to suppress slippage, and the cross-modal branch then corrects the pose, realizing an end-to-end closed loop from perception to control. The end-to-end latency is less than 15ms, which is about 62% lower than that of a pure vision decision-level fusion scheme.

[0162] Step 6, Visual-Tactile Bidirectional Closed-Loop Correction (through steps 1 to 5):

[0163] In this embodiment, vision and touch form a two-way closed-loop correction mechanism throughout the grasping process:

[0164] Before contact, the processor provides spatial anchors for tactile contact candidate areas using visual depth point clouds and calibrated extrinsic parameters (TTC) to avoid positioning offset caused by tactile skin deformation.

[0165] Upon contact, the processor uses the compensated tactile tensor Slip speed The cross-modal matching weights are used to inversely correct visual pose drift under occlusion and close-range defocusing conditions. The processor bases the process on state confidence. The system allocates visual and tactile dominance online based on the degree of occlusion: when visual perception is reliable, it provides global constraints; when visual perception deteriorates, tactile fast branches prioritize stable grasping, which is different from fixed-weight fusion or hard switching after contact.

[0166] Vision and touch are mutually calibrated in a unified operating state, enhancing grasping stability. Under dynamic disturbances (randomly applied external disturbance force not exceeding 3N), the grasping success rate is greater than 95.2%, an improvement of about 17 percentage points compared to the pure vision grasping baseline (approximately 78%); the accuracy rate of unknown object material classification reaches 96.5%, an improvement of about 8.5 percentage points compared to the baseline relying solely on touch (approximately 88%).

[0167] The vision-touch combined hand control method provided in this application acquires tactile sensing data and joint state data of the hand, and combines the fingertip surface curvature information determined by the joint state to perform deformation compensation on the tactile sensing data, which can reduce the impact of hand posture changes and contact deformation on tactile representation. By encoding visual observation data and compensated tactile data separately, and performing cross-modal attention fusion with tactile features as queries and visual features as keys and values, the alignment degree of vision-touch information in spatiotemporal semantics can be improved. Furthermore, by generating an operation state vector containing sliding speed and contact force information based on the cross-modal alignment features, and generating control commands based on this vector to drive the hand to perform grasping actions, the consistency of state estimation and control accuracy during the grasping process can be improved, and the grasping stability, real-time response capability and cross-scene environment adaptability can be enhanced.

[0168] Figure 3 A schematic diagram of the hand control device based on visual-touch interaction provided in this application is shown below. Figure 3 As shown, the hand control device 40 based on visual-touch combination provided in this embodiment includes:

[0169] The acquisition module 401 is used to acquire tactile sensing data and joint state data of the hand, determine the curvature information of the fingertip surface based on the joint state data, and perform deformation compensation on the tactile sensing data based on the curvature information to generate the compensated tactile tensor.

[0170] The fusion module 402 is used to acquire visual observation data of the hand, encode the visual observation data into a visual feature sequence, encode the compensated tactile tensor into a tactile feature sequence, and perform cross-modal attention fusion with the tactile feature sequence as the query and the visual feature sequence as the key and value to generate cross-modal aligned features.

[0171] The drive module 403 is used to generate an operation state vector based on cross-modal alignment features and to generate control commands based on sliding speed and contact force information to drive the gripper to perform grasping actions.

[0172] In one possible embodiment, the acquisition module 401 is used to calculate the posture of the finger segment involved in the contact based on the joint state data through positive kinematics, and obtain the curvature information of the fingertip surface.

[0173] A warping map of the tactile mesh is generated based on the curvature information. The warping map is used to record the sampling coordinates of the compensated tactile mesh position in the original tactile array.

[0174] The tactile sensing data is resampled by bilinear interpolation based on the warp mapping to generate a compensated tactile tensor.

[0175] In one possible embodiment, the acquisition module 401 is used to calculate the pressure center based on the compensated tactile tensor, and to calculate the slip velocity based on the difference between the pressure center at the current moment and the previous moment and the time interval.

[0176] In one possible embodiment, the fusion module 402 is used to encode the visual observation data into a visual feature sequence and to encode the compensated tactile tensor into a tactile feature sequence.

[0177] Using the tactile feature sequence as the query matrix and the visual feature sequence as the key and value matrices, the product of the query matrix and the key matrix is ​​calculated, and after normalization, it is multiplied by the value matrix to obtain the cross-modal alignment features.

[0178] In one possible embodiment, the fusion module 402 is used to multiply the tactile feature sequence with the stored projection weight matrix to obtain a tactile query matrix;

[0179] Multiply the visual feature sequence with the stored projection weight matrix to obtain the visual key matrix and the visual value matrix;

[0180] The product of the tactile query matrix and the visual key matrix is ​​calculated, normalized by scale and normalization exponential function, and then multiplied by the visual value matrix to obtain the cross-modal alignment feature.

[0181] In one possible embodiment, the driving module 403 is configured to decode the cross-modal alignment features using a decoder to generate an intermediate representation containing physical meaning;

[0182] The three-dimensional position and three-dimensional orientation of the target in the palm coordinate system are extracted from the intermediate representation;

[0183] Force information along the normal and tangential directions of the fingertip at the contact point is extracted from the intermediate representation. The force information is determined by the tactile normal pressure integral and the tangential pressure integral.

[0184] The fingertip curvature, sliding velocity, and friction coefficient estimates at the current moment are extracted from the intermediate representation. The sliding velocity is determined by the pressure center displacement rate of the compensated tactile tensor, and the friction coefficient estimates are determined by the ratio of normal force to tangential force.

[0185] The contact stiffness estimate, rebound time constant, and state confidence are extracted from the intermediate representation. The state confidence is determined by a combination of visual clarity, tactile signal-to-noise ratio, and cross-modal matching weights.

[0186] The analytically obtained state components are combined into a unified operational state vector.

[0187] In one possible embodiment, the driving module 403 is used to determine whether the triggering condition is met based on the slip velocity and state confidence in the operation state vector.

[0188] When the sliding speed is greater than the preset speed threshold or the state confidence is less than the preset confidence threshold, the control parameters are adjusted to obtain the adjusted control parameters, which include grip force, stiffness matrix and damping matrix.

[0189] The desired trajectory is updated based on the adjusted control parameters to obtain the updated desired trajectory;

[0190] Based on the adjusted control parameters and the updated desired trajectory, joint torque commands are generated to drive the gripper to perform the grasping action.

[0191] In one possible embodiment, the drive module 403 is used to acquire the current joint angle and current joint speed of the gripper;

[0192] Obtain the desired joint angle and desired joint velocity based on the desired trajectory;

[0193] The joint torque command is calculated based on the difference between the current joint angle and the desired joint angle, the difference between the current joint velocity and the desired joint velocity, the stiffness matrix, the damping matrix, and the contact force information.

[0194] In one possible embodiment, the tactile sensing data in the drive module 403 includes normal pressure data and tangential pressure data; the visual observation data includes color images and depth images; and the joint state data includes at least one of joint angle, joint velocity, and joint torque.

[0195] The hand control device based on visual-touch combination provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0196] Figure 4 This is a schematic diagram of the structure of the hand control device based on vision-touch combination provided in this application. As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 50 also includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0197] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0198] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0199] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0200] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0201] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0202] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0203] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0204] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0205] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0206] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0207] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0208] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0209] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0210] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0211] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A hand gripper control method based on visual-touch integration, characterized in that, include: Acquire tactile sensing data and joint state data of the hand, determine the curvature information of the fingertip surface based on the joint state data, and perform deformation compensation on the tactile sensing data based on the curvature information to generate a compensated tactile tensor. Visual observation data of the hand is acquired, the visual observation data is encoded into a visual feature sequence, the compensated tactile tensor is encoded into a tactile feature sequence, and cross-modal attention fusion is performed using the tactile feature sequence as a query and the visual feature sequence as a key and value to generate cross-modal aligned features. An operation state vector is generated based on the cross-modal alignment features, and control commands are generated based on the sliding velocity and contact force information in the operation state vector to drive the gripper to perform a grasping action.

2. The method according to claim 1, characterized in that, Based on the joint state data, the curvature information of the fingertip surface is determined, and deformation compensation is performed on the tactile sensing data based on the curvature information to generate a compensated tactile tensor, including: Based on the joint state data, the posture of the finger segments involved in the contact is calculated using forward kinematics to obtain the curvature information of the fingertip surface; A tactile mesh warping map is generated based on the curvature information. The warping map is used to record the sampling coordinates of the compensated tactile mesh position in the original tactile array. The tactile sensing data is resampled by bilinear interpolation based on the warp mapping to generate a compensated tactile tensor.

3. The method according to claim 2, characterized in that, Also includes: The pressure center is calculated based on the compensated tactile tensor, and the slip velocity is calculated based on the difference between the pressure center at the current moment and the previous moment and the time interval.

4. The method according to claim 1, characterized in that, The visual observation data is encoded into a visual feature sequence, the compensated tactile tensor is encoded into a tactile feature sequence, and cross-modal attention fusion is performed using the tactile feature sequence as a query and the visual feature sequence as a key and value to generate cross-modal aligned features, including: The visual observation data is encoded into a visual feature sequence, and the compensated tactile tensor is encoded into a tactile feature sequence; Using the tactile feature sequence as the query matrix and the visual feature sequence as the key matrix and value matrix, the product of the query matrix and the key matrix is ​​calculated, and after normalization, it is multiplied by the value matrix to obtain the cross-modal alignment feature.

5. The method according to claim 4, characterized in that, Using the tactile feature sequence as the query matrix and the visual feature sequence as the key and value matrices, the product of the query matrix and the key matrix is ​​calculated, normalized, and then multiplied by the value matrix to obtain cross-modal alignment features, including: The tactile feature sequence is multiplied by the stored projection weight matrix to obtain the tactile query matrix; The visual feature sequence is multiplied by the stored projection weight matrix to obtain the visual key matrix and the visual value matrix. The product of the tactile query matrix and the visual key matrix is ​​calculated, normalized by scale and normalization exponential function, and then multiplied by the visual value matrix to obtain the cross-modal alignment feature.

6. The method according to claim 1, characterized in that, Generate an operation state vector based on the cross-modal alignment features, including: The cross-modal alignment features are decoded by a decoder to generate an intermediate representation containing physical meaning; The three-dimensional position and three-dimensional orientation of the target in the palm coordinate system are extracted from the intermediate representation; Force information along the fingertip normal and tangential directions at the contact point is extracted from the intermediate representation. The force information is determined by the tactile normal pressure integral and the tangential pressure integral. The fingertip curvature, sliding speed, and friction coefficient estimate at the current moment are extracted from the intermediate representation, wherein the sliding speed is determined by the pressure center displacement rate of the compensated tactile tensor, and the friction coefficient estimate is determined by the ratio of normal force to tangential force. The contact stiffness estimate, rebound time constant, and state confidence are extracted from the intermediate representation. The state confidence is determined by a combination of visual clarity, tactile signal-to-noise ratio, and cross-modal matching weights. The analytically obtained state components are combined into a unified operational state vector.

7. The method according to claim 1, characterized in that, Generate control commands based on the sliding speed and the contact force information to drive the gripper to perform a grasping action, including: Determine whether the triggering condition is met based on the slip velocity and state confidence in the operation state vector; When the sliding speed is greater than a preset speed threshold or the state confidence is less than a preset confidence threshold, the control parameters are adjusted to obtain the adjusted control parameters, which include grip force, stiffness matrix and damping matrix. The desired trajectory is updated based on the adjusted control parameters to obtain the updated desired trajectory; Based on the adjusted control parameters and the updated desired trajectory, joint torque commands are generated to drive the gripper to perform a grasping action.

8. The method according to claim 7, characterized in that, Based on the adjusted grip force, stiffness matrix, damping matrix, and desired trajectory, a joint torque command is generated, including: Get the current joint angle and current joint velocity of the gripper; The desired joint angle and desired joint velocity are obtained based on the desired trajectory. The joint torque command is calculated based on the difference between the current joint angle and the desired joint angle, the difference between the current joint velocity and the desired joint velocity, the stiffness matrix, the damping matrix, and the contact force information.

9. The method according to any one of claims 1 to 8, characterized in that, The tactile sensing data includes normal pressure data and tangential pressure data; the visual observation data includes color images and depth images; the joint state data includes at least one of joint angle, joint velocity, and joint torque.

10. A hand control device based on visual-touch integration, characterized in that, include: The acquisition module is used to acquire tactile sensing data and joint state data of the hand, determine the curvature information of the fingertip surface based on the joint state data, and perform deformation compensation on the tactile sensing data based on the curvature information to generate a compensated tactile tensor. The fusion module is used to acquire visual observation data of the hand, encode the visual observation data into a visual feature sequence, encode the compensated tactile tensor into a tactile feature sequence, and perform cross-modal attention fusion using the tactile feature sequence as a query and the visual feature sequence as a key and value to generate cross-modal alignment features. The driving module is used to generate an operation state vector based on the cross-modal alignment features and to generate control commands based on the sliding speed and contact force information to drive the gripper to perform a grasping action.

11. A hand control device based on visual-touch combination, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-9.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-9.

Images