Multi-modal sensing method integrating joint-fingertip-contact target

Abstract

The invention discloses an integrated joint-fingertip-contact target multi-mode sensing method, and relates to the field of robot sensing. In order to solve the defects that in the prior art, a sensor is high in integration cost, complex in structure and incomprehensive in sensing information, and unified multi-mode sensing of a joint-fingertip-target cannot be achieved, the technical scheme provided by the invention comprises the following steps of: obtaining the displacement of a rope and the deformation of a spring; calculating a joint angle according to the rope displacement; calculating a joint driving torque according to the spring deformation amount; calculating a fingertip position according to the joint angle; reversely deducing fingertip contact force according to the joint driving torque; estimating target softness according to the contact force and the displacement; and recording a fingertip trajectory according to the contact state to identify a target contour. The method is suitable for multi-mode sensing work of the rope-driven dexterous hand in a complex operation task, and is especially suitable for application scenes of flexible grabbing, touch recognition and interaction control of a target object.

Description

Technical Field

[0001] The present invention relates to the field of robot perception, and specifically to a multimodal perception algorithm integrating joints, fingertips and contact targets. Background Art

[0002] With the continuous development of robotics, a robot's ability to perceive its environment has become a key enabler for accomplishing complex tasks. In particular, in high-precision scenarios such as service robotics, medical rehabilitation, intelligent manufacturing, and aerospace operations, anthropomorphic dexterous hands are gradually replacing traditional rigid grippers as end effectors. Their flexible, multi-degree-of-freedom structure enables more detailed grasping, manipulation, and interaction tasks. However, achieving dexterous manipulation of objects requires more than just mechanical structure; it relies on precise perception of both the robot's own state (such as joint angles and contact forces) and external objects (such as shape and softness).

[0003] Currently, mainstream solutions typically acquire this information by installing angle encoders and torque sensors at each joint of the robot hand, as well as high-precision tactile sensors at the fingertips. For example, Shadow Robot's dexterous hand system installs multiple sets of angle sensors on each knuckle and uses multimodal tactile sensors such as BioTac to sense pressure, vibration, and temperature. However, this approach presents the following problems: High cost: High-precision force and tactile sensors are expensive, and their number increases significantly with the number of degrees of freedom; Complex integration: The wiring of multiple sensors is complex and takes up a lot of space, which seriously restricts the miniaturization design of the robot; Lack of reliability: The sensor is easily affected by noise or damaged, and the maintenance cost is high; Difficulty in information integration: Due to the strong heterogeneity of different sensors, the fusion processing algorithm is complex and real-time performance is difficult to guarantee.

[0004] To simplify the structure of perception systems, some studies have attempted to use flexible sensing methods such as soft sensors, piezoresistive skin, and fiber optic sensor arrays for force and position measurement. However, most of these solutions are still limited to a single information modality, such as measuring only touch pressure or using them only for surface recognition. This makes it difficult to achieve multimodal unified perception of joints, fingertips, and external targets. At the same time, some studies have attempted to estimate some motion states through visual information, such as tracking joint motion trajectories with cameras. However, these methods are often limited to large-scale, low-precision posture recognition and are unable to obtain complex information such as fine-grained fingertip force and contact contours.

[0005] In summary, the existing technology has the defects of high sensor integration cost, complex structure, incomplete perception information, and inability to achieve unified multimodal perception of joints-fingertips-targets. Summary of the Invention

[0006] To address the shortcomings of the prior art, such as high sensor integration cost, complex structure, incomplete sensor information, and inability to achieve unified multimodal perception of joints, fingertips, and targets, the present invention provides the following technical solutions: A multimodal perception method integrating joints, fingertips and contact targets, comprising: Steps for obtaining rope displacement and spring deformation; Steps for calculating joint angles based on rope displacement; Steps for calculating joint driving torque based on spring deformation; Steps to calculate fingertip position based on joint angles; The steps of inferring the fingertip contact force according to the joint driving torque; Steps for estimating target softness based on contact force and displacement; The steps of recording the fingertip trajectory and identifying the target contour according to the contact state.

[0007] Furthermore, a preferred embodiment is provided in which the step of obtaining the rope displacement and the spring deformation includes the step of measuring the rope starting point position and the spring end point position by a camera.

[0008] Furthermore, a preferred embodiment is provided in which the step of calculating the joint angle according to the cable displacement includes the step of establishing a polynomial fitting relationship between the cable length change and each joint angle.

[0009] Furthermore, a preferred embodiment is provided in which the step of calculating the joint driving torque according to the spring deformation includes the step of converting the joint torque according to the spring stiffness and tension.

[0010] Furthermore, a preferred embodiment is provided in which the step of calculating the fingertip position according to the joint angle includes the step of performing forward kinematics derivation based on the DH parameter model.

[0011] Furthermore, a preferred embodiment is provided in which the step of inversely deriving the fingertip contact force based on the joint driving torque includes the step of using the Jacobian matrix pseudo-inverse to derive the contact force.

[0012] A multimodal sensing device integrating joints, fingertips and contact targets is also provided, comprising: Module to obtain rope displacement and spring deformation; Module for calculating joint angles based on rope displacements; A module that calculates the joint driving torque based on the spring deformation; A module that calculates fingertip positions based on joint angles; A module that infers fingertip contact force based on joint driving torque; A module for estimating target softness based on contact forces and displacements; A module that records the fingertip trajectory and identifies the target contour based on the contact state.

[0013] A computer storage medium is also provided for storing a computer program, and when the computer program is read by a computer, the computer executes the method.

[0014] A computer is also provided, comprising a processor and a storage medium, wherein when the processor reads a computer program stored in the storage medium, the computer executes the method.

[0015] A computer program product is also provided, which is a computer program that implements the method when the computer program is executed.

[0016] Compared with the prior art, the technical solution provided by the present invention is beneficial in that: This solution uses a vision system within the base to measure the displacement of the rope's starting point, replacing traditional angle encoders for acquiring joint angles. This avoids the high cost and structural complexity associated with installing encoders at each joint. Compared to existing solutions that rely on numerous hardware sensors, this visual perception approach offers a simpler structure and significantly reduces the complexity of system integration.

[0017] This solution indirectly measures the joint's driving torque by measuring the deformation of a tension spring connected in series with a cable, eliminating the need for a traditional standalone torque sensor. Compared to solutions that only measure cable tension, this approach, combined with a cable torque transmission model and a visual system, enables precise estimation of the driving torque, improving the accuracy of force perception.

[0018] By using the DH parameter model to model the kinematics of each finger, this solution can calculate the three-dimensional spatial position of the fingertip based solely on visually acquired rope length changes, replacing the independent measurement of each joint position. Compared to existing methods that rely on high-precision displacement sensors, this approach significantly reduces hardware dependence and provides more flexible position recognition.

[0019] For contact force estimation, this solution uses the pseudo-inverse of the Jacobian matrix combined with external torques to estimate fingertip contact forces, eliminating the need for expensive tactile sensors installed on the fingertips. This algorithm can infer fingertip forces based on acquired joint external torques, enabling computational acquisition of fine-grained tactile data.

[0020] This approach controls a single cable to drive a fingertip to press against a target surface, and combines visually measured tendon displacement and spring compression to infer the object's softness. Compared to traditional approaches that only sense contact or approximate pressure, this method can quantify the softness of the target, enhancing the robot's ability to determine object properties.

[0021] For target contour recognition, this solution treats fingertips as tactile sensors, detecting whether the fingertips are in contact (contact force greater than 0) and recording their position in real time to obtain the target surface contour. Compared to existing approaches that use high-density tactile arrays to sense surface topography, this solution is not only more cost-effective but also offers spatial scalability and dynamic recognition capabilities while maintaining tactile functionality.

[0022] It is suitable for multimodal perception of rope-driven dexterous hands in complex operation tasks, and is particularly suitable for application scenarios of flexible grasping, tactile recognition and interactive control of target objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the vision-based rope-driven hand structure; Figure 2 This is a schematic diagram of the ring finger structure; Figure 3 Schematic diagram of the DH coordinate system of the index finger. DETAILED DESCRIPTION

[0024] In order to make the advantages and benefits of the technical solution provided by the present invention more clearly reflected, the technical solution provided by the present invention is now further described in detail with reference to the accompanying drawings, specifically: Embodiment 1: This embodiment provides a multimodal perception method integrating joints, fingertips, and contact targets, including: Steps for obtaining rope displacement and spring deformation; Steps for calculating joint angles based on rope displacement; Steps for calculating joint driving torque based on spring deformation; Steps to calculate fingertip position based on joint angles; The steps of inferring the fingertip contact force according to the joint driving torque; Steps for estimating target softness based on contact force and displacement; The steps of recording the fingertip trajectory and identifying the target contour according to the contact state.

[0025] The step of obtaining the rope displacement and the spring deformation includes the step of measuring the rope starting point position and the spring end point position by a camera.

[0026] The step of calculating the joint angle according to the cable displacement includes the step of establishing a polynomial fitting relationship between the cable length change and each joint angle.

[0027] The step of calculating the joint driving torque according to the spring deformation includes the step of converting the joint torque according to the spring stiffness and tension.

[0028] The step of calculating the fingertip position according to the joint angle includes the step of performing forward kinematics derivation based on the DH parameter model.

[0029] The step of inverting the fingertip contact force according to the joint driving torque includes the step of deriving the contact force using the Jacobian matrix pseudo-inverse.

[0030] Implementation Method 2: This implementation method further describes the technical solution provided in Implementation Method 1 in detail. Specifically: A vision-based multimodal perception method is suitable for integrated perception scenarios of multi-degree-of-freedom, cable-driven dexterous hands. By integrating a camera module within the base and combining it with spring components and kinematic modeling, it achieves unified perception of multiple dimensions of information about the manipulator, including six modalities: joint angles, joint torques, fingertip position, contact force, contact target softness, and surface contour.

[0031] This implementation is performed as follows: The first step is to build and calibrate the visual system in the base: The camera is mounted in the center of the manipulator base, with a fixed viewing angle directed toward all rope starting points and spring terminals. The camera utilizes an industrial-grade high-speed vision module, combined with structured light-assisted illumination, to enhance image resolution and contour clarity. A one-time calibration establishes a mapping between image coordinates and actual physical displacements, ensuring high accuracy and robustness in subsequent displacement measurements.

[0032] The second step is to obtain the change in rope length and calculate the joint angle: Each finger of the robotic arm is driven by multiple rope segments. A camera captures the position of each rope's starting point in real time, calculating its displacement relative to its static initial position to determine the current rope elongation. Because the ropes are connected to the joints via a pulley system, and some joints are structurally coupled, a pre-fitted geometric model allows for a functional relationship between joint angle and rope length. To account for the elastic deformation of the rope during tensioning, a polynomial correction compensation function is used to improve the accuracy of angle calculations.

[0033] The third step is to measure the spring deformation and infer the rope tension: Each rope is connected in series to a linear tension spring, the other end of which is fixed to the base frame. The camera detects changes in the spring endpoints in the image and, combined with calibration data, calculates the spring compression (or extension). Based on the known spring stiffness, the current rope tension is calculated. This tension is applied to each joint through the transmission structure, generating a driving torque that provides the basis for joint load sensing.

[0034] The fourth step is to model the fingertip position based on kinematics: Taking the index finger as an example, a DH parameter model of the four joint segments is established based on its internal structure, and a step-by-step coordinate transformation matrix is ​​constructed with the base as the reference system. The joint angles calculated in the second step are substituted into the kinematic chain for a forward kinematic solution, ultimately outputting the absolute position of the fingertip in three-dimensional space. Similar modeling methods can be used for the remaining fingers. This result is not only used for spatial positioning but also lays the foundation for contact detection and contour recognition.

[0035] The fifth step is to calculate the fingertip contact force using Jacobian inversion: When a finger contacts an external object, an external counter-torque is generated at the joint. This torque is calculated as the sum of the rope tension and the return spring torque. This torque is combined with the Jacobian matrix obtained in step 4, and pseudo-inverse inversion is used to determine the magnitude and direction of the contact force applied by the fingertip on the target object at that moment. This method avoids the installation of traditional tactile sensors and offers the advantages of low cost, high precision, and high integration.

[0036] Step 6: Derivation of target softness based on displacement and force: While controlling a single rope to drive a fingertip toward a target surface, the rope displacement and corresponding applied force are recorded from the start of compression to maximum displacement. Because the softness of the target material affects its compression response under the same force, the slope of the force-position curve can be used to calculate a softness index. This method is adaptable to surfaces of varying materials and does not require a pre-established standard database.

[0037] Step 7: Record the fingertip trajectory and reconstruct the target contour: When the robot arm makes contact with the target, the system records the three-dimensional fingertip position of each finger while in contact. By controlling the robotic arm to slowly move the robot arm horizontally or along a curved surface, the fingertips continuously slide while in contact, collecting trajectory points. Ultimately, these contact points are combined to form the geometric outline of the target surface, enabling contour recognition. Compared to dense tactile arrays, this approach offers greater scalability and dynamic tracking capabilities.

[0038] The overall structure of the multimodal perception system is as follows: Robot body: The device consists of five fingers, each composed of multiple joints connected to a cable-driven system via pulleys, resulting in a high degree of freedom and biomimetic structural characteristics. The finger structure, as shown in the figure, exhibits a typical three-joint tandem design with compact fingertips.

[0039] Rope drive system: All cables originate from fixed positions within the manipulator's base and terminate at corresponding finger joints. These cables are routed through pulleys using a specific routing scheme to form a closed-loop control system. Each cable is connected in series with a linear spring to sense its tension and cushion impact forces.

[0040] Spring assembly: The springs are arranged in parallel within the base, providing both structural support and tension sensing. The camera primarily measures spring tip displacement, and their simple construction facilitates replacement and adjustment.

[0041] Pedestal Camera System: Installed in the center of the manipulator base, it uses an industrial camera and image processing algorithms to identify changes in the rope endpoints and spring ends. It has high frame rate and high resolution capabilities and can obtain deformation parameters in real time.

[0042] Visual image processing and modeling system: The embedded control module receives image input, extracts key parameters, and performs a series of calculations such as joint angle analysis, kinematic derivation, torque inversion and flexibility analysis to form a complete multimodal perception closed-loop system.

[0043] Implementation Method 3: Combination Figure 1-3 This embodiment further describes the above technical solution in detail through specific examples, specifically: Suitable for rope-driven manipulators, where all rope starting point displacements and spring deformations can be measured by vision inside the base, such as Figure 1 shown.

[0044] The implemented multimodal perception algorithm includes joint angle and torque perception, fingertip position and contact force perception, and target tactile information (softness, contour) perception. The following is the specific algorithm implementation: (1) Joint angle perception: For a rope-driven joint, the joint angle There is an obvious geometric relationship with the change of rope length l. Taking the ring finger as an example, Figure 1 As shown, the active joint and The length of the connected rope The relationship between (1) in represents the pulley radius on the i-th joint.

[0045] Since there is a 1:1 coupling structure design between joint 3 and joint 2, , so the relationship between the total joint angle of the ring finger and the change in rope length can be expressed as: (2) Since the vision inside the base can only measure the displacement m of the starting point of the rope, the rope will inevitably deform during the stretching process. , so the actual change in the length of the rope that controls the joint angle is . The function can be obtained using a cubic polynomial fit.

[0046] Will Substituting into formula (2) we can get the angles of all finger joints Rope displacement with visual measurement Direct relationship: (3) (2) Joint torque perception: Since the rope is connected in series with the tension spring, it can be deformed by the spring Indirect measurement of rope tension Where k is the stiffness of the tension spring. The driving torque generated by the cable tension on the joint is It can be expressed as (4) Since the finger joint is reset by the force of the return spring, the restoring torque on the joint is Expressed as (5) in Indicates the stiffness of the return spring.

[0047] When the fingers interact with the external environment (such as grasping, pressing), the external torque on the joints Expressed as (6) (3) Fingertip position perception The fingertip positions of the thumb, index finger, middle finger, ring finger and little finger of the dexterous hand are defined as , , , , The DH parameter method is used to model the finger kinematics. Taking the index finger as an example, the DH coordinate system is established as follows: Figure 3 shown.

[0048] According to the established coordinate system and geometric parameters, the DH parameter table of the index finger can be obtained, as shown in Table 1.

[0049]

[0050] According to the transformation relationship of the coordinate systems of adjacent phalanges, the coordinate system transformation matrix of adjacent phalanges can be obtained as shown in formula (7).

[0051] (7) First, this embodiment performs forward kinematic derivation of the thumb, index finger, and middle finger. All transformation matrices are multiplied together through product operations to ultimately obtain the homogeneous transformation matrix of the finger tips, as shown in Equation (8).

[0052] (8) According to formula (8), the fingertip end position can be calculated by the joint angle. Finally, the fingertip position of the index finger is obtained. As shown in formula (9).

[0053] (9) Where, , , , , , , .

[0054] (4) Fingertip contact force perception According to the principle of virtual work, the fingertip contact force External torque of the joint The relationship is ,in represents the Jacobian matrix, as shown in formula (10), Represents the pseudoinverse of the transpose of the Jacobian matrix.

[0055] (10) (5) Target softness perception The robot hand quantifies the softness of different objects by measuring the difference in compression displacement. The principle is that when the fingertip applies the same force to surfaces of different softness, the surfaces will produce different compression displacements due to the difference in material softness. When the fingertip is controlled by a single rope, the time point of contact detection is defined as This embodiment defines that the force applied by the fingertip is positively correlated with the deformation of the spring after contact, which is expressed as The fingertip displacement under pressure is positively correlated with the tendon displacement after contact, which is denoted as Therefore, the softness of an object can be defined as . and It can be obtained by visual measurement inside the base. In the process of controlling a single rope to drive the fingertips to press the target, the softness of the target can be obtained. .

[0056] (6) Target contour recognition The fingertip position and contact force information can be substituted into the above model using visual measurement information, so the fingertip can be used as a tactile sensor to identify the object's contour. The following are the specific steps for contour recognition: (1) Install the manipulator at the end of the robotic arm and align the coordinate systems of the robotic arm and manipulator to the world coordinate system.

[0057] (2) Place the surface of the object to be detected directly under the palm of your hand to ensure that the finger can touch the object when bent.

[0058] (3) Control the finger joints of the manipulator to gradually bend. If the fingertip contact force is greater than 0, the surface is in contact with the object at this time, and the fingertip position of each finger in the world coordinate system is recorded at this time; (4) Control the robotic arm to move slowly in the horizontal direction. The fingertips of the robotic arm will continue to contact the surface of the object (if the contact force is greater than 0). The position trajectory of the fingertips during the movement is recorded, which is the contour of the object surface.

[0059] Compared with the prior art, this embodiment has the following beneficial effects: This embodiment uses the displacement of the rope starting point measured by visual measurement inside the base and spring deformation Combined with the robot's kinematic model, this system integrates six modal sensing modes: joint angle, joint torque, fingertip position, fingertip force, target softness, and target contour. This significantly reduces the number and cost of angle encoders, torque sensors, and tactile sensors in the manipulator, significantly reducing the manipulator's manufacturing and design costs (wiring, communication, and control of multiple sensors). Compared to similar force-position fusion measurement devices that can only measure rope tension and joint angle, this invention integrates multimodal information perception. Furthermore, this invention takes into account the rope's own deformation during actuation, resulting in higher measurement accuracy than similar measurement devices.

[0060] The above further describes the technical solution provided by the present invention in detail through several specific embodiments in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the several specific embodiments described above are not intended to limit the present invention. Any reasonable modification and improvement of the present invention, combination of embodiments and equivalent replacement based on the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal perception method integrating joints, fingertips and contact targets, characterized in that: include: Steps for obtaining rope displacement and spring deformation; Steps for calculating joint angles based on rope displacement; Steps for calculating joint driving torque based on spring deformation; Steps to calculate fingertip position based on joint angles; The steps of inferring the fingertip contact force according to the joint driving torque; Steps for estimating target softness based on contact force and displacement; The steps of recording the fingertip trajectory and identifying the target contour according to the contact state.

2. The multimodal perception method integrating joints, fingertips and contact targets according to claim 1, characterized in that: The step of obtaining the rope displacement and the spring deformation includes the step of measuring the rope starting point position and the spring end point position by a camera.

3. The multimodal perception method integrating joints, fingertips and contact targets according to claim 1, characterized in that: The step of calculating the joint angle according to the cable displacement includes the step of establishing a polynomial fitting relationship between the cable length change and each joint angle.

4. The multimodal perception method integrating joints, fingertips and contact targets according to claim 1, characterized in that: The step of calculating the joint driving torque according to the spring deformation includes the step of converting the joint torque according to the spring stiffness and tension.

5. The multimodal perception method integrating joints, fingertips and contact targets according to claim 1, characterized in that: The step of calculating the fingertip position according to the joint angle includes the step of performing forward kinematics derivation based on the DH parameter model.

6. The multimodal perception method integrating joints, fingertips and contact targets according to claim 1, characterized in that: The step of inverting the fingertip contact force according to the joint driving torque includes the step of deriving the contact force using the Jacobian matrix pseudo-inverse.

7. A multimodal sensing device integrating joints, fingertips and contact targets, characterized in that: include: Module to obtain rope displacement and spring deformation; Module for calculating joint angles based on rope displacements; A module that calculates the joint driving torque based on the spring deformation; A module that calculates fingertip positions based on joint angles; A module that infers fingertip contact force based on joint driving torque; A module for estimating target softness based on contact forces and displacements; A module that records the fingertip trajectory and identifies the target contour based on the contact state.

8. A computer storage medium for storing a computer program, characterized in that When the computer program is read by a computer, the computer executes the method according to claim 1 .

9. A computer comprising a processor and a storage medium, characterized in that When the processor reads the computer program stored in the storage medium, the computer executes the method according to claim 1 .

10. A computer program product, being a computer program, characterized in that When the computer program is executed, the method according to claim 1 is implemented.

Images