Multi-modal sensor fusion based full area perception dexterous hand

Abstract

The application belongs to the technical field of robot perception and control, and particularly relates to a full-area perception dexterous hand based on multi-modal sensor fusion, which comprises a palm structure, a multi-dimensional force sensing component arranged at the wrist of the palm structure, a palm visual tactile sensor arranged at the palm of the palm structure, a plurality of fingertip visual tactile sensors, the number of which matches the number of fingers of the palm structure, the fingertip visual tactile sensors being arranged at the fingerpads of the fingers of the palm structure, a magnetic flexible tactile skin wrapped outside the palm structure, a control component arranged in the palm structure, the control component being electrically connected with the palm visual tactile sensor, the fingertip visual tactile sensor and the magnetic flexible tactile skin, and the control component being used for controlling the movement of the fingers and receiving feedback data of the palm visual tactile sensor, the fingertip visual tactile sensor and the magnetic flexible tactile skin.

Description

Technical Field

[0001] This invention belongs to the field of robot perception and control technology, and particularly relates to a full-area perception dexterous hand based on multimodal sensor fusion. Background Technology

[0002] Existing dexterous hands suffer from significant deficiencies in multimodal perception capabilities, making them ill-suited for complex operational scenarios. Traditional systems typically employ a discrete tactile sensor layout, such as placing a single-point tactile detection unit only in the fingertip area. This results in an inability to continuously perceive the pressure distribution on the contact surface when grasping curved or flexible objects, easily leading to misjudgments of object slippage or deformation. Furthermore, current technologies rely on a separate design of external vision modules and tactile sensors, resulting in significant mismatches in data acquisition timing and spatial registration. For instance, when rapidly grasping dynamic targets, visual delays and the asynchrony of tactile signals directly cause operational deviations. More critically, existing force feedback systems are mostly limited to three-dimensional force measurement, lacking the synchronous capture of six-dimensional force information, making it difficult to accurately control the direction of force application in precision assembly scenarios. Summary of the Invention

[0003] The purpose of this invention is to overcome the limitations of existing robotic hand sensing capabilities and construct a multimodal sensing fusion-based all-domain sensing system. The palm integrates visual-tactile sensors capable of continuous internal and external sensing, the fingertips are equipped with visual-tactile sensors covered with a highly wear-resistant coating, all other areas are equipped with magnetic flexible tactile sensors, and a six-dimensional force sensor is installed in the wrist, enabling full-area sensing by the dexterous hand. Combined with multimodal sensing data, the robotic hand can accurately grasp precision electronic components in industrial assembly, ensuring efficiency and safety in the assembly process. In medical surgery scenarios, it can sense tissue pressure in real time and dynamically adjust the force applied, effectively avoiding tissue damage and improving the minimally invasiveness and safety of the surgery. The flexible sensors have collision detection and automatic force adjustment functions, and can automatically adjust the joint angle based on tactile feedback, significantly reducing wear and failure rates of mechanical components and extending the overall lifespan of the system. Through its all-domain tactile sensing capabilities, the dexterous hand can monitor potential accidental collisions or pinching risks in real time, and adjust the force instantly based on multimodal data feedback, achieving safety protection and collaborative operation during human-machine interaction, and promoting the widespread application of intelligent robots in complex collaborative environments.

[0004] To achieve the above objectives, the present invention provides the following solution:

[0005] A full-area perception dexterous hand based on multimodal sensor fusion includes:

[0006] Palm structure;

[0007] The palm structure is equipped with a multi-dimensional force sensing component at the wrist.

[0008] A palm visual-tactile sensor is disposed in the palm of the hand structure;

[0009] A plurality of fingertip visual-tactile sensors, the number of which matches the number of fingers in the palm structure, wherein the fingertip visual-tactile sensors are disposed on the fingertips of the fingers in the palm structure;

[0010] Magnetic flexible tactile skin wraps around the outside of the palm structure;

[0011] A control component is disposed within the palm structure, and the control component is electrically connected to the palm visual-tactile sensor, the fingertip visual-tactile sensor, and the magnetic flexible tactile skin;

[0012] The control component is used to control the movement of the finger and to receive feedback data from the palm visual-tactile sensor, the fingertip visual-tactile sensor, and the magnetic flexible tactile skin.

[0013] Optionally, the palm structure includes:

[0014] A palm base, wherein the palm visual-tactile sensor is disposed in the palm of the palm base;

[0015] Five fingers, one end of which is connected to the palm base, and the fingertip visual-tactile sensor is located at the end of the finger away from the palm base;

[0016] A finger drive unit is connected to the finger and is located in the palm base. The finger drive unit is used to drive the movement of the finger.

[0017] Optionally, the finger driving unit includes:

[0018] The motor is fixedly attached to the palm base at its fixed end;

[0019] A composite metal wire, one end of which is connected to the output shaft of the motor, and the other end of which is connected to the finger;

[0020] The motor drives the corresponding finger movement via the composite metal ribbon;

[0021] The motor is electrically connected to the control component.

[0022] Optionally, the finger located at the index finger position is connected to the corresponding motor drive via a double composite metal wire.

[0023] Optionally, the palm base has a cavity in the middle;

[0024] The cavity is used to accommodate the palm visual-touch sensor, the motor, and the control components;

[0025] The cavity is open on one side of the palm.

[0026] The opening is used for mounting the palm visual-tactile sensor, the motor, and the control components;

[0027] The opening is fixed with a highly transparent outer shell.

[0028] Optionally, the palm visual-tactile sensor is equipped with a binocular camera in the middle, which is used to acquire continuous depth information.

[0029] Optionally, the magnetically flexible tactile skin is prepared by mixing 400-mesh neodymium iron boron magnet particles with a silicone solvent. The neodymium iron boron magnet particles are magnetically connected to a Hall sensor array, and the Hall sensor array is fixed to the inner side of the magnetically flexible tactile skin.

[0030] Optionally, the multidimensional force sensing component is a six-dimensional force sensor, which is disposed at the wrist of the palm base.

[0031] Compared with the prior art, the present invention has the following advantages and technical effects:

[0032] Compared to traditional technologies that only set a single-point tactile detection unit in the fingertip area, this invention uses a palm-view tactile sensor in conjunction with multiple fingertip-view tactile sensors to continuously sense the pressure distribution on the contact surface when grasping curved or flexible objects, avoiding misjudgments caused by object slippage or deformation. Through the integrated tactile and visual design of the fingertip-view tactile sensor and the palm-view tactile sensor, the timing and spatial registration of data acquisition are matched, avoiding the direct operational deviation caused by visual delay and tactile signal asynchrony. Furthermore, by setting up a multi-dimensional force sensing component, the direction of force application can be precisely controlled. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of the structure of the present invention;

[0035] Among them, 1. Fingertip visual-tactile sensor; 2. Magnetic flexible tactile skin; 3. Composite metal wire; 4. Motor; 5. Palm visual-tactile sensor; 6. High-transparency shell; 7. Double composite metal wire; 8. Palm base. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] Reference Figure 1 This invention discloses a full-area sensing dexterous hand based on multimodal sensor fusion, comprising:

[0039] Palm structure;

[0040] A multi-dimensional force sensing component is installed at the wrist of the hand structure;

[0041] The palm visual-tactile sensor 5 is installed in the palm of the hand structure;

[0042] A plurality of fingertip visual-tactile sensors 1, the number of which matches the number of fingers in the palm structure, are disposed on the fingertips of the fingers in the palm structure;

[0043] Magnetic flexible tactile skin 2, wrapped around the outer side of the palm structure;

[0044] The control component is located within the palm structure and is electrically connected to the palm visual-tactile sensor 5, the fingertip visual-tactile sensor 1, and the magnetic flexible tactile skin 2.

[0045] The control component is used to control finger movement and receive feedback data from the palm visual-tactile sensor 5, the fingertip visual-tactile sensor 1, and the magnetic flexible tactile skin 2.

[0046] The microcontroller unit (MCU) is responsible for overall control logic operations, data processing, and instruction issuance, and has rich interfaces to support multi-channel sensor data input.

[0047] The multi-channel analog / digital signal acquisition module is used to acquire analog or digital signals from different tactile sensors (palm visual tactile sensor 5, fingertip visual tactile sensor 1, magnetic flexible tactile skin 2) and multi-dimensional force sensing components.

[0048] The communication interface module includes USB (for connection to a host computer), I2C, and SPI interfaces. The palm visual-tactile sensor 5 and fingertip visual-tactile sensor 1 (integrated with a binocular camera) use the SPI interface to ensure independent high-speed communication between multiple fingertip sensors and avoid bus conflicts. The magnetic flexible tactile skin 2 (Hall sensor array) has a large number of sensors and uses the I2C interface, with multiple sensors sharing a bus.

[0049] Compared with traditional technologies, this application adds multiple sensor settings to the palm structure.

[0050] Compared to traditional technologies that only set a single-point tactile detection unit in the fingertip area, this invention uses a palm-view tactile sensor 5 in conjunction with multiple fingertip-view tactile sensors 1 to continuously sense the pressure distribution on the contact surface when grasping curved or flexible objects, avoiding misjudgments caused by object slippage or deformation. Through the integrated tactile and visual design of the fingertip-view tactile sensor 1 and the palm-view tactile sensor 5, the timing of data acquisition and spatial registration are matched, avoiding the direct operational deviation caused by visual delay and tactile signal asynchrony. Furthermore, by setting up a multi-dimensional force sensing component, the direction of force application can be precisely controlled.

[0051] The fingertip visual-tactile sensor 1 is installed at the fingertip of each finger to sense visual and tactile information from the outside world. The sensor integrates a binocular camera to achieve continuous perception of the inside and outside.

[0052] The magnetic flexible tactile skin 2 covers the surface of the fingers and palms. It is made of magnetic flexible material and can simulate the tactile feedback function of the skin. At the same time, it has a certain degree of flexibility and elasticity, which enhances the finger's flexible perception ability.

[0053] As an optional implementation, the hand structure includes:

[0054] The palm base 8 has a palm visual-tactile sensor 5 disposed in the palm of the palm base 8;

[0055] Five fingers, one end of which is connected to the palm base 8, and the fingertip visual-tactile sensor 1 is located at the end of the finger away from the palm base 8;

[0056] The finger drive unit is connected to the finger transmission and is located inside the palm base 8. The finger drive unit is used to drive finger movement.

[0057] As an optional implementation, the finger driving unit includes:

[0058] Motor 4, with its fixed end fixed to the palm base 8;

[0059] The composite metal wire 3 is connected at one end to the output shaft of the motor 4, and at the other end to the finger.

[0060] Motor 4 drives the corresponding finger movement via composite metal wire 3;

[0061] Motor 4 is electrically connected to the control components.

[0062] As an alternative implementation, the finger located at the index finger position is connected to the corresponding motor 4 via a double composite metal wire 7.

[0063] Composite wire 3 serves as a reinforcing material for finger bones and joints, providing mechanical strength and flexibility to support finger movement and structural stability.

[0064] Motor 4 is installed inside the palm, driving the movement of the fingers to achieve the bending and extension of the bionic fingers. The motor is electrically connected to the control unit, receiving control signals to achieve precise control.

[0065] The enhanced index finger mobility design employs a dual composite metal wire 7 structure to improve the index finger's flexibility and range of motion, thereby enhancing the overall hand device's dexterity and realism.

[0066] As an optional implementation, the palm base 8 has a cavity in the middle;

[0067] The cavity is used to house the palm visual-tactile sensor 5, the motor 4, and the control components;

[0068] The cavity is located on one side of the palm and has an opening;

[0069] The opening is for mounting the palm-mounted visual-tactile sensor 5, the motor 4, and the control components;

[0070] It has an open, fixed, highly transparent outer shell 6.

[0071] The highly transparent outer shell protects the internal structure while ensuring the visual sensor's line of sight remains unobstructed. The shell is made of highly transparent plastic, balancing strength and light transmittance.

[0072] As an optional implementation, a binocular camera is provided in the center of the palm visual-tactile sensor 5, which is used to acquire continuous depth information.

[0073] The binocular camera is integrated into the palm-mounted visual-tactile sensor 5. As the core component of visual perception, it can capture high-resolution stereo images and, together with the depth estimation neural network model, achieve high-precision spatial perception.

[0074] As an optional implementation, the magnetically flexible tactile skin 2 is prepared by mixing 400-mesh neodymium iron boron magnet particles with a silicone solvent. The neodymium iron boron magnet particles are magnetically connected to a Hall sensor array, which is fixed to the inside of the magnetically flexible tactile skin 2.

[0075] As an alternative implementation, the multidimensional force sensing component is a six-dimensional force sensor, which is disposed at the wrist of the palm base 8.

[0076] Furthermore, the hand base 8 adopts an anthropomorphic structure enhancement design to simulate the natural movement and structure of the human hand, enhancing the simulation effect and user comfort of the hand device.

[0077] Electrical connection relationships and their effects:

[0078] The fingertip visual-tactile sensor 1 is electrically connected to the motor 4 and the palm visual-tactile sensor 5 via internal circuitry, enabling synchronized control of visual and mechanical movements. The magnetically flexible tactile skin 2 is made of a mixture of 400-mesh neodymium magnet particles and silicone solvent. Beneath the skin is a Hall sensor array connected to the control unit for real-time processing of tactile signals. The motor 4 drives the composite metal wire 3 and a linkage structure to control finger movements. The index finger's dual composite metal wire 7, in conjunction with the motor 4, enables multi-degree-of-freedom flexible movements. The thumb is controlled by an independent motor. Visual data collected by the palm visual-tactile sensor 5 is processed and fed back to the control system, enabling continuous perception and intelligent response to the internal and external environment.

[0079] Specifically, the index finger has a lateral swing degree of freedom, provided by the base joint, with a swing angle ranging from approximately 0 to 20°. This increases the gripping stability of the dexterous hand and its ability to adapt to different objects. In contrast, the middle, ring, and little fingers have no lateral swing degree of freedom, only three joint degrees of freedom (proximal phalanx, middle phalanx, and distal phalanx), and these joint axes are arranged in parallel.

[0080] The index finger lateral swing base joint is located inside the palm plate (palm base). The lateral swing drive component is long and narrow, with the lateral swing axis perpendicular to the palm base and intersecting the direction of index finger extension.

[0081] The two composite metal wires in the dual-tendon cord design extend from the motor output to different phalanges or joints of the index finger via different paths, controlling flexion and lateral swing movements respectively. The mechanical transmission paths of the index finger's lateral swing base joint and the dual-tendon cords do not interfere with each other. The driving force for the lateral swing movement is transmitted through a rotating drive component connected to the lateral swing drive assembly, via a meshing mechanism, enabling the index finger to rotate around the lateral swing axis. One of the tendon cords is mechanically connected to this rotating drive component to complete the lateral swing drive. The other tendon cord is responsible for the standard flexion and extension movements of the index finger; the two tendon cords work together to achieve multi-degree-of-freedom movement of the index finger.

[0082] The thumb also has a degree of lateral freedom, but the rotation axis of its base joint is arranged vertically, and the angle between the thumb and the palm is about 45°, which facilitates precise grasping of objects.

[0083] Knuckle length and modular design:

[0084] Except for the thumb, the three phalanges of the four fingers are exactly the same size (distal phalanx, middle phalanx, and proximal phalanx). The middle, ring, and little fingers have identical structures, while the tendon ligament connection structure of the index finger is slightly different. The phalanx length of the thumb varies slightly.

[0085] Cone-driven design:

[0086] The index finger is driven by two tendon cords (bimetallic wires), one being the active cord and the other the connecting cord, which work together to achieve more flexible movement control. The other fingers are mainly driven by a single tendon cord.

[0087] Specifically, the output end of motor 4 is connected to the worm gear transmission mechanism to ensure smooth power transmission and a high reduction ratio. The worm gear transmission mechanism is connected to the output shaft of motor 4, providing a high reduction ratio to amplify motion torque and achieve high-precision position control. Composite metal wire serves as a transmission reinforcement material, connecting the worm gear drive mechanism to the finger structure.

[0088] Specifically, the output end of motor 4 is connected to the worm shaft, the worm meshes with the turbine, and the composite metal wire is wound around the turbine. By rotating the turbine, the composite metal wire is wound around the turbine, thus achieving finger bending.

[0089] Motor 4 is preferably a coreless motor.

[0090] Finger reset is achieved through a torsion spring structure set at the hinge point of the linkage structure.

[0091] The hand structure is no different from that of traditional bionic robotic hands, which is existing technology in this field. This application is not about improving the hand structure, so it will not be described in detail here.

[0092] Through the above structural design and electrical connection, the hand device of the present invention can not only achieve accurate visual and tactile perception, but also provide natural and flexible finger movements through humanoid structure, making it suitable for various application scenarios such as robots and prostheses.

[0093] The overall structure of this invention consists of a palm body, five fingers, and a wrist transmission mechanism. The mechanical part includes a palm base 8 and five fingers. Each finger includes a fingertip visual-tactile sensor 1 and a finger joint structure. Each finger joint is connected to a motor 4 via a composite metal wire 3 and a linkage mechanism to realize the driving and movement of the finger joints.

[0094] The palm base 8 is equipped with a palm visual-tactile sensor 5 and its highly transparent outer shell 6, forming a continuous visual-tactile sensing unit.

[0095] Among them, the index finger and the palm base 8 are designed to enhance mobility, that is, the double composite metal wire 7 is used to connect each finger joint with the linkage mechanism and the motor 4, so as to achieve flexible connection and structural stability.

[0096] The thumb and upper palm feature an enhanced, humanoid structure design to improve overall grip flexibility and aesthetics.

[0097] The specific explanation is as follows:

[0098] The dexterous hand integrates a palm visual-tactile sensor 5 that can continuously sense both inside and outside: the palm visual-tactile sensor 5 is embedded inside the palm base 8.

[0099] The palm-mounted visual-tactile sensor 5 includes a transparent outer layer, which preferably consists of an acrylic sheet, a thick silicone layer, an invisible fluorescent marker layer, and a thin silicone layer, forming a continuous force-sensitive interface from the inside out. The highly transparent outer layer enables the sensor to perceive external objects. The palm-mounted visual-tactile sensor 5 incorporates a high-resolution binocular camera, fixed in the palm area, ensuring that the field of view covers the entire internal space of the palm, thus forming a continuous visual perception capability.

[0100] Meanwhile, the palm-mounted visual-tactile sensor 5 incorporates a UV LED, which is positioned close to the acrylic plate and forms a rectangular ring around the entire edge of the acrylic plate. The color development of the marking layer is controlled by switching the UV LED on and off.

[0101] The palm-based visual-tactile sensor 5 acquires continuous depth and tactile information based on the principle of binocular depth estimation. It consists of an elastomer formed by a thick silicone layer, an invisible fluorescent marker layer, and a thin silicone layer. When the elastomer is pressed, an internal 360nm wavelength ultraviolet lamp is energized, turning the transparent markers red. By calculating the displacement of the markers during the deformation of the elastomer, contact force data is obtained, enabling real-time 3D reconstruction of the contact area between the object and the palm during grasping.

[0102] Specifically, marker displacement is achieved by capturing images using a stereo camera and then calibrating the displacement using computer vision methods. First, a segmentation model is used to obtain marker masks, and the Hough circle method is used to calculate the marker center. Marker displacement is represented by the difference in marker center coordinates between different frames. Stereo matching is performed on the marker image pairs captured by the stereo camera, and depth estimation, i.e., surface depth reconstruction, is completed through disparity calculation and pre-calibrated intrinsic parameters. Contact force data is obtained by training a neural network method to determine the mapping relationship between surface deformation and contact force.

[0103] Each fingertip is fitted with a fingertip visual-tactile sensor 1 covered with a highly wear-resistant, opaque coating. This coating is prepared using water transfer printing and a wire drawing process to improve mechanical durability and the accuracy of the marking points.

[0104] The fingertip visual-tactile sensor 1 integrates a monocular camera and a three-color light strip, and uses photometric stereo technology to achieve high-precision three-dimensional reconstruction to obtain delicate tactile information.

[0105] By analyzing the displacement of marker points caused by the deformation of the elastic body, continuous sliding detection of the fingertip is achieved, accurately reflecting minute changes during the contact process.

[0106] The fingertip visual-tactile sensor 1 works by analyzing the changes in image brightness of an object under different lighting conditions to deduce the object's three-dimensional shape. The core principle of photometric stereochemistry is to infer the three-dimensional shape of an object by utilizing the changes in brightness of its surface under different lighting conditions.

[0107] The general idea of ​​photometric stereo method is as follows: when the relative positions of the camera and the target object are fixed, the same target object is illuminated by light sources from different directions. The camera can capture images of the target object with different brightness and darkness distributions (at least three images are required). Then, by solving the reflection equations based on the Lambert reflection principle, the normal distribution or albedo diagram of the target surface can be obtained.

[0108] A magnetically flexible tactile skin 2 is deployed on the sidewalls of the knuckles and in the non-contact areas of the palm. Its flexible design mimics the softness of human skin, improving the safety of human-computer interaction. The magnetically flexible tactile skin 2 is prepared by mixing 400-mesh neodymium iron boron magnet particles with a silicone solvent. When an object comes into contact with the magnetically flexible tactile skin, the neodymium iron boron magnet particles shift, and the magnetic field changes with the displacement. The dexterous hand system acquires real-time information on the three-dimensional magnetic field changes through a dense array of Hall sensors distributed under the magnetically flexible tactile skin. Based on a pre-trained network, it calculates the contact force, maps the global contact state, and solves the blind spot problem of traditional single-point tactile detection.

[0109] In terms of electromechanical integration, the internal optical and electronic components of the sensor are fixed by a customized connecting plate and signal lines to ensure stable transmission of image signals. The magnetically flexible tactile skin 2 covers the sidewalls of the knuckles and the non-contact area of ​​the palm. It is prepared by mixing 400-mesh NdFeB and neodymium iron boron magnet particles with a silicone solvent, and embeds a Hall sensor array to detect changes in the three-dimensional magnetic field in real time. The Hall sensor array circuit is connected to the main control circuit board via flexible cables. After amplification and filtering, the signal is input to the processing unit, which uses a pre-trained neural network model to calculate the contact force in real time, achieving full-area tactile coverage and avoiding the blind spots of traditional single-point tactile detection.

[0110] Magnetic skin is a research direction in the field of flexible electronic skin. A complete magnetic skin consists of two modules: a magnetic elastomer and a magnetic sensor. When the magnetic elastomer undergoes elastic deformation due to pressure, the orientation and direction of the permanent magnet particles within the elastomer change, resulting in a change in the magnetic field. This change in magnetic field is recorded by a Hall effect sensor. The contact force data is obtained by training a neural network method to map the three-dimensional magnetic field change to the contact force.

[0111] A six-dimensional force sensor is installed at the end of the wrist transmission mechanism. The six-dimensional force sensor is rigidly connected to the wrist transmission shaft through a flange. It can simultaneously measure the X / Y / Z three-axis contact forces Fx, Fy, Fz and the axial torques Tx, Ty, Tz. The data is output to the motion controller after being filtered by Kalman.

[0112] Each component works collaboratively through a multi-layer signal fusion architecture: the vision-based tactile perception scheme aligns and fuses tactile data with visual data, then the pressure distribution and six-dimensional force information are associated through a contact mechanics model, and finally, an attention mechanism neural network is used to adaptively weight and fuse multi-source signals.

[0113] Compared with traditional technologies, this invention has the following advantages:

[0114] 1. The palm-mounted visual-tactile sensor 5 achieves unobstructed penetration and continuous acquisition of visual signals by covering the palm surface with a multi-layered, highly transparent elastomer and combining it with a built-in high-resolution binocular camera. This design enables the system to acquire continuous depth information in real time and accurately calculate contact force through the deformation and displacement of invisible fluorescent markers, thus reconstructing the three-dimensional force field of the contact area. Therefore, a single sensor can achieve a high degree of fusion of vision, touch, and tactile sensation, significantly improving the continuity and accuracy of perception.

[0115] 2. The fingertip-mounted tactile sensor 1 employs a monocular camera in conjunction with three-color LED strips. Based on the photometric stereo method, it performs 3D surface reconstruction to acquire high-precision tactile information. A highly wear-resistant, opaque coating covering the fingertip enhances durability and marker recognition accuracy through water transfer printing and a brushing process. This design enables continuous and detailed perception of fingertip tactile sensation. By analyzing the marker displacement caused by elastic body deformation, it accurately detects slippage changes on the fingertip surface, effectively overcoming the detection blind spots caused by the limited installation space of traditional tactile probes at the fingertip.

[0116] 3. A magnetically flexible tactile skin 2 is arranged on the sidewalls of the knuckles and in the non-contact areas of the palm. This skin, combined with a neural network algorithm based on the magnetic field signal of a Hall effect sensor array, enables contact sensing and force mapping. This design ensures the freedom and flexibility of the finger joints while effectively eliminating the detection blind spots of traditional single-point tactile sensors in areas such as the sidewalls of the knuckles. It achieves comprehensive coverage of tactile information over a large area, improving the safety and comfort of the overall human-computer interaction.

[0117] 4. The six-dimensional force sensor is rigidly connected to the wrist drive shaft via a flange, enabling simultaneous measurement of three-axis spatial contact force and axial torque. Combining multi-axis mechanical sensing with interference filtering technologies such as Kalman filtering, the system provides the dexterous hand with high-precision, multi-dimensional force interaction closed-loop feedback, significantly enhancing the robot's adaptability and stability in complex operating environments.

[0118] 5. A perception scheme based on visual and tactile fusion is adopted, achieving spatial-temporal alignment of tactile and visual data through a precise time synchronization mechanism. Utilizing a contact mechanics model, pressure distribution information is deeply integrated with six-dimensional force sensor data to form a comprehensive environmental perception framework. Based on this, an adaptive weighting strategy is introduced, combining the characteristics of multi-source signals to further optimize the information fusion process. This strategy significantly improves the system's responsiveness to environmental changes and its understanding of object states, thereby enhancing the robustness and accuracy of grasping and manipulation decisions, ensuring the robot's efficient operation in complex environments.

[0119] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this invention, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0120] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A dexterous hand based on multimodal sensor fusion for full-area perception, characterized in that, include: Palm structure; The hand structure is equipped with a multi-dimensional force sensing component at the wrist. A palm visual-tactile sensor (5) is disposed in the palm of the hand structure; A plurality of fingertip visual-tactile sensors (1) are provided, the number of which matches the number of fingers in the palm structure, and the fingertip visual-tactile sensors (1) are disposed on the fingertips of the fingers in the palm structure; Magnetic flexible tactile skin (2) is wrapped around the outside of the palm structure; A control component is disposed within the palm structure, and the control component is electrically connected to the palm visual-tactile sensor (5), the fingertip visual-tactile sensor (1), and the magnetic flexible tactile skin (2); The control component is used to control the movement of the fingers and to receive feedback data from the palm visual-tactile sensor (5), the fingertip visual-tactile sensor (1), and the magnetic flexible tactile skin (2).

2. The all-area perception dexterous hand based on multimodal sensor fusion according to claim 1, characterized in that, The palm structure includes: Palm base (8), wherein the palm visual-touch sensor (5) is disposed in the palm of the palm base (8); Five fingers, one end of which is connected to the palm base (8), and the fingertip visual-tactile sensor (1) is located at the end of the finger away from the palm base (8); A finger drive unit is connected to the finger drive unit and is located inside the palm base (8). The finger drive unit is used to drive the finger movement.

3. The all-area perception dexterous hand based on multimodal sensor fusion according to claim 2, characterized in that, The finger driving unit includes: The motor (4) is fixedly connected to the palm base (8) at its fixed end; The composite metal wire (3) is connected at one end to the output shaft of the motor (4) and at the other end to the finger; The motor (4) drives the corresponding finger to move via the composite metal wire (3); The motor (4) is electrically connected to the control component.

4. The all-area perception dexterous hand based on multimodal sensor fusion according to claim 3, characterized in that: The finger located at the index finger position is connected to the corresponding motor (4) via a double composite metal wire (7).

5. The all-area perception dexterous hand based on multimodal sensor fusion according to claim 3, characterized in that: The palm base (8) has a cavity in the middle; The cavity is used to accommodate the palm visual-tactile sensor (5), the motor (4), and the control components; The cavity is open on one side of the palm. The opening is used for mounting the palm visual-tactile sensor (5), the motor (4), and the control components; The opening is fixed to a highly transparent outer shell (6).

6. The all-area perception dexterous hand based on multimodal sensor fusion according to claim 1, characterized in that: The palm visual-tactile sensor (5) is equipped with a binocular camera in the middle, which is used to acquire continuous depth information.

7. The all-area perception dexterous hand based on multimodal sensor fusion according to claim 1, characterized in that: The magnetically flexible tactile skin (2) is prepared by mixing 400-mesh neodymium iron boron magnet particles with silicone solvent. The neodymium iron boron magnet particles are magnetically connected to a Hall sensor array, and the Hall sensor array is fixed to the inside of the magnetically flexible tactile skin (2).

8. The all-area perception dexterous hand based on multimodal sensor fusion according to claim 2, characterized in that: The multidimensional force sensing component is a six-dimensional force sensor, which is located at the wrist of the palm base (8).

Images