Full-region sensing dexterous hand based on multi-mode sensing fusion

Abstract

The invention belongs to the technical field of robot sensing and control, and particularly relates to a full-area sensing dexterous hand based on multi-mode sensing fusion. A multi-dimensional force sensing assembly is arranged at the wrist of the palm structure; the palm center visual touch sensor is arranged in the palm center of the palm structure; the number of the fingertip vision tactile sensors is matched with that of the fingers of the palm structure, and the fingertip vision tactile sensors are arranged at the finger pulps of the fingers of the palm structure; the magnetic flexible touch skin wraps the outer side of the palm structure; the control assembly is arranged in the palm structure, and the control assembly is electrically connected with the palm center visual tactile sensor, the fingertip visual tactile sensor and the magnetic flexible tactile skin; the control assembly is used for controlling finger movement and receiving feedback data of the palm center visual tactile sensor, the fingertip visual tactile sensor and the magnetic flexible tactile skin.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of robot perception and control, and particularly relates to a full-area perception dexterous hand based on multi-modal sensor fusion. BACKGROUND

[0002] The existing dexterous hand has significant defects in multi-modal perception ability, and is difficult to adapt to complex operation scenes. Traditional systems generally adopt a discrete tactile sensor layout, such as a single-point tactile detection unit arranged only in the finger pulp area, which causes the inability to continuously perceive the contact surface pressure distribution when grabbing curved or flexible material objects, and is prone to cause object slip or deformation misjudgment. In addition, the existing technology relies on a separate design of an external vision module and a tactile sensor, and the mismatch problem of the two in data acquisition timing and spatial registration is prominent. For example, when grabbing dynamic targets quickly, the visual delay and tactile signal asynchrony will directly cause operation deviation. More importantly, the existing force feedback system is mostly limited to three-dimensional force measurement, lacking the synchronous capture of six-dimensional force information, making it difficult to accurately control the force direction in the precision assembly scene. SUMMARY

[0003] The purpose of the present application is to break through the limitations of the existing mechanical hand perception ability and construct a full-area perception system based on multi-modal sensor fusion. The palm is integrated with visual-tactile sensors that can perceive continuously inside and outside, the fingertips are installed with visual-tactile sensors covered with high-wear-resistant coatings, and the other areas are all arranged with magnetic flexible tactile sensors, and a six-dimensional force sensor is installed at the wrist, which can realize the full-area perception of the dexterous hand. Combined with multi-modal sensor data, the mechanical hand can realize accurate grabbing of precision electronic components in industrial assembly, ensuring the efficiency and safety of the assembly process. In the medical surgery scene, the tissue pressure is perceived in real time and the operation force is adjusted dynamically, effectively avoiding damage to the tissue, and improving the minimally invasive and safety of the surgery. The flexible sensor has collision detection and force automatic adjustment functions, which can automatically adjust the joint angle according to the tactile feedback, significantly reducing the wear and failure rate of mechanical parts, and prolonging the service life of the whole system. Through the full-area tactile perception ability, the dexterous hand can monitor potential accidental collision or pinch risk in real time, and adjust the force in real time combined with multi-modal data feedback, realize safety protection and collaborative operation in the human-machine interaction process, and promote the wide application of intelligent robots in complex collaborative environments.

[0004] To achieve the above purpose, the present application provides the following scheme:

[0005] The full-area perception dexterous hand based on multi-modal sensor fusion comprises:

[0006] A palm structure;

[0007] A multi-dimensional force sensing component is arranged at the wrist of the palm structure;

[0008] A palm visual tactile sensor is arranged at the palm of the palm structure.

[0009] A plurality of fingertip visual tactile sensors are arranged at the palm of the palm structure.

[0010] A magnetic flexible tactile skin is arranged on the outside of the palm structure.

[0011] A control assembly is arranged in the palm structure, and the control assembly is electrically connected with the palm visual tactile sensor, the fingertip visual tactile sensor, and the magnetic flexible tactile skin.

[0012] The control assembly is configured to control the movement of the fingers and receive feedback data from the palm visual tactile sensor, the fingertip visual tactile sensor, and the magnetic flexible tactile skin.

[0013] Optionally, the palm structure comprises:

[0014] A palm base, wherein the palm visual tactile sensor is arranged at the palm of the palm base.

[0015] Five fingers, one end of which is connected to the palm base, and the fingertip visual tactile sensor is arranged at the end of the fingers away from the palm base.

[0016] A finger driving part is arranged in the palm base and is in transmission connection with the fingers, and the finger driving part is configured to drive the movement of the fingers.

[0017] Optionally, the finger driving part comprises:

[0018] A motor, a fixed end of which is fixedly connected to the palm base.

[0019] A composite wire, one end of which is connected to the output shaft of the motor, and the other end of the composite wire is connected to the fingers.

[0020] The motor drives the corresponding fingers to move through the composite wire.

[0021] The motor is electrically connected with the control assembly.

[0022] Optionally, the fingers at the index finger position are in transmission connection with the corresponding motor through double composite wires.

[0023] Optionally, the palm base is provided with a cavity in the middle part.

[0024] The cavity is configured to accommodate the palm visual tactile sensor, the motor, and the control assembly.

[0025] The cavity is located on the palm side and is provided with an opening;

[0026] The opening is used for mounting the palm visual tactile sensor, the motor and the control assembly;

[0027] The opening is fixed with a high-transparency shell.

[0028] Optionally, the palm visual tactile sensor is provided with a binocular camera in the middle, and the binocular camera is used to acquire continuous depth information.

[0029] Optionally, the magnetic flexible tactile skin is prepared by mixing 400-mesh ferrous boron magnet particles with a silica gel solvent, the ferrous boron magnet particles are magnetically connected with a Hall sensor array, and the Hall sensor array is fixed on the inner side of the magnetic flexible tactile skin.

[0030] Optionally, the multi-dimensional force sensing assembly is a six-dimensional force sensor, and the six-dimensional force sensor is arranged at the wrist of the palm base body.

[0031] Compared with the prior art, the present application has the following advantages and technical effects:

[0032] Compared with the prior art, the present application has the following advantages and technical effects: Compared with the prior art, the present application has the following advantages and technical effects:

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor:

[0034] Figure 1 The structure of the present application is shown in the figure;

[0035] Among them, 1, fingertip visual tactile sensor; 2, magnetic flexible tactile skin; 3, composite wire; 4, motor; 5, palm visual tactile sensor; 6, high-transparency shell; 7, double composite wire; 8, palm base body. DETAILED DESCRIPTION

[0036] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0037] In order to make the above objectives, characteristics and advantages of the present application more apparent, clear and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0038] With reference to Figure 1 The present application discloses a full-area perception dexterous hand based on multi-modal sensor fusion, comprising:

[0039] a palm structure;

[0040] a multi-dimensional force sensing component is arranged at the wrist of the palm structure;

[0041] a palm visual tactile sensor 5 is arranged at the palm of the palm structure;

[0042] a plurality of fingertip visual tactile sensors 1 are arranged at the fingerpads of the fingers of the palm structure, and the number of the fingertip visual tactile sensors 1 matches the number of the fingers of the palm structure;

[0043] a magnetic flexible tactile skin 2 is wrapped outside the palm structure;

[0044] a control component is arranged in the palm structure, and the control component is electrically connected with the palm visual tactile sensor 5, the fingertip visual tactile sensor 1 and the magnetic flexible tactile skin 2;

[0045] The control component is used for controlling the movement of the fingers and receiving feedback data of the palm visual tactile sensor 5, the fingertip visual tactile sensor 1 and the magnetic flexible tactile skin 2.

[0046] A microcontroller unit (MCU) is responsible for overall control logic operation, data processing and instruction issuing, and has rich interface support for multi-channel sensor data input.

[0047] A multi-channel analog / digital signal acquisition module is used for acquiring analog or digital signals from different tactile sensors (the palm visual tactile sensor 5, the fingertip visual tactile sensor 1 and the magnetic flexible tactile skin 2) and the multi-dimensional force sensing component.

[0048] The communication interface module includes a USB interface (for connection with an upper computer), an I2C and an SPI interface type. The palm visual tactile sensor 5 and the fingertip visual tactile sensor 1 (an integrated binocular camera) use the SPI interface, which ensures independent high-speed communication of multiple fingertip sensors and avoids bus conflicts. The magnetic flexible tactile skin 2 (a Hall sensor array) has a large number of sensors, and the I2C interface is used for multiple sensors to share a bus.

[0049] Compared with the conventional technology, the application is characterized in that multiple sensors are arranged on the palm structure.

[0050] Compared with the conventional technology in which only a single-point tactile detection unit is arranged in the finger pad area, the application can realize continuous sensing of the contact surface pressure distribution when a curved or flexible material object is gripped, avoid misjudgment of object slipping or deformation, and match the data acquisition timing and spatial registration through the tactile visual integrated design of the fingertip visual tactile sensor 1 and the palm visual tactile sensor 5, so as to avoid the operation deviation caused by the asynchrony of visual delay and tactile signals. In addition, the multi-dimensional force sensing component can accurately control the force direction.

[0051] The fingertip visual tactile sensor 1 is installed at the fingertip of each finger and is used for sensing visual and tactile information. The sensor is internally integrated with a binocular camera to realize continuous sensing inside and outside.

[0052] The magnetic flexible tactile skin 2 is covered on the surface of the fingers and the palm and is made of magnetic flexible material, which can simulate the tactile feedback function of the skin and has certain flexibility and stretchability, thereby enhancing the flexible sensing ability of the fingers.

[0053] As an optional embodiment, the palm structure includes:

[0054] The palm base body 8, the palm visual tactile sensor 5 is arranged on the palm of the palm base body 8;

[0055] Five fingers, one end of which is connected with the palm base body 8, and the fingertip visual tactile sensor 1 is located at the end of the finger away from the palm base body 8;

[0056] The finger driving part is connected with the finger transmission, and the finger driving part is located in the palm base body 8. The finger driving part is used for driving the finger to move.

[0057] As an optional embodiment, the finger driving part includes:

[0058] The motor 4 is fixedly connected to the palm base body 8;

[0059] The composite wire 3 is connected with the output shaft of the motor 4 at one end, and the other end of the composite wire 3 is connected with the finger;

[0060] The motor 4 drives the corresponding finger through the composite wire 3;

[0061] The motor 4 is electrically connected with the control assembly.

[0062] As an optional embodiment, the finger at the index finger position is connected with the corresponding motor 4 through double composite wires 7.

[0063] The composite wire 3 serves as the reinforcing material for the finger bones and joints, providing mechanical strength and flexibility to support the movement and structural stability of the fingers.

[0064] The motor 4 is installed inside the palm to drive the movement of the fingers, realizing the bending and stretching of the bionic fingers. The motor is connected with the control unit through electrical connection to receive control signals for precise control.

[0065] The index finger mobility enhancement design adopts a double composite wire 7 structure to improve the flexibility and range of motion of the index finger, and to improve the flexibility and simulation of the overall hand device.

[0066] As an optional embodiment, a cavity is provided in the middle of the palm base 8;

[0067] The cavity is used to accommodate the palm visual tactile sensor 5, the motor 4, and the control assembly;

[0068] The cavity is provided with an open mouth on one side of the palm;

[0069] The open mouth is used for the installation of the palm visual tactile sensor 5, the motor 4, and the control assembly;

[0070] The open mouth is fixedly connected with a high-transparency shell 6.

[0071] The high-transparency shell 6 protects the internal structure while ensuring that the visual sensor's line of sight is not blocked. The shell material is high-transparency plastic, taking into account strength and light transmission.

[0072] As an optional embodiment, the palm visual tactile sensor 5 is provided with a binocular camera in the middle, which is used to obtain continuous depth information.

[0073] The binocular camera is integrated into the palm visual tactile sensor 5, serving as the core component of visual perception, capable of capturing high-resolution stereoscopic images, and cooperating with a depth estimation neural network model to realize high-precision spatial perception.

[0074] As an optional embodiment, the magnetic flexible tactile skin 2 is prepared by mixing 400-mesh ferrous boron magnet particles with a silicone solvent. The ferrous boron magnet particles are magnetically connected with a Hall sensor array, which is fixedly connected inside the magnetic flexible tactile skin 2.

[0075] As an optional embodiment, the multi-dimensional force sensing assembly is a six-dimensional force sensor, which is arranged at the wrist of the palm base 8.

[0076] Further, the palm base 8 adopts a human-imitating structure to enhance the design for simulating the natural movement and structure of a human hand, thereby enhancing the simulation effect and use comfort of the hand device.

[0077] The electrical connection relationship and the effect:

[0078] The fingertip visual tactile sensor 1 is electrically connected with the motor 4 and the palm visual tactile sensor 5 through internal circuits, so as to realize the synchronous control of vision and mechanical action. The magnetic flexible tactile skin 2 is made of 400-mesh neodymium magnet particles mixed with a silica gel solvent, and a Hall sensor array is arranged below the skin and connected with a control unit to process the tactile signals in real time. The motor 4 drives the composite metal wire 3 and the connecting rod structure to control the finger movement, and the double composite metal wire 7 of the index finger is driven by the motor 4 to realize multi-degree-of-freedom flexible movement. The thumb is controlled by an independent motor. The visual data collected by the palm visual tactile sensor 5 is fed back to the control system after processing, so as to realize the continuous perception and intelligent response of the internal and external environment.

[0079] Specifically, the index finger has a side swing degree of freedom, and the degree of freedom is provided by the base joint, and the side swing angle range is about 0-20°, which increases the gripping stability and the ability to adapt to different objects of the dexterous hand. In contrast, the middle finger, the ring finger and the little finger have only three joint degrees of freedom (proximal phalange, middle phalange and distal phalange), and the joint axes are arranged in parallel.

[0080] The index finger side swing base joint is located inside the palm plate (palm base), the side swing driving assembly is in a strip shape, the side swing axis is perpendicular to the palm base plate, and intersects with the extension direction of the index finger.

[0081] The two composite metal wires designed by the double tendon rope can respectively extend from the motor output end to different phalanges or joints of the index finger through different paths, and control the bending and side swing actions respectively. The mechanical transmission path of the index finger side swing base joint and the double tendon rope does not interfere with each other. The driving force of the side swing action can be transmitted through the meshing member by the rotary driving member connected to the side swing driving assembly, so as to realize the rotation of the index finger around the side swing axis. One of the two tendon ropes can be mechanically connected with the rotary driving member to complete the side swing driving. The other tendon rope is responsible for the standard bending action of the index finger, and the two tendon ropes cooperate to realize the multi-degree-of-freedom movement of the index finger.

[0082] The thumb also has a side swing degree of freedom, but the rotary axes of the base joints thereof are arranged vertically, and the included angle between the thumb and the palm is about 45°, which facilitates the accurate grasping of objects.

[0083] Phalange length and modular design:

[0084] Except for the thumb, the three phalangeal segments of the four fingers are of the same size (distal phalanx, middle phalanx, proximal phalanx). The middle finger, the ring finger and the little finger are of the same structure, and the tendon connection structure of the index finger is slightly different. The lengths of the phalangeal segments of the thumb are slightly different.

[0085] Tendon-driven design:

[0086] The index finger is driven by two tendons (double metal wires), namely a driving tendon and a connecting tendon, which are used in cooperation to achieve more flexible motion control. The other fingers are mainly driven by a single tendon.

[0087] Specifically, the output end of the motor 4 is connected with the worm gear transmission mechanism to ensure smooth power transmission and high speed reduction ratio. The worm gear transmission mechanism is connected to the output shaft of the motor 4 to provide high speed reduction ratio, realize motion torque amplification and high precision of position control. The composite metal wire is used as a transmission reinforcing material to connect the worm gear drive mechanism and the finger structure.

[0088] Specifically, the output end of the motor 4 is connected with the worm shaft, the worm is engaged with the turbine, and the composite metal wire is wound on the turbine. By rotating the turbine, the composite metal wire is wound on the turbine to realize the bending of the finger.

[0089] The motor 4 is preferably a hollow cup motor.

[0090] The finger reset is realized by the torsional spring structure arranged at the hinge point of the connecting rod structure.

[0091] The palm structure is the same as the traditional bionic mechanical hand structure, which is the prior art in the field. The present application is not intended to improve the palm structure, so this will not be described here.

[0092] Through the above structure design and electrical connection, the hand device of the present application not only can realize accurate visual tactile perception, but also can provide natural and flexible finger movement through the humanoid structure, and is suitable for various application scenarios such as robots, prostheses, etc.

[0093] The overall structure of the present application is composed of a palm body, five fingers and a wrist transmission mechanism. The mechanical part includes a palm base 8 and five fingers, the fingers including a fingertip visual tactile sensor 1 and a phalangeal structure, each phalangeal segment being connected with a motor 4 through a composite metal wire 3 and a connecting rod mechanism to realize the driving and movement of the finger joint.

[0094] The palm base 8 is internally provided with a palm heart visual tactile sensor 5 and a high-transparency shell 6 to form an internal and external continuous visual tactile sensing unit.

[0095] Among them, the index finger and the palm base 8 are connected through a movability enhanced design, that is, a double composite metal wire 7 is used to realize the connection of each phalangeal segment with the connecting rod mechanism and the motor 4, realizing flexible connection and stable structure.

[0096] The thumb part and the upper part of the palm are provided with a human-imitating structure reinforcing design, which improves the overall gripping flexibility and aesthetic level.

[0097] The specific description is as follows:

[0098] The dexterous hand is integrated with a palm visual tactile sensor 5 capable of continuous internal and external perception: the palm visual tactile sensor 5 is embedded inside the palm base 8.

[0099] The palm visual tactile sensor 5 comprises a transparent outer layer, which is preferably an acrylic plate, a thick silica gel layer, a layer of invisible fluorescent marker points and a thin silica gel layer from inside to outside, forming a continuous force-sensitive interface. The outer layer with extremely high transparency enables the sensor to have external perception capability. The palm visual tactile sensor 5 is embedded with a high-resolution binocular camera, which is fixed in the palm area, ensuring that the visual field covers the entire internal space of the palm, forming continuous internal and external visual perception capability.

[0100] Meanwhile, the palm visual tactile sensor 5 is embedded with ultraviolet lamp beads, which are close to the acrylic plate and form a rectangular ring around the entire edge of the acrylic plate. Whether the marker point layer is colored or not is controlled by controlling the switch of the ultraviolet lamp beads.

[0101] The palm visual tactile sensor 5 obtains continuous depth information and tactile information based on the binocular depth estimation principle. The elastic body is composed of the thick silica gel layer, the layer of invisible fluorescent marker points and the thin silica gel layer. When the elastic body is pressed, the internal 360 nanometer nm wavelength ultraviolet light belt is powered on, the transparent marker points become red, and the contact force data is calculated through the displacement of the marker points during the deformation of the elastic body, realizing real-time three-dimensional reconstruction of the contact area between the object and the palm during the grabbing process.

[0102] Specifically, the marker point displacement is captured by the binocular camera, and then the displacement is calibrated by computer vision method. First, a segmentation model is used to obtain a marker point mask, and a Hough circle method is used to calculate the center of the marker point. The marker point displacement is represented as the difference in marker point center coordinates between different frames. The marker point image pair captured by the binocular camera is stereo matched, and the depth estimation is completed through parallax calculation and pre-calibrated intrinsic parameters, i.e. surface depth reconstruction. The contact force data is obtained from the mapping relationship between surface deformation and contact force trained by the neural network method.

[0103] Each fingertip is provided with a fingertip visual tactile sensor 1 covered with a high-wear-resistant opaque coating. The coating is prepared by water transfer printing marker points and wire drawing process, which improves the mechanical durability and marker point recognition accuracy.

[0104] The fingertip visual tactile sensor 1 is internally integrated with a monocular camera and a three-color light-emitting lamp belt, which realizes high-precision three-dimensional reconstruction by using Photometric Stereo technology, and obtains delicate tactile information.

[0105] The fingertip visual tactile sensor 1 works by analyzing the image brightness changes of the object under different lighting conditions to calculate the three-dimensional shape of the object. The core principle of photometric stereo is to use the brightness changes of the object surface under different lighting conditions to infer its three-dimensional shape.

[0106] The working principle of the fingertip visual tactile sensor 1 is to analyze the image brightness changes of the object under different lighting conditions to calculate the three-dimensional shape of the object. The core principle of photometric stereo is to use the brightness changes of the object surface under different lighting conditions to infer its three-dimensional shape.

[0107] The general idea of photometric stereo is as follows: when the relative position between the camera and the target object is fixed, different directions of light are used to illuminate the same target object, and the camera can capture images of the target object with different brightness distributions (at least three images are needed). Then, by solving the reflection equation set based on the Lambertian reflection principle, the normal distribution or albedo map of the target surface is obtained.

[0108] The magnetic flexible tactile skin 2 is arranged on the knuckle side wall and the non-contact area of the palm, which simulates human soft skin through flexible design and improves the safety of human-computer interaction. The magnetic flexible tactile skin 2 is prepared by mixing 400-mesh ferritic boron magnet particles with silicone solvent. When an object contacts the magnetic flexible tactile skin, the ferritic boron magnet particles displace, and the magnetic field changes with displacement. The dexterous hand system obtains real-time change information of the three-dimensional magnetic field through the dense array of Hall sensors under the magnetic flexible tactile skin, calculates the contact force according to the pre-trained network, maps the global contact state, and solves the blind area problem of traditional single-point tactile detection.

[0109] In terms of electromechanical combination, the internal optical elements and electronic elements of the sensor are fixed through custom connection boards and signal lines to ensure stable transmission of image signals. The magnetic flexible tactile skin 2 is covered on the knuckle side wall and the non-contact area of the palm, which is prepared by mixing 400-mesh ferritic boron NdFeB, neodymium iron boron magnet particles with silicone solvent, and embedded with a Hall sensor array to detect real-time three-dimensional magnetic field changes. The Hall sensor array circuit is connected to the main control circuit board through flexible flat cables, and the signal is input into the processing unit after amplification and filtering. The pre-trained neural network model is used to calculate the contact force in real time, realize global tactile coverage, and avoid the blind area problem of traditional single-point tactile detection.

[0110] Magnetic skin is a research direction in the field of flexible electronic skin. A complete magnetic skin is composed of two modules: magnetic elastomer and magnetic sensor. When the magnetic elastomer deforms elastically due to pressing, the orientation of the permanent magnet particles in the magnetic elastomer changes, causing a change in the magnetic field, which is recorded by the Hall sensor unit. The contact force data is obtained from the mapping relationship between the three-dimensional magnetic field changes and the contact force trained by the neural network method.

[0111] The wrist transmission mechanism end is provided with a six-dimensional force sensor, the six-dimensional force sensor is rigidly connected with the wrist transmission shaft through a flange, and X / Y / Z three-axis contact forces Fx, Fy and Fz and around-axis torques Tx and Ty can be synchronously measured, and data is output to a motion controller after Kalman filtering.

[0112] Each component cooperates through a multi-layer signal fusion architecture: a visual-based tactile perception scheme highly aligns and fuses tactile data with visual data, then associates pressure distribution with six-dimensional force information through a contact mechanics model, and finally uses an attention mechanism neural network to adaptively weight and fuse multi-source signals.

[0113] Compared with the traditional technology, the present application has the following advantages:

[0114] 1. The palm visual-tactile sensor 5 realizes unobstructed penetration and continuous acquisition of visual signals by covering the palm surface with a high-transparency multi-layer elastomer combined with a built-in high-resolution binocular camera. This design enables the system to obtain continuous depth information in real time and accurately calculate contact force through the deformation displacement of invisible fluorescent marker points, completing three-dimensional force field reconstruction of the contact area. As a result, a single sensor can achieve high integration of vision, touch and contact, significantly improving the continuity and accuracy of perception.

[0115] 2. The fingertip visual-tactile sensor 1 mounted on the fingertips uses a monocular camera with three color light strips to perform three-dimensional surface reconstruction based on the Photometric Stereo method to obtain high-precision tactile information. The high-wear-resistant opaque coating on the fingertips enhances durability and marker point recognition accuracy through water transfer printing and wire drawing processes. This design realizes continuous and detailed perception of fingertip tactile, enabling accurate detection of fingertip surface slip changes by analyzing the marker point displacement caused by elastomer deformation, effectively breaking through the detection blind spot caused by limited installation space of traditional tactile probes on fingertips.

[0116] 3. The magnetic flexible tactile skin 2 arranged on the knuckle side wall and the non-contact area of the palm realizes contact perception and contact force mapping in combination with a neural network algorithm based on Hall sensor array magnetic field signals. This design ensures the freedom and flexibility of the finger joints, effectively eliminates the detection blind spot of traditional single-point tactile sensors in the knuckle side wall and other areas, realizes full-area coverage of tactile information without dead angles, and improves the safety and comfort of overall human-machine interaction.

[0117] 4. The six-dimensional force sensor is rigidly connected with the wrist transmission shaft through a flange, capable of synchronously measuring three-axis spatial contact forces and around-axis torques. In combination with multi-axis mechanics perception and Kalman filtering and other interference filtering technologies, the system provides high-precision, multi-dimensional force interaction closed-loop feedback for the dexterous hand, significantly enhancing the adaptability and stability of the manipulator to complex operating environments.

[0118] 5、Adopt the perception scheme based on the fusion of vision and touch, and realize the space-time consistency alignment of the tactile data and the visual data through the precise time synchronization mechanism. The pressure distribution information is deeply integrated with the six-dimensional force sensor data by using the contact mechanics model, so as to form a comprehensive environment perception framework. On this basis, an adaptive weighting strategy is introduced, and the features of multi-source signals are combined to further optimize the information fusion process. The strategy significantly improves the response ability of the system to environmental changes and the understanding of the object state, thereby enhancing the robustness and accuracy of the grasping and operation decision, and ensuring the efficient operation of the robot in the complex environment.

[0119] In the description of the present application, it should be understood that the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0120] The above embodiments only describe the preferred modes of the present application, and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. A multi-modal sensor fusion based full area perception dexterous hand, characterized in that, The application relates to a hand structure, which comprises: a palm structure; a multi-dimensional force sensing component arranged at the wrist of the palm structure; a palm visual tactile sensor (5) arranged at the palm of the palm structure; a plurality of fingertip visual tactile sensors (1) arranged at the fingerpads of the fingers of the palm structure, the number of the fingertip visual tactile sensors (1) matching the number of the fingers of the palm structure; a magnetic flexible tactile skin (2) wrapped outside the palm structure; a control component arranged in the palm structure, the control component being electrically connected with the palm visual tactile sensor (5), the fingertip visual tactile sensor (1) and the magnetic flexible tactile skin (2); the control component is used for controlling the movement of the fingers and receiving feedback data of the palm visual tactile sensor (5), the fingertip visual tactile sensor (1) and the magnetic flexible tactile skin (2).

2. The multi-modal sensor fusion based full area perception dexterous hand according to claim 1, wherein, The palm structure comprises: a palm base (8), wherein the palm visual tactile sensor (5) is arranged at the palm of the palm base (8); five fingers, one end of each finger being connected with the palm base (8), and the fingertip visual tactile sensor (1) being arranged at the end of each finger away from the palm base (8); a finger driving part in transmission connection with the fingers, the finger driving part being arranged in the palm base (8) and used for driving the fingers to move.

3. The multi-modal sensor fusion based full area perception dexterous hand according to claim 2, wherein, The finger driving part comprises: a motor (4) fixedly connected with the palm base (8); a composite wire (3) having one end connected with the output shaft of the motor (4) and the other end connected with the fingers; the motor (4) drives the corresponding fingers to move through the composite wire (3); the motor (4) is electrically connected with the control component.

4. The multi-modal sensor fusion based full area perception dexterous hand according to claim 3, wherein: The finger at the index finger position is in transmission connection with the corresponding motor (4) through double composite wires (7).

5. The multi-modal sensor fusion based full area perception dexterous hand according to claim 3, wherein: The palm base (8) is provided with a cavity in the middle part; the cavity is used for accommodating the palm visual tactile sensor (5), the motor (4) and the control component; the cavity is provided with an open mouth on the side of the palm; the open mouth is used for mounting the palm visual tactile sensor (5), the motor (4) and the control component; the open mouth is fixedly connected with a high-transparency shell (6).

6. The multi-modal sensor fusion based full area perception dexterous hand according to claim 1, wherein: The palm visual tactile sensor (5) is provided with a binocular camera in the middle part, and the binocular camera is used for acquiring continuous depth information.

7. The multi-modal sensor fusion based full area perception dexterous hand according to claim 1, wherein: The magnetic flexible tactile skin (2) is prepared by mixing 400-mesh ferrous boron magnet particles with a silica gel solvent, the ferrous boron magnet particles are magnetically connected with a Hall sensor array, and the Hall sensor array is fixedly connected inside the magnetic flexible tactile skin (2).

8. The multi-modal sensor fusion based full area perception dexterous hand according to claim 2, wherein: The multi-dimensional force sensing component is a six-dimensional force sensor, and the six-dimensional force sensor is arranged at the wrist of the palm base (8).

Images