A multi-modal haptic feedback system for prosthetic hands

Abstract

The application discloses a multi-modal tactile feedback system for a prosthetic hand, which comprises a prosthetic hand including a main body and a joint angle sensor, a fingertip three-dimensional force sensor and a palm temperature sensor arranged on the main body; a tactile feedback device including a helmet body and a plurality of contact points and a plurality of high-frequency vibration modules arranged on the helmet body, the helmet body being arranged on the head of a user, the plurality of contact points being arranged corresponding to the hairy skin area on the top of the head of the user, and the plurality of high-frequency vibration modules being arranged corresponding to the non-hairy skin area on the forehead of the user; and a control processing unit which collects joint angle information, fingertip three-dimensional force information and palm temperature information sent by the joint angle sensor, the fingertip three-dimensional force sensor and the palm temperature sensor, and determines a tactile stimulation mode according to the relationship between the multi-modal information and a preset tactile stimulation mode, so as to control the contact points and / or the high-frequency vibration modules to work and perform tactile feedback on the surface of the specific skin area of the scalp of the user.

Description

Technical Field

[0001] This invention relates to the field of tactile feedback, and more particularly to a multimodal tactile feedback system for prosthetic hands. Background Technology

[0002] In recent years, haptic feedback technology has become a core support for prosthetic perception and environmental interaction. By transmitting mechanical and electrical stimuli to the human skin, it can reconstruct the tactile perception of missing limbs, and is a key technology for achieving precise grasping of prostheses and improving the immersion and safety of operation. Among them, the mainstream haptic feedback solutions currently used mainly include three categories: vibration stimulation, electrical stimulation, and implantable neural interfaces, which have already been initially applied in fields such as prosthetic control and remote control.

[0003] However, existing tactile feedback technologies still have significant shortcomings in practical applications, making it difficult to meet the complex information transmission requirements of fine grasping tasks. For example, when using vibration stimulation for feedback, long-term continuous stimulation of a single vibration mode can easily lead to skin sensory adaptation and cause information recognition to decay over time, making it impossible to stably transmit fine tactile information such as texture and pressure gradients. While electrical stimulation can achieve high-frequency information transmission, the stimulation current is prone to interfering with electromyographic signal acquisition, creating a natural conflict with the mainstream electromyographic control methods for prostheses, resulting in poor compatibility. At the same time, implantable neural interfaces generally require surgical intrusion into the human body, posing a risk of irreversible tissue damage, and have limited applicability to a limited population, making large-scale promotion and application difficult.

[0004] In recent years, although some studies have attempted to combine vibration and pressure modalities to achieve tactile feedback, these schemes lack a systematic design guided by bionics and still have technical shortcomings. On the one hand, they fail to distinguish the differences in receptor density between hairy and hairless skin; on the other hand, they fail to match the temporal and frequency characteristics of information with the response characteristics of neural channels, which can easily lead to mutual masking between multimodal stimuli, ultimately making it difficult to effectively improve the richness and clarity of tactile perception. More importantly, existing tactile feedback schemes mostly adopt fixed mapping strategies, which have poor neurophysiological adaptability and do not fully utilize the multi-channel neurophysiological characteristics of the human tactile system. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: The present invention discloses a multimodal tactile feedback system for prosthetic hands, in order to solve the problem that existing tactile feedback technologies have single modality, poor neurophysiological adaptability and insufficient multimodal coordination, which makes it impossible to meet the tactile feedback requirements of precise grasping of prosthetic hands.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a multimodal tactile feedback system for a prosthetic hand, characterized in that it includes: The prosthetic hand includes a main body and joint angle sensors, fingertip three-dimensional force sensors and palm temperature sensors located on the main body; A tactile feedback device includes a helmet body and multiple contact points and multiple high-frequency vibration modules disposed on the helmet body. The helmet body is placed over the user's head. The multiple contact points are disposed corresponding to the hairy skin area on the top of the user's head, and the multiple high-frequency vibration modules are disposed corresponding to the hairless skin area on the user's forehead. The control and processing unit acquires joint angle information, fingertip three-dimensional force information, and palm temperature information sent by the joint angle sensor, fingertip three-dimensional force sensor, and palm temperature sensor. It analyzes the current joint angle information, fingertip three-dimensional force information, and palm temperature information to determine the tactile stimulation mode based on the relationship between multimodal information and preset tactile stimulation mode. Based on the tactile stimulation mode, it controls the touch point and / or the high-frequency vibration module to work to provide tactile feedback to a specific skin area of ​​the user's scalp.

[0007] To address the shortcomings of existing tactile feedback technologies, such as limited modality, poor neurophysiological adaptability, and insufficient multimodal coordination, which prevent them from meeting the tactile feedback requirements for precise grasping in prosthetic hands, this invention designs a novel multimodal tactile feedback system for prosthetic hands. This system collects multimodal information by incorporating joint angle sensors, fingertip three-dimensional force sensors, and palm temperature sensors on the prosthetic hand. Based on the processing of this multimodal information, tactile feedback is provided to specific skin areas of the user's scalp via a tactile feedback device placed over the user's head. This helps upper limb amputees perceive the joint position, grip strength, and temperature of objects in contact with their prosthetic hand, enabling them to perform precise grasping and dynamic response tasks in daily life. This improves operational quality and safety, and enhances the user's operational accuracy and on-the-spot judgment.

[0008] Meanwhile, in practical applications, the scalp, as a large-area tactile interface, has anatomical characteristics such as hairy skin on the top rich in slow-adapting receptors suitable for spatial encoding, and hairless skin on the forehead dense with Pacinian bodies suitable for dynamic tracking, which have not yet been effectively utilized in prosthetic feedback. This invention's multimodal tactile feedback system addresses this gap by proposing a multimodal partitioning mapping method based on tactile neurophysiology. Human skin contains various tactile receptors, including Meissner bodies, Merkel cells, and Pacinian bodies. These different receptors exhibit significant differentiation in frequency response range, stimulus adaptation speed, and spatial distribution density, and can respectively correspond to encoding different types of tactile information such as dynamic vibration, static pressure, and high-frequency texture. By setting the arrangement of touch points and high-frequency vibration modules to correspond to different skin areas of the user's scalp, the information transmission rate and perception recognition are both improved through the synergy of touch point compression and high-frequency vibration modules.

[0009] Furthermore, in the multimodal haptic feedback system described in this invention, the joint angle information specifically includes:

[0010] in, , and These correspond to the flexion-extension, abduction, and pronation degrees of freedom of the thumb root joint in the prosthetic hand, respectively. to These correspond to the flexion and extension angles of the mid-joints of the index, middle, ring, and little fingers, respectively.

[0011] Furthermore, in the multimodal haptic feedback system described in this invention, the three-dimensional force information of the fingertip specifically refers to:

[0012] in, Indicates the lateral force on the left and right sides of the fingertips; Indicates the tangential force at the front and back of the fingertip; The vertical pressing force of the fingertip is represented; t represents the time variable, which represents any moment in continuous signal acquisition and processing; j = 1, 2, 3, 4, 5, which correspond to the fingertips of the thumb, index finger, middle finger, ring finger and little finger of the prosthetic hand, respectively.

[0013] Furthermore, in the multimodal haptic feedback system described in this invention, the haptic stimulation method includes a point-of-contact stimulation method: Multiple contact points are arranged on the helmet of the haptic feedback device. The control processing unit determines the multiple contact points in the corresponding column on the haptic feedback device based on the joint angle information, and activates the multiple contact points in the corresponding column to squeeze the user's scalp in a preset time sequence according to the sequence encoding of the movement direction; wherein the contact points squeeze the user's scalp for a preset duration.

[0014] Furthermore, in the multimodal haptic feedback system described in this invention, under the touch stimulation mode, it further includes: Multiple contact points are arranged on the helmet of the haptic feedback device. The control processing unit obtains the current palm temperature based on the palm temperature information, converts the palm temperature into Arabic numerals, and distributes them to multiple contact points so that the multiple contact points can be used to write on the user's scalp.

[0015] Furthermore, in the multimodal tactile feedback system described in this invention, the tactile stimulation method includes vibration stimulation: The control processing unit calculates the normal force and tangential force based on the three-dimensional force information of the fingertip, and calculates the slip ratio based on the normal force and tangential force. The normal force is linearly mapped to the vibration amplitude A:

[0016] in, Indicates the normal force at the fingertip; Indicates the upper limit of the normal force range; The slip ratio is mapped to the vibration frequency f:

[0017] Where S represents the slip ratio; Based on the combination of the vibration amplitude A and the vibration frequency f, multiple high-frequency vibration modules are controlled to comprehensively feed back multi-dimensional force information from the fingertips.

[0018] Furthermore, in the multimodal haptic feedback system described in this invention, the calculation of the normal force and tangential force based on the three-dimensional force information of the fingertip specifically involves:

[0019] in, Let be the three-dimensional force vector at the fingertip of the j-th finger at time t. Let be the unit normal vector of the fingertip contact surface of the j-th finger; Represented as Transpose of;

[0020] in, The square of the lateral force on the left and right sides of the fingertips; The square of the tangential force at the front and back of the fingertip; It is the square of the normal force applied vertically by the fingertip.

[0021] Furthermore, in the multimodal tactile feedback system described in this invention, under the vibration stimulation mode, it further includes: When the slip ratio is greater than or equal to the preset slip ratio threshold, the multiple high-frequency vibration modules are controlled to perform emergency vibration according to the preset amplitude and preset frequency.

[0022] Furthermore, in the multimodal haptic feedback system of the present invention, the helmet of the haptic feedback device is also equipped with a speaker. The control processing unit analyzes the current three-dimensional force information of the fingertips and the temperature information of the palm to determine the auditory stimulation mode according to the relationship between the preset multimodal information and the auditory stimulation mode, and controls the speaker to work based on the auditory stimulation mode so as to provide auditory feedback to the user's head using the speaker.

[0023] Furthermore, in the multimodal tactile feedback system described in this invention, the auditory stimulation method includes: The temperature alarm stimulation method controls the speaker to emit sound in the first working mode when the palm temperature obtained based on the palm temperature information is greater than the preset maximum temperature or less than the preset minimum temperature. The slip alarm stimulation mode controls the speaker to emit sound in the second working mode when the slip rate obtained based on the three-dimensional force information of the fingertip is greater than or equal to the preset slip rate alarm value.

[0024] The beneficial effects of this invention are as follows: The multimodal tactile feedback system for prosthetic hands designed in this invention can assist in the use of prosthetic hands. It can be used in tactile feedback scenarios that require precise grasping and dynamic response, and its core service is the reconstruction of upper limb prosthetic functions. In practical applications, this multimodal tactile feedback system detects and feeds back joint angle information, fingertip three-dimensional force information, and palm temperature information on the relevant prosthetic hand in real time to obtain information on grasping force distribution, joint position, and object temperature. This helps amputees complete delicate operations such as picking up and putting away fragile items and recognizing temperature-sensitive objects in daily life, reducing visual dependence and cognitive load, and improving the quality and safety of task completion.

[0025] Based on this, when the multimodal haptic feedback system of the present invention is applied in practice, it can collaboratively encode multimodal information such as joint angle information, fingertip three-dimensional force information, and palm temperature information, thereby obtaining high information bandwidth and spatial immersion characteristics. It can be extended to virtual reality interaction and remote surgical training, enhancing the operator's sense of environmental embedding and on-site judgment ability. In any scenario where the machine state needs to be converted into human-perceptible tactile signals to achieve closed-loop control, the multimodal haptic feedback system of the present invention can be used to obtain efficient and low-latency cross-modal information transmission, which has good prospects for promotion and application value. Attached Figure Description

[0026] Figure 1 This is a block diagram illustrating the working principle of the multimodal tactile feedback system for a prosthetic hand according to one embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a prosthetic hand in one embodiment of the multimodal tactile feedback system for a prosthetic hand according to the present invention; Figure 3 This is a schematic diagram of the tactile feedback device in one embodiment of the multimodal tactile feedback system for prosthetic hand of the present invention; Figure 4 This is a schematic diagram illustrating the matching principle of the control processing unit of the multimodal tactile feedback system for prosthetic hand of the present invention, which matches tactile stimulation mode and auditory stimulation mode according to multimodal information in one embodiment. Figure 5This invention relates to a method for encoding contact array information and driving contact points in a tactile feedback device of a multimodal tactile feedback system for prosthetic hands, as described in one embodiment of the present invention. Figure 6 This invention relates to a method for encoding high-frequency vibration module array information and driving high-frequency vibration modules in a tactile feedback device of a multimodal tactile feedback system for prosthetic hands, as described in one embodiment of the present invention. Figure 7 This invention relates to a method for encoding speaker information and driving speakers in one embodiment of a multimodal tactile feedback system for prosthetic hands.

[0027] Label Explanation: 1. Prosthetic hand; 11. Joint drive module; 12. Joint angle sensor; 13. Fingertip three-dimensional force sensor; 14. Palm temperature sensor; 15. Voice interaction module; 2. Tactile feedback device; 21. Contact point; 22. High-frequency vibration module; 23. Speaker module; 24. Helmet body; 25. Circuit system. Detailed Implementation

[0028] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0029] To address the problems of existing tactile feedback technologies, such as single modality, poor neurophysiological adaptability, and insufficient multimodal synergy, which prevent them from meeting the tactile feedback requirements of the prosthetic hand 1 for fine grasping, this invention designs a new multimodal tactile feedback system for the prosthetic hand 1.

[0030] like Figure 1 As shown, and in conjunction with references Figure 2 and Figure 3 In this invention, a multimodal tactile feedback system for a prosthetic hand 1 is designed. The multimodal tactile feedback system specifically includes: a prosthetic hand 1, a tactile feedback device 2, and a control processing unit. The control processing unit is used to receive multimodal information transmitted from the prosthetic hand 1 and to analyze and process the multimodal information, thereby controlling the tactile feedback device 2 placed on the user's head to provide tactile and auditory feedback to the user's head. This enables amputees to use the prosthetic hand 1 to perform delicate operations such as picking up and placing fragile items and identifying temperature-sensitive objects in daily life, reducing visual dependence and cognitive load, and improving the quality and safety of task completion.

[0031] In the multimodal tactile feedback system designed in this invention, in practical application, the prosthetic hand 1 can specifically include a main body and a joint angle sensor 12, a fingertip three-dimensional force sensor 13, and a palm temperature sensor 14 disposed on the main body; at the same time, in order to ensure that the tactile feedback device 2 can be properly placed on the user's head and realize various forms of tactile stimulation, it can specifically include a helmet 24 and multiple contact points 21 and multiple high-frequency vibration modules 22 disposed on the helmet 24. The helmet 24 is placed on the user's head, the multiple contact points 21 are disposed corresponding to the hairy skin area on the top of the user's head, and the multiple high-frequency vibration modules 22 are disposed corresponding to the hairless skin area on the user's forehead.

[0032] Based on this, the control processing unit can effectively acquire the joint angle information, fingertip three-dimensional force information, and palm temperature information sent by the joint angle sensor 12, fingertip three-dimensional force sensor 13, and palm temperature sensor 14 on the prosthetic hand 1, and analyze the current joint angle information, fingertip three-dimensional force information, and palm temperature information to determine the tactile stimulation mode according to the relationship between multimodal information and preset tactile stimulation mode, and control the operation of the contact point 21 and / or the high-frequency vibration module 22 based on the tactile stimulation mode to provide tactile feedback to the surface of a specific skin area of ​​the user's scalp.

[0033] It should be noted that multiple contact points 21 and multiple high-frequency vibration modules 22 are arranged on the helmet body 24 of the aforementioned tactile feedback device 2 to correspond to different skin areas of the user's scalp. At the same time, the core reason for using the control processing unit to select contact points 21 and / or high-frequency vibration modules 22 to work respectively based on the relationship between multimodal information and preset tactile stimulation methods is to embody the principle of "bionic matching and regional separation". The inventors would like to emphasize that the human skin's tactile system contains four classic receptors, each with significantly different frequency response and adaptive characteristics. Meissner bodies and Merkel cells, as receptors, are distributed in hairy skin areas, sensitive to low-frequency pressure (5-30Hz) and adapt slowly, making them suitable for encoding spatial location and static information; Pacinian bodies are concentrated in hairless skin (such as fingertips and forehead), sensitive to high-frequency vibrations (optimal 200-300Hz, extending to 1000Hz) and adapt quickly, making them suitable for tracking dynamic changes.

[0034] Based on this invention, multiple contact points 21 are arranged in the hairy skin area of ​​the scalp, utilizing their low-frequency compression (<1Hz) to activate the Meissner body-Merkel cell pathway, achieving spatial topological encoding of joint angles. Simultaneously, multiple high-frequency vibration modules 22 are arranged in the hairless skin area of ​​the forehead, utilizing their high-frequency vibration (1000Hz) to target Pacinian bodies, achieving continuous dynamic feedback of three-dimensional force at the fingertips. This correspondence between anatomical regions and neural mechanisms avoids the energy waste and sensory blurring caused by the insensitivity of hairy skin to high-frequency vibrations, while leveraging the high-density distribution of Pacinian bodies in the forehead skin, thus lowering the perception threshold and increasing the information transmission rate at the same vibration intensity.

[0035] Of course, in order to optimize the actual application experience, such as Figure 2 As shown, in some embodiments, the main body of the prosthetic hand 1 may specifically be equipped with seven joint drive modules 11, seven joint angle sensors 12, four fingertip three-dimensional force sensors 13, one palm temperature sensor 14, and one voice interaction module 15. The joint drive modules 11 enable independent motion control of the thumb in three degrees of freedom in the X, Y, and Z directions, and the other four fingers in one degree of freedom each. The seven joint angle sensors 12 are distributed along each joint axis and provide real-time feedback of joint angle information. The fingertip three-dimensional force sensors 13 are embedded between the five fingers to provide real-time feedback of fingertip three-dimensional force information, thereby detecting normal and tangential forces. The palm temperature sensor 14 enables non-contact temperature sensing. The voice interaction module 15 supports users to input natural language commands and control the prosthetic hand 1.

[0036] Accordingly, see Figure 3 As shown, in some embodiments, the helmet 24 of the haptic feedback device 2 may be specifically equipped with 40 bistable contacts 21, 5 high-frequency vibration modules 22, and 2 speaker modules 23, and the helmet 24 is equipped with a circuit system 25 to power and drive the aforementioned electrical components. Specifically, the bistable contacts 21 may be arrayed in the hairy skin area of ​​the scalp, achieving low-frequency, high-amplitude vibration and compressing the user's scalp to provide haptic feedback; the high-frequency vibration modules 22 may be specifically distributed in the hairless skin area of ​​the forehead, achieving high-frequency, low-amplitude vibration to provide haptic feedback; the speaker modules 23 provide auditory alarms and may be distributed according to the user's ear positions to provide auditory feedback.

[0037] In this embodiment, the joint angle information output by the prosthetic hand 1 is specifically as follows:

[0038] in, , and These correspond to the flexion-extension, abduction, and pronation degrees of freedom of the thumb root joint in prosthetic hand 1, respectively. to These correspond to the flexion and extension angles of the mid-joints of the index, middle, ring, and little fingers, respectively.

[0039] It should be noted that, based on the obtained joint angle information, the aforementioned control processing unit will further use a first-order difference approximation method to calculate the joint angle change rate:

[0040] in, Indicates the rate of change of joint angle; This represents the instantaneous joint angle of the i-th joint at the current sampling time t; This indicates that the i-th joint was at the previous sampling time. Joint angles; This indicates the sampling period, measured in milliseconds (ms). In some specific implementations, the sampling period can be set. =10ms, and corresponding sampling frequency =100Hz.

[0041] Therefore, the aforementioned control processing unit can further classify the joint motion state based on the joint rate of change and by setting a threshold for the joint rate of change:

[0042] In some embodiments, the aforementioned joint change rate threshold It can be specifically set to 5° / s.

[0043] It should be noted that joint change rate classification is a key preliminary step in multimodal information routing decision-making. Its technical role is to dynamically select the optimal tactile feedback channel based on the time-frequency characteristics of the information, as explained below: (1) Achieve adaptive switching between fast and slow channels When the joint change rate When the condition is static, the joint angle information is presented via a low-frequency compression method through the SMA contact array. Due to the ≥5s cooling interval of the contacts, it is only suitable for carrying slowly changing or static spatial position information. Utilizing the slow adaptation characteristics of Meissner bodies and Merkel cells in hairy skin, it provides continuous positional anchoring for the user.

[0044] When the joint change rate When the condition is determined to be dynamic, the joint angle information is switched to the vibration motor channel to provide feedback on the finger movement trend via high-frequency modulation. Because the Pacinian body has rapid adaptation and response characteristics to high-frequency vibration, it can track joint dynamics in real time, compensating for the inability of contact points to reflect rapid changes.

[0045] (2) Avoid sensory adaptation and channel conflict Continuous vibration stimulation applied to a stationary joint will rapidly induce sensory adaptation in the Pacinian bodies, causing the user to lose positional awareness within seconds. Conversely, if tactile feedback is used on rapidly moving joints, the 5-second cooling interval will create a significant time delay, making real-time closed-loop control impossible. Therefore, by classifying the rate of change, this multimodal tactile feedback system ensures that each type of information is transmitted only through its corresponding physiological channel, avoiding cross-modal masking and receptor fatigue.

[0046] Accordingly, in this embodiment, the fingertip three-dimensional force information transmitted by the prosthetic hand 1 of the present invention is specifically as follows:

[0047] in, Indicates the lateral force on the left and right sides of the fingertips; Indicates the tangential force at the front and back of the fingertip; The vertical pressing force of the fingertip is represented; t represents the time variable, which represents any moment in continuous signal acquisition and processing; j = 1, 2, 3, 4, 5, which correspond to the fingertips of the thumb, index finger, middle finger, ring finger and little finger of the prosthetic hand, respectively.

[0048] At this point, the control processing unit of the multimodal haptic feedback system can calculate the normal force and tangential force based on the three-dimensional force information of the fingertip, and calculate the slip ratio based on the normal force and tangential force.

[0049] It should be noted that, in this invention, the calculation of the normal force based on the three-dimensional force information of the fingertip specifically refers to:

[0050] in, Let be the three-dimensional force vector at the fingertip of the j-th finger at time t; Let be the unit normal vector of the fingertip contact surface of the j-th finger; Represented as The transpose of .

[0051] Similarly, in this invention, the calculation of the tangential force based on the three-dimensional force information of the fingertip specifically involves:

[0052] in, The square of the lateral force on the left and right sides of the fingertips; The square of the tangential force at the front and back of the fingertip; It is the square of the normal force applied vertically by the fingertip.

[0053] Therefore, based on the above normal and tangential forces, the slip ratio can be obtained:

[0054] In some implementations, the control processing unit can specifically set the slip ratio when calculating the slip ratio. , which is a regularization constant to prevent division by zero.

[0055] Therefore, based on the obtained slip ratio, the control processing unit in this multimodal tactile feedback system can classify the current safety state according to a preset slip ratio threshold:

[0056] In some embodiments, a slip ratio threshold can be preset. , .

[0057] In addition, the control processing unit can obtain the current palm temperature T based on the palm temperature information and set the judgment relationship between temperature and dangerous state:

[0058] In some embodiments, a preset safety threshold can be implemented. , .

[0059] It should be noted that the reason for setting up a slip ratio safety state classification is that this classification directly determines the collaborative feedback strategy between the high-frequency vibration module 22 and the subsequent loudspeaker, forming a three-level defense mechanism: For example, under the safety classification, the high-frequency vibration module 22 can operate in a normal continuous mode, and its amplitude is linearly mapped to the normal force. That is, the current normal force Fz is linearly mapped to the PWM duty cycle of the high-frequency vibration module 22. For example, if Fz=8N and the range Fz,max=50N, then the amplitude A=16%. At this time, the vibration frequency can be maintained at the 1000Hz fundamental frequency, providing only continuous perception of the gripping force without triggering the speaker.

[0060] At the danger level, the amplitude of the high-frequency vibration module 22 is increased to a medium intensity (e.g., 30%), the frequency of the high-frequency vibration module 22 is modulated to a 10Hz intermittent pulse, and the speaker can emit a 0.5s interval prompt tone to remind the user that "slipping tendency has appeared" and that the user needs to actively adjust the grip force, but it has not yet constituted an emergency threat.

[0061] Under critical grading, the amplitude of the high-frequency vibration module 22 can be controlled to jump to 50%, the frequency to drop to a strong pulse of 15Hz, and the speaker emits a rapid 0.2s cycle frequency conversion scan. The contact array can mark the sliding position by flashing at a low frequency of 1Hz corresponding to the finger column. The three modes are activated simultaneously, forcibly interrupting the user's current attention and driving them to immediately increase the normal force to prevent the object from slipping.

[0062] Accordingly, the reason for setting the relationship between temperature and dangerous conditions is to determine whether temperature information is presented to the user's scalp only through touch stimulation, or whether it needs to be superimposed with sound from a speaker to provide auditory stimulation and achieve emergency warning, so as to avoid users coming into contact with objects of extreme temperatures and causing burns or frostbite.

[0063] See Figure 4 As shown, in the multimodal tactile feedback system designed in this invention, the control processing unit can preset the encoding methods for the joint angle information, fingertip three-dimensional force information and palm temperature information in the above multimodal information, so that after parsing, the corresponding preset tactile stimulation method can be selected, thereby controlling the contact point 21 and / or high-frequency vibration module 22 in the tactile feedback device 2 to work and realize tactile feedback.

[0064] Of course, see further. Figure 4 As shown, in some embodiments, since the helmet 24 of the haptic feedback device 2 is also equipped with a speaker, after the control processing unit analyzes the joint angle information, fingertip three-dimensional force information and palm temperature information, it can also select the corresponding auditory stimulation mode to provide auditory feedback to the user's head using the speaker.

[0065] Therefore, as Figure 5 and Figure 6 As shown, in practical applications, in this multimodal tactile feedback system designed by the present invention, when the control processing unit selects the tactile stimulation method, it can specifically select: touch stimulation method and vibration stimulation method. like Figure 5 As shown, this tactile stimulation method includes touch point 21 stimulation, which can be selected based on joint angle information and palm temperature information: When acquiring joint angle information, multiple contact points 21 are arrayed on the helmet body 24 of the tactile feedback device 2. The control processing unit determines the multiple contact points 21 in the corresponding column on the tactile feedback device 2 based on the joint angle information, and activates the multiple contact points 21 in the corresponding column to squeeze the user's scalp in a preset timing sequence according to the sequence encoded movement direction; wherein, the contact points 21 squeeze the user's scalp for a preset duration.

[0066] For example, such as Figure 5 As shown, and in conjunction with references Figure 4In some implementations, the joint angle range can be evenly divided into 8 layers, i.e., n=8, with a layer spacing of 11.25° between each layer. Taking the right prosthetic hand 1 as an example, it can control the left 1st to left 5th columns of the contact point 21 array to correspond to the thumb to little finger of the right prosthetic hand 1, and establish the relationship between the joint angle information of each finger and the contact point 21 array. This allows the contact points 21 in the corresponding column to be activated sequentially from top to bottom or from bottom to top, i.e., according to the sequential encoding. The system activates multiple contacts 21 in the corresponding column according to a preset timing sequence to squeeze the user's scalp. Each contact 21 can squeeze the user's scalp for a preset duration of 100ms and be activated according to a 50ms timing interval. At the same time, multiple contacts can be time-division multiplexed according to physical drive. Since SMA requires 5000ms to cool down after each activation, the system divides the 40 contacts into multiple groups and activates them in rotation according to the groups. The grouping strategy can be that four adjacent contacts are grouped together, and the cooling between groups can be controlled to 5000ms to achieve time-division multiplexing.

[0067] When acquiring palm temperature information, multiple contact points 21 are arrayed on the helmet 24 of the tactile feedback device 2. The control processing unit acquires the current palm temperature based on the palm temperature information, converts the palm temperature into Arabic numerals, and distributes them to the multiple contact points 21 so that they can be written onto the user's scalp surface using the multiple contact points 21. For example, the tens digit of the palm temperature can be written first using the contact point array 21, and the units digit of the palm temperature can be written last using the contact point array 21.

[0068] like Figure 6 As shown, this tactile stimulation method includes vibration stimulation, which can be selected based on the three-dimensional force information of the fingertip: After acquiring the three-dimensional force information of the fingertip, the control processing unit calculates the normal force and tangential force based on the information, and then calculates the slip ratio based on the normal force and tangential force. Next, the normal force is linearly mapped to the PWM duty cycle, i.e., the vibration amplitude A; and the slip ratio is mapped to the vibration frequency f. The normal force is linearly mapped to the vibration amplitude A:

[0069] in, Indicates the normal force at the fingertip; This represents the upper limit of the normal force range, used for normalization processing; The slip ratio is mapped to the vibration frequency f:

[0070] Where S represents the slip ratio; Based on the combination of the vibration amplitude A and the vibration frequency f, multiple high-frequency vibration modules 22 are controlled to comprehensively feed back multi-dimensional force information from the fingertip.

[0071] It should be noted that, under this vibration stimulation mode, when the slip ratio is ≥ a preset slip ratio threshold, the multiple high-frequency vibration modules 22 are controlled to perform emergency vibration according to the preset amplitude and preset frequency; for example, in some embodiments, when the slip ratio S ≥ 0.3, the emergency mode is triggered, and the vibration amplitude A of the high-frequency vibration module 22 is controlled to be 100% and the vibration frequency is 20Hz, and the multiple high-frequency vibration modules 22 intermittently pulse vibrate.

[0072] Accordingly, such as Figure 7 As shown, in practical applications, in this multimodal tactile feedback system designed by the present invention, the control processing unit can also analyze the joint angle information, fingertip three-dimensional force information, and palm temperature information, and then use a speaker to provide auditory feedback to the user's head. This auditory feedback method can specifically include the following stimulation methods: The temperature alarm stimulation method controls the speaker to emit sound in the first working mode when the palm temperature obtained from the palm temperature information is greater than the preset maximum temperature or less than the preset minimum temperature. The slip alarm stimulation method, after obtaining the slip rate based on the three-dimensional force information of the fingertip, controls the speaker to emit sound in the second working mode when the slip rate is greater than or equal to the preset slip rate alarm value.

[0073] In practical applications, when the speaker is in its first working mode, it can be specifically set to emit sound at a frequency of 900Hz for the speaker closest to the user's left ear and at 1100Hz for the speaker closest to the user's right ear. This creates a spatial movement illusion with a frequency difference of 200Hz and lasts for 500ms, thereby achieving a temperature alarm and completing auditory feedback. When the speaker is in the second working mode, it can be controlled to emit sound at a frequency between 1000Hz and 1500Hz with a period of 200ms to create a sense of urgency and achieve a sliding alarm.

[0074] It is important to note that, such as Figure 7As shown, the auditory feedback method of this speaker can also include a temperature display stimulation method. When the touch point 21 is pressed against the user's scalp to display the temperature based on the palm temperature information, a 50ms prompt tone can be controlled to be emitted by the single speaker near the user's left ear when the touch point 21 displays the temperature in analog numbers; a 50ms prompt tone can be emitted by the single speaker near the user's right ear between the strokes of the analog numbers on the touch point 21; at the same time, a 100ms prompt tone is emitted by the two speakers on both ears simultaneously between the number intervals; and after the analog temperature display of the touch point 21 ends, a 50ms prompt tone can be emitted by the two speakers on both ears simultaneously.

[0075] In summary, for ease of understanding, this invention further specifies the application of the multimodal tactile feedback system to the prosthetic hand 1 and the implementation process of its operation: Controlling the prosthetic hand 1 to grasp a smooth glass cup containing hot water (temperature 65℃, slippery and fragile surface) involves the following three stages: movement stage: approach and contact (temperature detection), grasp establishment (force and position synchronization), lifting and maintenance (slippage detection and dynamic adjustment), and release and reset.

[0076] Phase 1: Approach and Contact (Temperature Sensing) In this stage, the palm of the prosthetic hand 1 is first brought close to the smooth glass. The palm temperature sensor 14 contacts the glass wall and detects a palm temperature of T=65℃. At this time, the control processing unit receives the palm temperature information and selects the stimulation mode of the contact point 21 to encode the writing of the number "65" in the first to fourth rows of the contact point 21 array. The tens digit "6" is activated sequentially from top to bottom by the contact points 21 in the first and second rows (each contact point 21 is activated and squeezed for 100ms with a 50ms interval). The units digit "5" is drawn in the same way by the third and fourth rows. At this time, the control processing unit also selects the auditory stimulation mode so that the speakers on both sides of the left and right ears simultaneously emit an 800Hz prompt tone for 100ms while the contact points 21 are writing the temperature numbers. At the same time, the high temperature danger triggering control processing unit selects the temperature alarm stimulation mode, controls the sound frequency of the speaker near the left ear to be 950Hz and the sound frequency of the speaker near the right ear to be 1050Hz, and they work for 500ms respectively to provide a temperature alarm. Accordingly, after acquiring the fingertip three-dimensional force information sent by the fingertip three-dimensional force sensor 13 of the prosthetic hand 1, the control processing unit selects a vibration stimulation mode and uses encoding to control multiple high-frequency vibration modules 22 to vibrate the hairless area of ​​the user's forehead with 5Hz intermittent pulses and 10% amplitude, thereby providing tactile feedback to confirm the establishment of contact.

[0077] In other words, in this first stage, the control processing unit can control the tactile feedback device 2 to make the user perceive the number "65" on the top of their head, hear a high-frequency warning in their left ear, feel a slight vibration on their forehead, and make a comprehensive judgment that "high temperature is dangerous and you should hold it lightly".

[0078] Phase Two: Grasp Establishment (Synchronization of Force and Position) When the five fingers of the prosthetic hand 1 are bent, the control processing unit continuously receives joint angle information sent by multiple joint angle sensors 12. The thumb θ1=45° is mapped to the activation of the fourth contact point in the first column, the index finger θ2=60° is mapped to the activation of the fifth contact point in the second column, and so on for the other fingers. This is to determine the multiple contact points 21 in the corresponding column on the tactile feedback device 2 through the joint angle information, and to activate the multiple contact points 21 in the corresponding column to squeeze the user's scalp according to the sequence encoding of the movement direction and the preset timing sequence. At this time, the control processing unit also receives the fingertip three-dimensional force information sent by the fingertip three-dimensional force sensor 13, and obtains the vibration amplitude of the normal force mapping and the vibration frequency of the slip ratio mapping based on the fingertip three-dimensional force information, so as to control the vibration amplitude and vibration frequency of the high-frequency vibration module 22.

[0079] In other words, in this second stage, the control processing unit can control the tactile feedback device 2 to make the contact point 21 on the user's head provide feedback to the user that "the hand shape has been established", and use the high-frequency vibration module 22 to provide feedback to the user's forehead, so that the user's forehead feels a stable weak vibration, confirming that "the grip is safe".

[0080] Phase 3: Enhanced Maintenance (Slip Detection and Dynamic Adjustment) When prosthetic hand 1 grasps a smooth glass, the glass tilts. At this moment, based on the received three-dimensional force information from the fingertip, the control processing unit can calculate a sudden increase in tangential force: Ft = 4N, Fz = 10N. Ft is the tangential force (the resultant force of Fx and Fy), Fz is the normal force, and the slip ratio s is:

[0081] in, To prevent division by zero regularization constants; the slip ratio s = 0.4 ≥ the preset slip ratio threshold 0.3, at which point a critical vibration alarm is triggered. The high-frequency vibration module 22 used for vibration stimulation controls its own vibration amplitude and frequency to switch to emergency mode, for example: the amplitude jumps to 50% and the frequency drops to 15Hz; at the same time, the control processing unit can control the speaker to perform the slip alarm stimulation mode, that is, it can control the speaker to perform frequency conversion sound between 1000Hz and 1500Hz, with a period of 0.2s, to generate a sense of urgency and realize the slip alarm; at this time, the above-mentioned control processing unit can control the contact 21 to perform the contact 21 stimulation mode, using low-frequency flashing to mark the slip position.

[0082] In other words, in this third stage, the control processing unit can control the haptic feedback device 2 to make the user perceive "index finger slippage + insufficient force" so that the user can fine-tune the grip force.

[0083] Phase Four: Release and Reset After the user drinks water, the prosthetic hand 1 releases the smooth glass. The joint angle sensor 12, fingertip three-dimensional force sensor 13, and palm temperature sensor 14 of the prosthetic hand 1 no longer detect multimodal information. The high-frequency vibration module 22 becomes silent. The array composed of multiple contact points 21 can be activated sequentially once, and the speakers corresponding to the left and right ears can emit a 50ms completion tone.

[0084] In other words, in this fourth stage, the control processing unit can control the aforementioned haptic feedback device 2 to make the user perceive "task completed, system in standby".

[0085] In summary, the joint angle sensor 12, three-dimensional force sensor and temperature sensor of the prosthetic hand 1 of the multimodal haptic feedback system designed in this invention are used to detect the tilt angle of the relevant joints, the magnitude of the force and the temperature of the palm in real time, and send the detected prosthetic detection data to the haptic feedback helmet. The control processing unit in the haptic feedback helmet analyzes the received multimodal data using a low-frequency, high-amplitude feedback method for position signals, a high-frequency, low-amplitude feedback method for force signals, and a discrete coding feedback method for temperature signals, and reproduces them as tactile and auditory stimulation modes. Finally, the tactile and auditory stimulation modes are fused and transmitted to the user's head through the feedback device in the haptic feedback helmet, thereby achieving multimodal tactile feedback.

[0086] In practical applications, the haptic feedback device 2 of this multimodal haptic feedback system integrates a contact point 21, a high-frequency vibration module 22, and a speaker, enabling the spatiotemporal fusion of three different feedback forms. These three devices are spatially partitioned according to a biomimetic correspondence and coordinated temporally through priority arbitration to achieve natural information distribution and synergistic enhancement.

[0087] In practical applications, the core of this invention's multimodal tactile feedback system lies in constructing a triple-matching information encoding system of "perceptual attributes - neural mechanisms - stimulation modalities." The system first analyzes the multidimensional information of the prosthetic dexterous hand, and then, based on the physiological characteristics of human skin's tactile receptors, maps different types of information to the optimal stimulation modality: the spatial position information of the joint angle is presented through low-frequency squeezing stimulation of contact point 21, matching the slow adaptation characteristics of Meissner bodies and Merkel cells; the dynamic changes in the three-dimensional force of the fingertips are transmitted through high-frequency vibration of the vibration motor, targeting the rapid response mechanism of the Pacinian bodies; the absolute value of the palm temperature is encoded through the spatial pattern of the contact point 21 array, utilizing the skin's continuous perception of static pressure; and emergency alarms are interrupted through cross-modal auditory interruption by the speaker, circumventing the adaptive limitations of the tactile channel. Ultimately, the three different forms of stimulation are distributed synergistically in specific areas of the scalp: contact point 21 covers the hairy skin on the top of the head to provide spatially precise but slow information, vibration is concentrated on the hairless skin on the forehead to achieve high-frequency continuous feedback, and auditory stimulation, independent of the body surface modality, provides a time anchor point, forming a cross-modal perception reconstruction that conforms to the laws of neurophysiology.

[0088] Based on this, the multimodal tactile feedback system for prosthetic hand 1 designed in this invention has the following beneficial effects: 1. This invention is based on a multimodal information mapping method of biomimetic neurophysiology. Based on the differences in frequency response characteristics of Meissner bodies, Merkel cells and Pacinian bodies, spatial position information is mapped to low-frequency squeezing stimulation of hairy skin and dynamic force information is mapped to high-frequency vibration stimulation of hairless skin. By adjusting the placement of the contact point 21 and the high-frequency vibration module 22, the precise matching of information attributes and neural channels is achieved, effectively avoiding the sensory adaptation and information masking problems of traditional single-modal feedback. 2. The multimodal tactile feedback system designed in this invention can utilize a partitioning strategy of spatial encoding with hairy skin on the top of the head and dynamic encoding with hairless skin on the forehead when performing tactile stimulation. This breaks through the area limitation of traditional forearm feedback, providing a larger area that can be stimulated and higher spatial resolution. At the same time, it has good concealment and wearing comfort, and does not affect the daily activities of the residual limb. 3. The multimodal tactile feedback system designed in this invention can realize the collaborative coding of the touch point 21, the high-frequency vibration module 22 and the speaker. Through the slow spatial display of the touch point 21, the fast continuous modulation of the vibration motor and the cross-modal time marking of the speaker, it realizes the natural diversion and parallel transmission of static / dynamic, continuous / discrete and routine / emergency information, which significantly improves the information transmission rate and closed-loop control efficiency. 4. The multimodal tactile feedback system designed in this invention is a non-invasive tactile feedback system that achieves feedback based on physical contact and vibration stimulation. It does not require surgical implantation of electrodes, thus avoiding interference of electrical stimulation with electromyographic signals and the long-term health risks of implants.

[0089] Therefore, the multimodal tactile feedback system designed in this invention can be applied to the field of rehabilitation assistance, helping upper limb amputees to perceive the joint position, grip strength and temperature of the object in contact with the prosthetic hand 1, complete precise gripping and dynamic response tasks in daily life, and improve the quality and safety of operation.

[0090] Meanwhile, this multimodal tactile feedback system can also be applied to amputee rehabilitation and virtual reality interaction, transforming physical contact attributes (hardness, temperature, slippage) in the virtual environment into real tactile stimulation, enhancing the user's sense of environmental immersion and operational immersion.

[0091] It should be noted that the multimodal tactile feedback system designed in this invention can be effectively applied in the field of robotic arm teleoperation. It can feed back the contact force, position and temperature information of the end effector of the robotic arm to the teleoperation user, improve the user's operation accuracy and on-site judgment ability, and has good prospects for promotion and application value.

[0092] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A multimodal tactile feedback system for a prosthetic hand, characterized in that, include: The prosthetic hand includes a main body and joint angle sensors, fingertip three-dimensional force sensors and palm temperature sensors located on the main body; A tactile feedback device includes a helmet body and multiple contact points and multiple high-frequency vibration modules disposed on the helmet body. The helmet body is placed over the user's head. The multiple contact points are disposed corresponding to the hairy skin area on the user's scalp to achieve low-frequency high-amplitude vibration and to compress the user's scalp. The multiple high-frequency vibration modules are disposed corresponding to the hairless skin area on the user's forehead to achieve high-frequency low-amplitude vibration. The control and processing unit acquires joint angle information, fingertip three-dimensional force information, and palm temperature information sent by the joint angle sensor, fingertip three-dimensional force sensor, and palm temperature sensor. It analyzes the current joint angle information, fingertip three-dimensional force information, and palm temperature information to determine the tactile stimulation mode based on the relationship between multimodal information and preset tactile stimulation mode. Based on the tactile stimulation mode, it controls the touch point and / or the high-frequency vibration module to work to provide tactile feedback to a specific skin area of ​​the user's scalp.

2. The multimodal haptic feedback system according to claim 1, characterized in that, The joint angle information is specifically as follows: in, , and These correspond to the flexion-extension, abduction, and pronation degrees of freedom of the thumb root joint in the prosthetic hand, respectively. to These correspond to the flexion and extension angles of the mid-joints of the index, middle, ring, and little fingers, respectively.

3. The multimodal haptic feedback system according to claim 1, characterized in that, The specific three-dimensional force information of the fingertip is as follows: in, Indicates the lateral force on the left and right sides of the fingertips; Indicates the tangential force at the front and back of the fingertip; The vertical pressing force of the fingertip is represented; t represents the time variable, which represents any moment in continuous signal acquisition and processing; j = 1, 2, 3, 4, 5, which correspond to the fingertips of the thumb, index finger, middle finger, ring finger and little finger of the prosthetic hand, respectively.

4. The multimodal haptic feedback system according to claim 1, characterized in that, The tactile stimulation methods include point stimulation methods: Multiple contact points are arranged on the helmet of the haptic feedback device. The control processing unit determines the multiple contact points in the corresponding column on the haptic feedback device based on the joint angle information, and activates the multiple contact points in the corresponding column to squeeze the user's scalp in a preset time sequence according to the sequence encoding of the movement direction; wherein the contact points squeeze the user's scalp for a preset duration.

5. The multimodal haptic feedback system according to claim 4, characterized in that, In the aforementioned contact stimulation method, it also includes: Multiple contact points are arranged on the helmet of the haptic feedback device. The control processing unit obtains the current palm temperature based on the palm temperature information, converts the palm temperature into Arabic numerals, and distributes them to multiple contact points so that the multiple contact points can be used to write on the user's scalp.

6. The multimodal haptic feedback system according to claim 1, characterized in that, The tactile stimulation methods include vibration stimulation methods: The control processing unit calculates the normal force and tangential force based on the three-dimensional force information of the fingertip, and calculates the slip ratio based on the normal force and tangential force. The normal force is linearly mapped to the vibration amplitude A: in, Indicates the normal force at the fingertip; Indicates the upper limit of the normal force range; The slip ratio is mapped to the vibration frequency f: Where S represents the slip ratio; Based on the combination of the vibration amplitude A and the vibration frequency f, multiple high-frequency vibration modules are controlled to comprehensively feed back multi-dimensional force information from the fingertips.

7. The multimodal haptic feedback system according to claim 6, characterized in that, The calculation of the normal force and tangential force based on the aforementioned three-dimensional force information of the fingertip is as follows: in, Let be the three-dimensional force vector at the fingertip of the j-th finger at time t. Let be the unit normal vector of the fingertip contact surface of the j-th finger; Represented as Transpose of; in, The square of the lateral force on the left and right sides of the fingertips; The square of the tangential force at the front and back of the fingertip; It is the square of the normal force applied vertically by the fingertip.

8. The multimodal haptic feedback system according to claim 6, characterized in that, The vibration stimulation method also includes: When the slip ratio is greater than or equal to the preset slip ratio threshold, the multiple high-frequency vibration modules are controlled to perform emergency vibration according to the preset amplitude and preset frequency.

9. The multimodal haptic feedback system according to claim 1, characterized in that, The helmet of the haptic feedback device is also equipped with a speaker. The control processing unit analyzes the current three-dimensional force information of the fingertips and the temperature information of the palm to determine the auditory stimulation mode according to the relationship between the preset multimodal information and the auditory stimulation mode, and controls the speaker to work based on the auditory stimulation mode so as to provide auditory feedback to the user's head using the speaker.

10. The multimodal haptic feedback system according to claim 9, characterized in that, The auditory stimulation methods include: The temperature alarm stimulation method controls the speaker to emit sound in the first working mode when the palm temperature obtained based on the palm temperature information is greater than the preset maximum temperature or less than the preset minimum temperature. The slip alarm stimulation mode controls the speaker to emit sound in the second working mode when the slip rate obtained based on the three-dimensional force information of the fingertip is greater than or equal to the preset slip rate alarm value.

Images