Wearable multi-mode touch rendering device, system and method

Abstract

The invention discloses a wearable multi-mode tactile rendering device, system and method, and belongs to the field of tactile feedback, and the device processes a to-be-rendered multi-channel multi-mode tactile signal through a main controller, converts the to-be-rendered multi-channel multi-mode tactile signal into a control instruction for driving a vibration feedback module and a force feedback module, and outputs the control instruction to the vibration feedback module and the force feedback module. The main controller has the functions of data analysis, time sequence management, synchronous scheduling and the like at the same time, and a multi-channel multi-mode tactile signal to be rendered can be quickly mapped into tactile output in the wearable glove through the main controller, so that real-time interaction of a virtual object or a remote operation environment is realized; a nonlinear vibration signal processing method is adopted by the main controller, and the problem of perception distortion caused by linear mapping of vibration signals is solved; a unified scheduling mechanism is adopted, a time synchronization constraint mark is introduced, the consistency of vibration feedback and force feedback in a time domain is ensured, and the problem of time sequence mismatch caused by multi-mode independent control is solved.

Description

Technical Field

[0001] This invention belongs to the field of haptic feedback, and more specifically, relates to a wearable multimodal haptic rendering device, system and method. Background Technology

[0002] In human-computer interaction, high-fidelity haptic feedback is key to achieving an immersive haptic experience. Since humans primarily perceive the physical world through vibration and force, vibrational and force-based haptic feedback have been extensively studied in human-computer interaction.

[0003] In the field of vibration-haptic feedback, linear vibration actuators are widely used due to their fast response speed and compact structure. However, existing vibration rendering schemes mainly use linear amplitude mapping for signal processing, which fails to consider the fundamental law that human tactile perception follows a power law. This deficiency leads to weak signal events being easily overwhelmed by strong signals when rendering complex tactile events, while strong signal events are prone to perceptual saturation, making it difficult to accurately reproduce realistic tactile sensations. While vibration feedback can effectively simulate surface textures and collisions, it struggles to simulate the continuous pressure or resistance experienced during grasping.

[0004] In terms of force and tactile feedback, pneumatic drive solutions have unique advantages in simulating continuous pressure and dynamic impedance due to their natural compliance and direct force generation mechanism. Compared with rigid motor drives, pneumatic actuators can achieve gentle and smooth force output without complex transmission mechanisms, which is more in line with the mechanical characteristics of human tissues when gripping, and their lightweight characteristics are more suitable for the portability requirements of wearable devices.

[0005] However, in existing multimodal systems, the vibration actuator and the force feedback module are usually driven by separate controllers. Due to the lack of a unified clock reference and a unified scheduling mechanism, time delay differences and timing mismatches inevitably occur between the vibration signal and the force feedback drive signal. This sense of timing disconnect severely damages the user's immersive experience and makes it impossible to form a unified and realistic tactile perception. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the prior art, the present invention provides a wearable multimodal haptic rendering device, system and method, thereby solving the problem of perception distortion caused by linear mapping of vibration signals and the problem of timing mismatch caused by multimodal independent control.

[0007] To achieve the above objectives, according to a first aspect of the present invention, a wearable multimodal haptic rendering device is provided, comprising: Wearable gloves; The vibration feedback module includes a multi-channel multiplexer, N vibration drive modules and N linear vibration actuators connected in a one-to-one correspondence, forming N vibration feedback channels; N≥1; The force feedback module includes N pneumatic drive modules, N air pumps, N pneumatic pipelines and N flexible airbags that are connected one-to-one, forming N force feedback channels; The N vibration feedback channels and N force feedback channels correspond one-to-one to form N sets of channels; the N linear vibration actuators and N flexible airbags are distributed one-to-one at the N fingers of the wearable glove; The main controller includes a first processing unit and a second processing unit; The first processing unit is used to process the vibration tactile signals and force tactile signals to be rendered in each group of channels to obtain vibration feedback drive signals and force feedback drive signals with time synchronization markers. Wherein, both the vibration tactile signal to be rendered and the force tactile signal to be rendered are digital signals; The processing includes: performing data standardization on the vibration tactile signal and the force tactile signal to be rendered, converting them into digital signals with a unified time base and numerical dimensions; sequentially filtering and amplitude normalizing the vibration tactile signal to be rendered, and simultaneously extracting its envelope features and instantaneous phase information to obtain the envelope signal and instantaneous phase; downsampling the envelope signal to adapt it to the operating frequency of the linear vibration actuator; quantizing the downsampled envelope signal to adapt it to the digital format of the linear vibration actuator to obtain the vibration feedback drive signal; processing the force tactile signal to be rendered to obtain the force feedback drive signal; identifying the tactile feature points of the vibration signal based on the envelope signal and instantaneous phase, the tactile feature points including the intensity peak point of the transient vibration signal or the starting point of the continuous vibration signal; calculating the advance triggering amount of the force feedback drive signal relative to the tactile feature points, and adding a time synchronization mark to the force feedback drive signal and the vibration feedback drive signal based on the advance triggering amount; The second processing unit is used to send the vibration feedback drive signals and force feedback drive signals with time synchronization markers from each group of channels to the vibration feedback module and the feedback module, respectively, so that the linear vibration actuators and flexible airbags of the corresponding channels perform vibration tactile feedback and force tactile feedback, respectively, to complete the rendering of the vibration tactile signals and force tactile signals.

[0008] According to a second aspect of the present invention, a multimodal haptic rendering method is provided, applied to a wearable multimodal haptic rendering device as described in the first aspect, comprising: The vibration tactile signal and force tactile signal to be rendered are subjected to data standardization processing, converting them into digital signals with a unified time base and numerical dimensions. The vibration tactile signal to be rendered is then filtered and amplitude normalized sequentially, while its envelope features and instantaneous phase information are extracted to obtain the envelope signal and instantaneous phase. The envelope signal is downsampled to adapt to the operating frequency of the linear vibration actuator, and the downsampled envelope signal is quantized to adapt to the digital format of the linear vibration actuator, resulting in a vibration feedback drive signal. The force tactile signal to be rendered is processed to obtain a force feedback drive signal. Based on the envelope signal and instantaneous phase, tactile feature points of the vibration signal are identified, including the intensity peak point of the transient vibration signal or the starting point of the continuous vibration signal. The advance triggering amount of the force feedback drive signal relative to the tactile feature points is calculated, and a time synchronization marker is added to the force feedback drive signal and vibration feedback drive signal based on this advance triggering amount. The vibration feedback drive signal and force feedback drive signal with time synchronization mark of each group of channels are sent to the vibration feedback module and the feedback module respectively, so that the linear vibration actuator and flexible airbag of the corresponding channel perform vibration tactile feedback and force tactile feedback respectively, and complete the rendering of the vibration tactile signal and the force tactile signal.

[0009] According to a third aspect of the present invention, a wearable multimodal haptic rendering system is provided, including a data interface module, a host computer, and a wearable multimodal haptic rendering device as described in the first aspect. The data interface module is used to receive the vibration tactile signal and the force tactile signal to be rendered and perform analog-to-digital conversion on them. The host computer is used to send the analog-to-digital converted vibration tactile signals and force tactile signals to be rendered to the wearable multimodal tactile rendering device for rendering.

[0010] According to a fourth aspect of the present invention, a multimodal haptic rendering method is provided, applied to a wearable multimodal haptic rendering system as described in the second aspect, comprising: After receiving and converting the vibration tactile signal and force tactile signal to be rendered from analog to digital, the system performs data standardization to convert them into digital signals with a unified time base and numerical dimensions. The vibration tactile signal to be rendered is then filtered and amplitude normalized sequentially, while its envelope features and instantaneous phase information are extracted to obtain the envelope signal and instantaneous phase. The envelope signal is downsampled to match the operating frequency of the linear vibration actuator, and then quantized to match the digital format of the linear vibration actuator, resulting in a vibration feedback drive signal. The force tactile signal to be rendered is processed to obtain a force feedback drive signal. Based on the envelope signal and instantaneous phase, tactile feature points of the vibration signal are identified, including the intensity peak of the transient vibration signal or the starting point of the continuous vibration signal. The advance triggering amount of the force feedback drive signal relative to the tactile feature points is calculated, and a time synchronization marker is added to both the force feedback drive signal and the vibration feedback drive signal based on this advance triggering amount. The vibration feedback drive signal and force feedback drive signal with time synchronization mark of each group of channels are sent to the vibration feedback module and the feedback module respectively, so that the linear vibration actuator and flexible airbag of the corresponding channel perform vibration tactile feedback and force tactile feedback respectively, and complete the rendering of the vibration tactile signal and the force tactile signal.

[0011] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: The wearable multimodal tactile rendering device provided by this invention integrates vibration feedback and aerodynamic feedback into a wearable glove, enabling multi-channel, multimodal tactile feedback. A main controller processes the multi-channel, multimodal tactile signals to be rendered, converting them into control commands to drive the vibration feedback and force feedback modules. This main controller also possesses functions such as data parsing, timing management, and synchronization scheduling. Through this main controller, the multi-channel, multimodal tactile signals to be rendered can be quickly mapped to tactile outputs in the wearable glove, thereby achieving real-time interaction with virtual objects or remote operating environments. The main controller employs a nonlinear vibration signal processing method, solving the perceptual distortion problem caused by linear mapping of vibration signals. A unified scheduling mechanism is adopted, introducing time synchronization constraint markers to ensure the consistency of vibration feedback and force feedback in the time domain, resolving the timing mismatch problem caused by independent multimodal control.

[0012] As a further preferred embodiment, the main controller in the device provided by this invention also performs nonlinear mapping processing on the envelope signal. By amplifying the weak-amplitude vibration signal and appropriately compressing the strong-amplitude vibration signal, the overall distribution of the vibration signal is optimized, ensuring it remains within the range of human tactile perception. This makes the dynamic range of the output signal conform to the characteristics of human tactile perception. This mapping strategy helps to enhance the perception of low-amplitude vibration signals and appropriately suppress strong-amplitude segments, improving the user's subtle perception of vibrational tactile stimulation.

[0013] In summary, the wearable multimodal haptic rendering device provided by this invention has the following advantages: Multimodal integration: Simultaneously achieving high-frequency vibration feedback and low-frequency pneumatic feedback in a single wearable device, enriching the forms of tactile expression.

[0014] High-precision rendering: The vibration feedback module adopts closed-loop drive and independent channel control, and the force feedback module adopts closed-loop control. Through pneumatic pressure adjustment, high-resolution and multi-layered tactile rendering is achieved.

[0015] Low latency and high synchronization: Low latency and multi-channel synchronization are guaranteed through optimized processing architecture and communication links.

[0016] In line with human sensory characteristics: Through sensory optimization and signal optimization, the output signal is made closer to the human tactile experience in terms of sensory dimension, and the consistency between channels is improved.

[0017] The wearable multimodal haptic rendering system provided by this invention features a flexible deployment architecture, allowing for the selection of different implementation paths based on application requirements. In remote control or cloud-based interaction scenarios, signal processing is implemented through an external lower-level machine independent of the wearable multimodal haptic rendering device. This approach has the advantage of offloading complex computations to an external device, reducing glove power consumption and extending battery life, while utilizing the lower-level machine as a relay node to ensure the stability of long-distance communication. In close-range scenarios, signal processing can be directly implemented through the main controller within the wearable multimodal haptic rendering device. This approach eliminates communication latency in intermediate transmission stages and improves device portability. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the structure of the wearable multimodal haptic rendering device provided in an embodiment of the present invention; Figure 2 A schematic diagram of the vibration feedback module provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a wearable multimodal haptic rendering system provided in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the workflow of a wearable multimodal haptic rendering system provided in an embodiment of the present invention. Figure 5 This is a flowchart of a wearable multimodal haptic rendering method provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0020] This invention provides a wearable multimodal haptic rendering device, such as... Figure 1 As shown, it includes: Wearable gloves; The vibration feedback module includes a multi-channel multiplexer, N vibration drive modules and N linear vibration actuators connected in a one-to-one correspondence, forming N vibration feedback channels; N≥1; The force feedback module includes N pneumatic drive modules, N air pumps, N pneumatic pipelines and N flexible airbags that are connected one-to-one, forming N force feedback channels; The N vibration feedback channels and N force feedback channels correspond one-to-one to form N sets of channels; the N linear vibration actuators and N flexible airbags are distributed one-to-one at the N fingers of the wearable glove; The main controller includes a first processing unit and a second processing unit; The first processing unit is used to process the vibration tactile signals and force tactile signals to be rendered in each group of channels to obtain vibration feedback drive signals and force feedback drive signals with time synchronization markers. Wherein, both the vibration tactile signal to be rendered and the force tactile signal to be rendered are digital signals; The processing includes: performing data standardization on the vibration tactile signal and the force tactile signal to be rendered, converting them into digital signals with a unified time base and numerical dimensions; sequentially filtering and amplitude normalizing the vibration tactile signal to be rendered, and simultaneously extracting the envelope features and instantaneous phase information of the normalized signal to obtain the envelope signal and instantaneous phase; downsampling the envelope signal to adapt it to the operating frequency of the linear vibration actuator; quantizing the downsampled envelope signal to adapt it to the digital format of the linear vibration actuator to obtain the vibration feedback drive signal; processing the force tactile signal to be rendered to obtain the force feedback drive signal; identifying the tactile feature points of the vibration signal based on the envelope signal and instantaneous phase, the tactile feature points including the intensity peak point of the transient vibration signal or the start time point of the continuous vibration signal; calculating the advance triggering amount of the force feedback drive signal relative to the tactile feature points, and adding a time synchronization mark to the force feedback drive signal and the vibration feedback drive signal based on the advance triggering amount. The second processing unit is used to send the vibration feedback drive signal and force feedback drive signal with time synchronization mark of each group of channels to the vibration feedback module and the feedback module respectively, so that the linear vibration actuator and flexible airbag of the corresponding channel perform vibration tactile feedback and force tactile feedback respectively, thereby realizing the rendering of the vibration tactile signal and the mechanical tactile signal.

[0021] Specifically, the main controller is used to process multi-channel multimodal tactile signals (i.e., vibration tactile signals to be rendered and force tactile signals to be rendered), convert them into appropriate driving signals (i.e. vibration feedback driving signals and force feedback driving signals), and coordinate the synchronous output of the vibration feedback module and the force feedback module.

[0022] The vibration feedback module includes an N-channel multiplexer, N vibration drive modules, and N linear vibration actuators, used to output tactile vibration feedback corresponding to the vibration tactile signal to be rendered. That is, the vibration feedback module outputs n-channel vibration feedback signals in a one-to-one correspondence through n linear vibration actuators, where n=1,2,…,N.

[0023] A multi-channel multiplexer is positioned between the main controller and multiple independent vibration drive modules. The main controller selectively communicates with one or more of the vibration drive modules via the multi-channel multiplexer. The main controller module sends control commands to one or more independent vibration drive modules through the multi-channel multiplexer to achieve independent or synchronous group control of one or more linear vibration actuators.

[0024] Preferably, the vibration drive module adopts a closed-loop drive mode. By adding a current and voltage feedback module of the actuator to the vibration feedback module, closed-loop control of the vibration drive is realized. The vibration drive module supports real-time waveform control to realize independent driving and time-domain characteristic adjustment of the multiple linear vibration actuators.

[0025] The force feedback module includes N pneumatic drive modules, N air pumps, N pneumatic pipelines, and N flexible airbags, which are used to output force tactile feedback corresponding to the force tactile signal to be rendered. That is, the force feedback module outputs n channels of force feedback signals through n flexible airbags, n=1,2,…,N.

[0026] The force feedback module can apply controllable air pressure to multiple finger or palm areas to achieve tactile effects of varying intensities. Its main function is to apply controllable pressure to the hand area to simulate the force effects from virtual objects or a remote environment (i.e., to render the force tactile signal to be rendered). In each force feedback channel, an air pump inflates or deflates a flexible airbag through a drive pipe, causing changes in the airbag's pressure and thus applying external force to the finger or palm area. This method can simulate tactile experiences in interactive scenarios such as grasping, squeezing, or pushing and pulling.

[0027] Preferably, the force feedback module further includes a pressure feedback module (which may be a pressure sensor) for real-time detection of airbag pressure and closed-loop regulation via the main controller for pressure stabilization. That is, by setting a pressure sensor in each force feedback channel and monitoring the pressure applied by the airbag to the finger corresponding to that channel in real time, closed-loop control of each force feedback channel can be achieved: the main controller continuously adjusts the air pump output based on the pressure information fed back by the pressure sensor to ensure that the airbag pressure is consistent with the preset value.

[0028] Both the vibration feedback module and the force feedback module are integrated into the same wearable glove. The overall device employs a flexible design, allowing each module to closely conform to the curvature of the hand, ensuring effective contact between the actuators (i.e., the linear vibration actuator and the flexible airbag) and the skin, while simultaneously improving wearing comfort. A rational structural layout optimizes the distribution of the vibration actuator and airbag across key areas of the hand (such as the fingertips and backs of the fingers), avoiding interference between modules and ensuring synchronized operation of tactile and force feedback. Furthermore, the use of a flexible substrate allows for compact integration of the modules within a limited space, effectively utilizing space and maintaining system synergy.

[0029] That is, the vibration feedback module and the force feedback module are both integrated on the same flexible substrate; the flexible substrate has embedded circuitry that connects the execution module and the main controller; the flexible substrate is adapted to conform to the curved surface of the hand, and the linear vibration actuator and flexible airbag are distributed in key parts of the hand to ensure effective contact between the execution module and the skin and to avoid interference between modules.

[0030] As an example, such as Figure 2 As shown, N linear vibration actuators are distributed one-to-one at the fingertips of the N fingers of the wearable glove, and N flexible airbags are distributed one-to-one on the back of the N fingers of the wearable glove.

[0031] It is understandable that the vibration tactile signal and the force tactile signal to be rendered can have multiple sources, such as local storage, network data stream, or be generated in real time by external devices.

[0032] The first processing unit of the main controller is used to process the vibration tactile signals and force tactile signals to be rendered from each group of channels to obtain vibration feedback drive signals and force feedback drive signals with time synchronization markers, specifically including: S1, perform data standardization processing on the vibration tactile signal and the force tactile signal to be rendered, and convert them into digital signals with a unified time reference and numerical dimensions.

[0033] Understandably, due to the differences in sampling rate and data units between the vibration tactile signals and the force tactile signals to be rendered, direct processing would lead to errors in subsequent timing calculations. Therefore, data standardization processing includes: 1. Time alignment: Map both onto a unified time axis to ensure that the time base of both is consistent when calculating the "early trigger amount" in the subsequent step S4; 2. Numerical normalization: Heterogeneous data with different bit widths (such as 16-bit / 12-bit) are uniformly mapped to standard floating-point numbers (0.0-1.0) so that envelope and phase features can be uniformly extracted in subsequent step S4.

[0034] S2, the following processing is performed on the vibration tactile signal to be rendered after processing by S1: (1) Preprocessing: remove the DC component and high-frequency noise in the vibration signal to ensure the quality of the vibration signal, and then perform amplitude normalization processing; (2) Envelope and phase extraction: use time domain and / or frequency domain analysis methods to extract the details of the tactile intensity change (i.e., the envelope curve of the tactile intensity change with time) and instantaneous phase information of the vibration tactile signal to be rendered after preprocessing, and obtain the envelope signal and instantaneous phase of the vibration tactile signal to be rendered after preprocessing; the envelope feature is used to characterize the change of tactile intensity, and the instantaneous phase information is used for subsequent synchronization scheduling; (3) Downsampling: convert the envelope signal into a frequency range suitable for the execution module, adapt to the working frequency of the actuator, and thus adapt to the working characteristics of the linear vibration actuator, ensuring that the signal can be accurately expressed within the working frequency of the linear vibration actuator; (4) Quantization: convert the envelope signal after downsampling into a digital format suitable for the linear vibration actuator to obtain the vibration feedback drive signal.

[0035] Envelope extraction can employ time-domain energy envelope calculation methods or combine frequency-domain analysis to extract energy distribution across different frequency bands, thereby capturing more precise details of tactile signal changes.

[0036] Preferably, before quantizing the downsampled envelope signal, the method further includes a perception optimization process for the envelope signal, that is, performing nonlinear mapping on the envelope signal. Through nonlinear mapping, the low amplitude portion of the envelope signal is enhanced and its high amplitude portion is compressed, thereby optimizing the tactile intensity distribution to the range that is perceptible to human touch, so that the dynamic range of the output signal conforms to the characteristics of human tactile perception.

[0037] S3, the force tactile signal to be rendered after processing by S1 is processed as follows: establish the correspondence between the force tactile signal to be rendered and the physical force value, map the force tactile signal to be rendered after processing by S1 to the target force value, and obtain the force feedback driving signal.

[0038] It's understandable that digital signals have bit depth; that is, the force tactile signal to be rendered is a multi-bit binary number. The strength of the force tactile signal is represented by a quantized value composed of multiple binary bits. For example, suppose the force sensor used to acquire the force tactile signal has a range of 0-10N, corresponding to an output voltage of 0-3.3V. Since the acquired force tactile signal is an analog signal, it needs to be converted into a digital signal. Assuming the ADC precision is 12 bits, the mapping process is as follows: No pressure state: The force sensor outputs 0V, and after conversion, the digital signal quantization value is 0, so the target force value (i.e. the target pressure generated by the flexible airbag on the finger) is 0N; Under mild pressure: The force sensor outputs 1.65V (half range), and the quantized value of the converted digital signal is 2048, so the target force value is 5N; Full-scale pressure condition: The force sensor outputs 3.3V, and the converted digital signal quantization value is 4095, so the target force value is 5N.

[0039] If the force feedback module adopts a closed-loop control method, the quantized force value fed back by the pressure sensor is calibrated as the current force value, and the control deviation is calculated based on the target force value and the current force value to obtain the force feedback drive signal.

[0040] S4, performing phase-compensation-based synchronization scheduling on the vibration feedback drive signal and the force feedback drive signal, specifically including: using the instantaneous phase and envelope information of the vibration tactile signal to be rendered, identifying the tactile feature points of the vibration tactile signal to be rendered, wherein the tactile feature points include the intensity peak point of the transient signal or the start time point of the continuous signal; combining the response hysteresis characteristics of the force feedback module, calculating the advance triggering amount of the force feedback drive signal relative to the tactile feature points; adding a synchronization time marker to the force feedback drive signal and the vibration feedback drive signal according to the advance triggering amount, for controlling the pneumatic drive unit to act before the linear vibration actuator, so that the pressure establishment time of the force feedback is aligned with the tactile feature point in the perception timing.

[0041] To more intuitively illustrate the specific process of calculating the advance trigger amount based on response hysteresis characteristics, let's take a virtual grasping interaction scenario as an example: Assume that the main controller identifies the tactile feature point representing the instant the finger contacts the virtual object (i.e., the grasping start point) at 300ms on the time axis based on the vibration tactile signal to be rendered. At this time, by consulting the pre-calibrated hysteresis characteristic table of the force feedback module, it can be seen that the physical rise time required for the force feedback module to inflate the flexible airbag from its current state to generate the target grasping resistance (i.e., the target force value) is 60ms (i.e., the response hysteresis of the force feedback module), while the inherent response delay of the linear vibration actuator simulating the contact collision sensation is only 5ms. Based on this, the main controller determines the triggering sequence of the force feedback drive signal and the vibration feedback drive signal by working backward, calculates the advance trigger amount of the force feedback drive signal relative to the tactile feature point, and thus adds a time synchronization mark to the force feedback drive signal and the vibration feedback drive signal: the triggering time of the force feedback drive signal is set to 240ms, and the triggering time of the vibration feedback drive signal is set to 295ms. Through this scheduling, the main controller controls the pneumatic drive unit to act 55ms ahead of the vibration actuator, so that the gripping resistance established by the airbag and the contact feedback simulated by the vibration are physically precisely converged at the tactile feature point of 300ms. This ensures that the user can simultaneously feel 'hardness' and 'impact' the moment they touch the virtual object, achieving alignment of perception timing.

[0042] Understandably, the hysteresis characteristics of the force feedback module can be obtained through a pre-calibrated hysteresis characteristic table of the force feedback module, which records the time required for the gripping resistance generated by the flexible airbag on the finger to change from the current state (i.e., a certain force value) to the target state (i.e., a certain target force value).

[0043] The second processing unit sends the vibration feedback drive signal and force feedback drive signal with time synchronization mark of each group of channels to the vibration feedback module and the feedback module respectively, so that the linear vibration actuator and flexible airbag of the corresponding channel perform vibration tactile feedback and force tactile feedback respectively, and complete the two types of tactile output of vibration and pneumatic pressure.

[0044] Through the unified scheduling of the first and second processing units, vibration feedback and force feedback can be kept in sync in time, achieving composite tactile feedback.

[0045] The wearable multimodal haptic rendering device provided by the present invention will be further illustrated below with a specific example.

[0046] In this example, the main controller uses an ESP32-WROOM-32E-N8 microcontroller, which features Wi-Fi and Bluetooth communication capabilities and can process control signals from external devices. The main controller is responsible for receiving input signals, parsing, processing, and scheduling them, while ensuring the timing synchronization of the device and coordinating the operation of vibration and force feedback. All control signals, feedback commands, and timing management are scheduled by the main controller, which communicates with other modules via the I2C bus to ensure data exchange and system coordination. The main controller is also responsible for signal parsing and timing scheduling to ensure that vibration and force feedback can be executed in real time and synchronously.

[0047] In this example, the vibration feedback module consists of multiple linear vibration actuators and a DRV2605L driver chip. Figure 2 As shown, each LRA actuator is controlled by an independent drive module, capable of generating vibration feedback of different frequencies and intensities based on the control signal. The DRV2605L drive chip allows for adjustment of vibration frequency and intensity, ensuring the response time meets human perception thresholds and providing precise vibration feedback.

[0048] Because each DRV2605L drive module may experience conflicts on the control bus, a PCA9548 multiplexer is used in this example. This multiplexer is positioned between the main controller and the multiple independent drive modules. By configuring this multiplexer, the main controller can selectively establish communication links with one or more of the independent drive modules, thereby enabling independent control of a single LRA actuator or synchronous group control of multiple LRA actuators (understandably, if the vibration tactile signal to be rendered is a single-channel signal, then the corresponding single LRA actuator is controlled independently; if the vibration tactile signal to be rendered is an n-channel signal, then the corresponding n LRA actuators are controlled synchronously in a one-to-one correspondence).

[0049] Each LRA actuator's drive voltage range covers its rated operating voltage range, ensuring efficient and precise vibration control. Vibration feedback signals are transmitted to each LRA actuator via an I2C bus, allowing for adjustment of vibration frequency and intensity in the time domain. Different tactile effects, such as surface texture or impact feedback, are simulated through PWM duty cycle modulation and waveform control, providing a nuanced tactile experience.

[0050] In this example, the force feedback module achieves force feedback through an air pump, pneumatic tubing, and a flexible airbag. The main controller uses a CS25N06C4N channel MOSFET and a TB6612FNG driver chip to control the on / off state of the air pump and solenoid valve, thereby controlling the airbag pressure. By adjusting the airbag pressure, the system simulates interactive actions such as grasping, squeezing, or pushing / pulling virtual objects. Pressure sensors monitor the pressure exerted on the hand by the airbag in real time and feed the data back to the main controller. The main controller adjusts the airbag pressure based on the real-time pressure feedback to ensure the stability and consistency of the force feedback signal. Each finger of the wearable glove has its own airbag connected to the air pump via an independent pneumatic tubing, enabling independent control of each finger. The system can simulate physical tactile effects such as grasping, squeezing, or pushing / pulling according to the user's interaction needs, providing a more realistic feedback experience.

[0051] In this example, a 3.7V lithium battery is used for power supply, along with an MT3608 boost converter and an SC662K buck converter to provide 5V and 3.3V voltages respectively, ensuring the stability and continuous operation of the device during long-term use. The battery has a rechargeable design, ensuring continuous power supply during prolonged haptic feedback operation and ensuring the device's high-efficiency operation.

[0052] In this example, both the force feedback actuator and the vibration feedback actuator are integrated onto a flexible substrate. This flexible substrate conforms closely to the curvature of the hand, making the device both comfortable and secure to wear. Through a well-designed layout, the device achieves integrated and coordinated vibration and force feedback within a limited space, while simultaneously enhancing wearing comfort. This design improves wearing comfort while ensuring full contact between the sensors and actuators and the skin, providing precise tactile feedback.

[0053] When a user inputs vibration and force tactile signals to be rendered via an external device, the main controller receives and parses these signals, converting them into commands that can drive vibration and force feedback. The main controller exchanges data with other modules via the I2C bus to ensure signal synchronization and stable operation of each module.

[0054] The vibration and force haptic signals to be rendered can also come from a virtual reality headset or a computer. The host computer generates corresponding raw haptic data based on the user's interactive operation commands (such as grasping) in the virtual environment, and sends it as the signals to be rendered to the main controller. The main controller receives and parses these signals, converting them into commands that can drive vibration and force feedback.

[0055] The vibration feedback and force feedback signals must maintain a high degree of consistency in the time domain to avoid a perceived disconnect in the signal for the user. The main controller implements synchronous scheduling of multi-channel signals to ensure the temporal consistency of vibration feedback and force feedback. This device can achieve high temporal resolution synchronization, ensuring coordinated output of different types of tactile signals.

[0056] This device ensures coordinated operation between its various modules through hardware design and signal scheduling. The vibration feedback module outputs vibration feedback through multiple LRA actuators to simulate the surface tactile sensation of virtual objects; the force feedback module adjusts the air pressure of the airbags through an air pump to simulate interactive processes such as grasping, squeezing, or pushing and pulling. All these operations are coordinated through scheduling by the main controller to ensure that the user can experience a harmonious tactile interaction.

[0057] This invention provides a multimodal haptic rendering method, applied to a wearable multimodal haptic rendering device as described in any of the above embodiments, comprising: The vibration tactile signal and force tactile signal to be rendered are subjected to data standardization processing, converting them into digital signals with a unified time base and numerical dimensions. The vibration tactile signal to be rendered is then filtered and amplitude normalized sequentially, while its envelope features and instantaneous phase information are extracted to obtain the envelope signal and instantaneous phase. The envelope signal is downsampled to adapt to the operating frequency of the linear vibration actuator, and the downsampled envelope signal is quantized to adapt to the digital format of the linear vibration actuator, resulting in a vibration feedback drive signal. The force tactile signal to be rendered is processed to obtain a force feedback drive signal. Based on the envelope signal and instantaneous phase, tactile feature points of the vibration signal are identified, including the intensity peak point of the transient vibration signal or the starting point of the continuous vibration signal. The advance triggering amount of the force feedback drive signal relative to the tactile feature points is calculated, and a time synchronization marker is added to the force feedback drive signal and vibration feedback drive signal based on this advance triggering amount. The vibration feedback drive signal and force feedback drive signal with time synchronization mark of each group of channels are sent to the vibration feedback module and the feedback module respectively, so that the linear vibration actuator and flexible airbag of the corresponding channel perform vibration tactile feedback and force tactile feedback respectively, and complete the rendering of the vibration tactile signal and the force tactile signal.

[0058] This invention provides a wearable multimodal haptic rendering system, which covers the complete link from receiving, processing, and transmitting multimodal haptic signals to execution feedback, including a data interface module, a host computer, and a wearable multimodal haptic rendering device as described in any of the above embodiments; The data interface module is used to receive the vibration tactile signal and the force tactile signal to be rendered and perform analog-to-digital conversion on them. The host computer is used to send the analog-to-digital converted vibration tactile signals and force tactile signals to be rendered to the wearable multimodal tactile rendering device for rendering.

[0059] Preferably, the system further includes a lower-level machine, the function of which is the same as that of the first processing unit of the main controller in the wearable multimodal tactile rendering device. In remote control or cloud interaction scenarios, the lower-level machine replaces the first processing unit to process the vibration tactile signals and force tactile signals to be rendered in each group of channels, so as to obtain vibration feedback drive signals and force feedback drive signals with time synchronization markers.

[0060] The system provided by the present invention will be further illustrated below with a specific example.

[0061] Taking the system acquiring raw tactile signals related to tactile interaction (i.e., vibration tactile signals and force tactile signals to be rendered) through the signal acquisition module as an example, such as... Figures 3-4 As shown, the system includes a signal acquisition module (i.e. Figure 4 The system includes an acquisition terminal that collects real vibration tactile signals through an accelerometer, a real force tactile signal through a force sensor, a data interface module, a host computer, a slave computer, and a wearable multimodal tactile device. Through the collaborative work of these modules, a complete closed loop is achieved from tactile signal acquisition, transmission, processing to feedback output, ensuring that users receive coordinated tactile feedback during interaction.

[0062] The host computer, acting as a forwarding module, is responsible for buffering, packaging, and forwarding the received digital signals without altering the semantics and order of the original signals, ensuring the integrity and real-time performance of the data stream. The packaged data is then transmitted to the lower-level control module via a communication link. This link can be either a wired serial port or a wireless communication method, such as Wi-Fi or Bluetooth, to adapt to the bandwidth and latency requirements of different application scenarios.

[0063] In close-range scenarios, the main controller of the wearable multimodal haptic device is the processing core of the system, used to process the data transmitted from the host computer in real time, which can eliminate communication delays in intermediate transmission links and improve the portability of the device.

[0064] In remote control or cloud-based interaction scenarios, the lower-level machine is the processing core of the system, replacing the first processing unit in the main controller of the wearable multimodal haptic device to process data transmitted from the upper-level machine in real time. This replacement method can offload complex calculations to external devices to reduce the power consumption of the wearable multimodal haptic device and extend its battery life, while using the lower-level machine as a relay node to ensure the stability of long-distance communication.

[0065] In terms of signal scheduling, the main controller or lower-level machine is responsible for the timing coordination of multi-channel output. Through a precise clock management mechanism, the vibration feedback and force feedback are strictly aligned on the time axis, ensuring that the tactile response felt by the user is seamless. Whether it is continuous friction, intermittent tapping, or gripping, the system can guarantee the synchronous output of feedback signals on each actuator, improving the continuity of interaction.

[0066] The wearable multimodal haptic device serves as the actuator, integrating multiple sensors, a drive module, a linear vibratory actuator, an air pump, a solenoid valve, and flexible airbags via a flexible substrate. When worn on the user's hand, the LRA vibratory actuator at the fingertips generates localized vibrations based on drive signals, mimicking the texture of an object's surface or the effect of impact. Pneumatic tubing extends from the wrist to the five fingers, and the air pump controls the inflation and deflation of the airbags at each finger via solenoid valves, simulating the mechanical resistance generated by grasping or pressing. Pressure sensors feed pressure information back to the pneumatic drive module, ensuring the stability and consistency of air pressure regulation. The entire device is compact and conforms to the curvature of the hand, ensuring effective contact between the actuator and the skin while minimizing the feeling of restriction during wear.

[0067] like Figure 4 As shown, the system achieves a complete link from real-time acquisition and digital processing of tactile signals to synchronous output through the coordinated operation of modules, ensuring that the system can stably complete the entire process of acquisition, transmission and feedback, and providing a reliable hardware and software foundation for subsequent tactile rendering methods.

[0068] This invention provides a multimodal haptic rendering method, applied to a wearable multimodal haptic rendering system as described in any of the above embodiments, comprising: After receiving and converting the vibration tactile signal and force tactile signal to be rendered from analog to digital, the system performs data standardization to convert them into digital signals with a unified time base and numerical dimensions. The vibration tactile signal to be rendered is then filtered and amplitude normalized sequentially, while its envelope features and instantaneous phase information are extracted to obtain the envelope signal and instantaneous phase. The envelope signal is downsampled to match the operating frequency of the linear vibration actuator, and then quantized to match the digital format of the linear vibration actuator, resulting in a vibration feedback drive signal. The force tactile signal to be rendered is processed to obtain a force feedback drive signal. Based on the envelope signal and instantaneous phase, tactile feature points of the vibration signal are identified, including the intensity peak of the transient vibration signal or the starting point of the continuous vibration signal. The advance triggering amount of the force feedback drive signal relative to the tactile feature points is calculated, and a time synchronization marker is added to both the force feedback drive signal and the vibration feedback drive signal based on this advance triggering amount. The vibration feedback drive signal and force feedback drive signal with time synchronization mark of each group of channels are sent to the vibration feedback module and the feedback module respectively, so that the linear vibration actuator and flexible airbag of the corresponding channel perform vibration tactile feedback and force tactile feedback respectively, and complete the rendering of the vibration tactile signal and the force tactile signal.

[0069] The above method will be further illustrated with a specific example below.

[0070] This method encompasses the entire process from standardization, preprocessing, feature extraction, amplitude mapping, signal generation to multi-channel synchronous scheduling and feedback output. While ensuring real-time performance, this method utilizes multi-stage signal processing and precise control strategies to enable stable and synchronous output of vibration and force feedback.

[0071] In remote control or cloud-based interaction scenarios, the lower-level machine performs signal processing after receiving the raw signal. In this example, signal format unification is the first step, converting data from different channels into a unified standard structure to facilitate subsequent batch processing. Next, amplitude normalization is performed to eliminate errors caused by inconsistencies in amplitude from different sensors. Then, the vibration and tactile signal to be rendered is subjected to bandpass filtering to remove DC components and high-frequency noise. Common implementations include Butterworth filters and rolling average methods to ensure a pure and smooth input signal.

[0072] After preprocessing the vibration tactile signal to be rendered, the feature extraction and rendering stage begins. First, a Hilbert transform is applied to the signal to obtain an analytic signal, separating the instantaneous amplitude characterizing vibration intensity changes from the instantaneous phase used as a time base reference for multimodal synchronization. After obtaining the initial envelope, a piecewise cubic Hermite interpolation algorithm is used to reconstruct the discrete signal in the time domain. This algorithm's monotonicity is utilized to suppress overshoot distortion while establishing a continuous, high-precision waveform reference. Subsequently, a nonlinear mapping is performed on the reconstructed signal based on Stevens' tactile perception law. An exponential transform enhances the discriminability of low-amplitude signals and dynamically compresses high-amplitude signals. Finally, downsampling is performed: periodic signal aggregation is executed to convert the high-sampling-rate mapped signal into a low-frequency drive sequence adapted to the narrowband characteristics of the linear vibration actuator, ensuring that key energy characteristics are preserved while reducing data transmission rate.

[0073] The processed vibration tactile signal to be rendered is used to generate multi-channel drive signals. In this embodiment, the envelope data is converted into a PWM duty cycle sequence or drive waveform and sent to the DRV2605L driver chip via the I2C bus. The latter controls each LRA actuator to output vibration feedback with corresponding intensity and frequency. Simultaneously, the force feedback module adjusts the air pump speed and solenoid valve state based on the processed force tactile signal to be rendered, changing the internal pressure of the airbag to simulate mechanical tactile sensations such as gripping and squeezing. To ensure consistency across different channels, the lower-level computer performs synchronous scheduling through precise clock management, ensuring that vibration feedback and force feedback are output at the same time, avoiding any disconnect in user perception.

[0074] Throughout the execution of this method, the lower-level machine undertakes key functions such as data reception, real-time processing, and task scheduling. In scenarios requiring greater mobility, the ESP32's code compatibility and sufficient performance allow for processing and driving of the received tactile signals. This lower-level machine maintains real-time communication with the upper-level machine, sensors, and actuator modules through multiple communication interfaces such as UART, I2C, and SPI, ensuring a stable and reliable data link. Its internal timer is configured for millisecond-level periodic interrupts to drive signal scheduling tasks, enabling the lower-level machine to synchronize multi-channel outputs with a time resolution of approximately 1ms. The lower-level machine's task scheduler coordinates the output order of vibration feedback and force feedback, ensuring strict timing alignment even in complex interactive scenarios. For force feedback, the lower-level machine monitors the data returned by the pressure sensor in real time and adjusts the PWM duty cycle based on the deviation between the target pressure and the current pressure to control the air pump speed, achieving closed-loop control. For vibration feedback, the lower-level machine generates PWM or waveform indexes based on envelope characteristics and calls the DRV2605L's built-in waveform storage or real-time waveform driving function to ensure that the actuator response time is synchronized with the signal update frequency. Through this multi-task parallel processing and precise clock management, the entire system achieves low-latency, highly synchronized tactile rendering, enabling the vibration and pressure changes felt by the user in the glove to be output in a timely manner, ensuring the naturalness and real-time nature of the tactile experience during interaction.

[0075] This invention provides an electronic device, including: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0076] This invention provides a computer-readable storage medium storing computer instructions that cause a processor to perform the method described in any of the above embodiments.

[0077] This invention provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method described in any of the above embodiments.

[0078] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A wearable multimodal haptic rendering device, characterized in that, include: Wearable gloves; The vibration feedback module includes a multi-channel multiplexer, N vibration drive modules and N linear vibration actuators connected in a one-to-one correspondence, forming N vibration feedback channels; N≥1; The force feedback module includes N pneumatic drive modules, N air pumps, N pneumatic pipelines and N flexible airbags that are connected one-to-one, forming N force feedback channels; The N vibration feedback channels and N force feedback channels correspond one-to-one to form N sets of channels; the N linear vibration actuators and N flexible airbags are distributed one-to-one at the N fingers of the wearable glove; The main controller includes a first processing unit and a second processing unit; The first processing unit is used to process the vibration tactile signals and force tactile signals to be rendered in each group of channels to obtain vibration feedback drive signals and force feedback drive signals with time synchronization markers. Wherein, both the vibration tactile signal to be rendered and the force tactile signal to be rendered are digital signals; The processing includes: performing data standardization on the vibration tactile signal and the force tactile signal to be rendered, converting them into digital signals with a unified time base and numerical dimensions; sequentially filtering and amplitude normalizing the vibration tactile signal to be rendered, and simultaneously extracting its envelope features and instantaneous phase information to obtain the envelope signal and instantaneous phase; downsampling the envelope signal to adapt it to the operating frequency of the linear vibration actuator; quantizing the downsampled envelope signal to adapt it to the digital format of the linear vibration actuator to obtain the vibration feedback drive signal; processing the force tactile signal to be rendered to obtain the force feedback drive signal; identifying the tactile feature points of the vibration signal based on the envelope signal and instantaneous phase, the tactile feature points including the intensity peak point of the transient vibration signal or the starting point of the continuous vibration signal; calculating the advance triggering amount of the force feedback drive signal relative to the tactile feature points, and adding a time synchronization mark to the force feedback drive signal and the vibration feedback drive signal based on the advance triggering amount; The second processing unit is used to send the vibration feedback drive signal and force feedback drive signal with time synchronization mark of each group of channels to the vibration feedback module and the feedback module respectively, so that the linear vibration actuator and flexible airbag of the corresponding channel perform vibration tactile feedback and force tactile feedback respectively, and complete the rendering of the vibration tactile signal and the force tactile signal.

2. The apparatus as claimed in claim 1, characterized in that, Before the first processing unit downsamples the envelope signal to adapt it to the operating frequency of the linear vibration actuator, it further includes: The envelope signal is subjected to nonlinear mapping processing.

3. The apparatus as described in claim 1 or 2, characterized in that, The vibration feedback module also includes an actuator current and voltage feedback module, and the vibration feedback module adopts a closed-loop drive mode. The force feedback module also includes a pressure feedback module, which adopts a closed-loop drive mode.

4. The apparatus as claimed in claim 1, characterized in that, The vibration feedback module and the force feedback module are both integrated on the same flexible substrate, which is fitted to the curved surface of the hand.

5. A multimodal haptic rendering method, applied to the wearable multimodal haptic rendering device as described in any one of claims 1-4, characterized in that, include: The vibration tactile signal and force tactile signal to be rendered are subjected to data standardization processing, converting them into digital signals with a unified time base and numerical dimensions. The vibration tactile signal to be rendered is then filtered and amplitude normalized sequentially, while its envelope features and instantaneous phase information are extracted to obtain the envelope signal and instantaneous phase. The envelope signal is downsampled to adapt to the operating frequency of the linear vibration actuator, and the downsampled envelope signal is quantized to adapt to the digital format of the linear vibration actuator, resulting in a vibration feedback drive signal. The force tactile signal to be rendered is processed to obtain a force feedback drive signal. Based on the envelope signal and instantaneous phase, tactile feature points of the vibration signal are identified, including the intensity peak point of the transient vibration signal or the starting point of the continuous vibration signal. The advance triggering amount of the force feedback drive signal relative to the tactile feature points is calculated, and a time synchronization marker is added to the force feedback drive signal and vibration feedback drive signal based on this advance triggering amount. The vibration feedback drive signal and force feedback drive signal with time synchronization mark of each group of channels are sent to the vibration feedback module and the feedback module respectively, so that the linear vibration actuator and flexible airbag of the corresponding channel perform vibration tactile feedback and force tactile feedback respectively, and complete the rendering of the vibration tactile signal and the force tactile signal.

6. A wearable multimodal haptic rendering system, characterized in that, Includes a data interface module, a host computer, and a wearable multimodal haptic rendering device as described in any one of claims 1-4; The data interface module is used to receive the vibration tactile signal and the force tactile signal to be rendered and perform analog-to-digital conversion on them. The host computer is used to send the analog-to-digital converted vibration tactile signals and force tactile signals to be rendered to the wearable multimodal tactile rendering device for rendering.

7. The system as described in claim 6, characterized in that, It also includes a lower-level machine, which has the same function as the first processing unit. In remote control or cloud interaction scenarios, it replaces the first processing unit to process the vibration tactile signals and force tactile signals to be rendered for each group of channels.

8. A multimodal haptic rendering method, applied to the wearable multimodal haptic rendering system as described in claim 6 or 7, characterized in that, include: After receiving and converting the vibration tactile signal and force tactile signal to be rendered from analog to digital, the system performs data standardization to convert them into digital signals with a unified time base and numerical dimensions. The vibration tactile signal to be rendered is then filtered and amplitude normalized sequentially, while its envelope features and instantaneous phase information are extracted to obtain the envelope signal and instantaneous phase. The envelope signal is downsampled to match the operating frequency of the linear vibration actuator, and then quantized to match the digital format of the linear vibration actuator, resulting in a vibration feedback drive signal. The force tactile signal to be rendered is processed to obtain a force feedback drive signal. Based on the envelope signal and instantaneous phase, tactile feature points of the vibration signal are identified, including the intensity peak of the transient vibration signal or the starting point of the continuous vibration signal. The advance triggering amount of the force feedback drive signal relative to the tactile feature points is calculated, and a time synchronization marker is added to both the force feedback drive signal and the vibration feedback drive signal based on this advance triggering amount. The vibration feedback drive signal and force feedback drive signal with time synchronization mark of each group of channels are sent to the vibration feedback module and the feedback module respectively, so that the linear vibration actuator and flexible airbag of the corresponding channel perform vibration tactile feedback and force tactile feedback respectively, and complete the rendering of the vibration tactile signal and the force tactile signal.

Images