A multimodal haptic feedback interaction system, control method, and storage medium

Abstract

This invention discloses a multimodal haptic feedback interaction system, control method, and storage medium, comprising: a contact-type haptic feedback subsystem, including an electrostatic touchscreen and a piezoelectric vibratory actuator, for providing texture and vibration feedback when a user touches the screen; a non-contact haptic feedback subsystem, including an ultrasonic transducer array arranged in a specific geometric configuration and a driving module, for generating focused aerial haptic feedback in space; an interactive sensing subsystem, including an infrared touchscreen and a visual display module, for tracking the interaction position and synchronizing visual feedback; and a central processing unit, running an interactive logic engine, configured to coordinate the output of corresponding haptic signals by the contact-type and non-contact haptic feedback subsystems according to the user's interactive intent. This invention can significantly improve the interactive experience in fields such as gaming, education, and healthcare, support new interactive modes such as augmented reality and remote collaboration, and bring a stronger sense of user immersion and satisfaction.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology, specifically relating to a multimodal haptic feedback interaction system, control method, and storage medium. Background Technology

[0002] With the rapid development of Human-Computer Interaction (HCI) technology, the shift from traditional keyboard and mouse interaction to Natural User Interface (NUI) has become an inevitable trend. Haptic feedback technology, as a bridge connecting the physical and digital worlds, plays a crucial role in Virtual Reality (VR), Augmented Reality (AR), in-vehicle central control systems, and public self-service terminals. Currently, mainstream haptic feedback technologies are mainly divided into two categories: contact-based haptic feedback and non-contact-based haptic feedback. Among these, the most common contact-based haptic feedback technologies include piezoelectric vibration and electro-vibration. Piezoelectric vibration utilizes the inverse piezoelectric effect of piezoelectric ceramics to simulate a "click" sensation through the overall vibration of the screen. Examples include linear motors (LRA) or eccentric rotor motors (ERM) commonly used in smartphones and automotive screens. However, this type of technology can only provide a single vibrational sensation and cannot simulate the microscopic texture differences of different materials such as silk and sandpaper. Furthermore, large-amplitude vibrations can easily cause the screen structure to loosen. Electro-vibration simulates texture by applying an alternating electric field to the screen surface, changing the friction of the fingertips. While technologies like 3M MicroTouch can achieve nuanced texture perception, their limitation lies in their reliance on physical contact. When the user's finger is not touching the screen, no guidance or feedback is provided, restricting their application in air gesture interaction. The second type of non-contact haptic feedback technology utilizes an ultrasonic transducer array to emit high-frequency sound waves, generating constructive interference at the focal point in the air, creating perceptible radiated pressure. Existing solutions, such as UltraLeap systems, typically employ FPGAs or dedicated DACs to generate complex driving waveforms, which are then used in conjunction with linear power amplifiers to drive a 40kHz transducer array.

[0003] Based on the above existing technologies, at least three technical problems urgently need to be solved: (1) Fragmented interaction modes and lack of coherence. Existing systems cannot provide a coherent tactile experience in the entire interaction chain of "non-contact (air) - transition (approach) - contact (screen)". When users switch from air gestures to touch screen, the tactile feedback often changes abruptly or breaks, which destroys the sense of immersion. (2) Noise and energy efficiency bottlenecks of ultrasonic tactile feedback. Traditional ultrasonic driving schemes (such as fixed 50% duty cycle PWM or linear amplification) have obvious audible noise, low energy efficiency and high heat generation. This not only limits the applicable scenarios of the device (such as quiet libraries or offices), but also increases the hardware heat dissipation cost and system instability. (3) Virtual controls lack physical reality. Simple screen vibration or electrostatic texture is difficult to provide a "confirmation" similar to mechanical buttons. When users perform click or swipe operations, they often have a psychological feeling of misoperation or uncertainty due to the lack of clear tactile anchor points. Summary of the Invention

[0004] The primary objective of this invention is to provide a multimodal haptic feedback interaction system, control method, and storage medium to overcome the aforementioned shortcomings of the prior art.

[0005] To achieve this objective, in a basic implementation, the present invention provides a multimodal haptic feedback interaction system, characterized in that it includes: The tactile feedback subsystem includes an electrostatic touchscreen and a piezoelectric vibratory actuator to provide texture and vibration feedback when the user touches the screen; The non-contact tactile feedback subsystem includes an array of ultrasonic transducers arranged in a specific geometric configuration and a drive module, used to generate focused aerial tactile feedback in space. The interactive perception subsystem includes an infrared touch screen and a visual display module, which are used to track the interaction position and synchronize visual feedback. The central processing unit runs the interaction logic engine and is configured to coordinate the output of corresponding tactile signals by the contact and non-contact tactile feedback subsystems according to the user's interaction intent.

[0006] Furthermore, the driving module in the non-contact tactile feedback subsystem includes: a microcontroller for generating a pulse width modulation (PWM) signal with a frequency of 40 kHz; and an L298N driving module connected to the microcontroller for amplifying the PWM signal to the rated voltage and current levels required by the ultrasonic transducer. The microcontroller is also configured to dynamically adjust the duty cycle of the PWM signal to focus the ultrasonic wave at a specific location and reduce acoustic noise caused by signal modulation.

[0007] Furthermore, the contact-type haptic feedback subsystem is configured to generate a specific piezoelectric vibration waveform in response to user operations on screen controls; the central processing unit runs the Unity engine and Python, wherein the Unity engine is used to render the visual interface, and Python is used to include detecting user interaction, calculating phase difference, serial communication, transmitting instructions, and, upon detecting a touch operation, sending instructions to a signal generator to generate a sine wave signal with a frequency of 200Hz and a peak voltage of 5V to drive the piezoelectric ceramic actuator to generate haptic feedback.

[0008] Furthermore, the ultrasonic transducer array achieves focusing in the following way: multiple transducers in the array receive amplified PWM signals and emit ultrasonic waves. By precisely controlling the phase difference of the signals emitted by each transducer, the sound waves generate constructive interference at a specific spatial point above the array, thereby forming a perceptible tactile focus.

[0009] Furthermore, the interactive sensing subsystem is configured to support the operation of various interactive controls, including but not limited to buttons, knobs, sliders and switches, and the system can call different haptic profiles according to different control types.

[0010] Another object of the present invention is to provide a multimodal haptic feedback control method, applicable to the aforementioned system, characterized by comprising the following steps: Initialization steps: Establish communication between the central processing unit and the signal generator, and initialize the positioning and tracking function of the infrared touch screen; Interaction detection steps: The interactive interface is presented through the visual display module, and the system detects whether the user makes a contact touch or a non-contact approach. Signal processing steps: If contact touch is detected, electrostatic tactile feedback or piezoelectric vibration feedback is triggered to generate the corresponding waveform and voltage; if non-contact proximity is detected, the target focus position is calculated and a PWM signal with the corresponding duty cycle is generated. Drive output steps: The processed signals are sent to the electrostatic touch screen, piezoelectric actuator or ultrasonic array respectively to generate synchronized visual and tactile feedback.

[0011] Furthermore, the piezoelectric vibration feedback processing steps specifically include: Python recognizes user clicks on controls on a touchscreen; In response to this operation, the control signal generator outputs a sine wave signal with a frequency of 200Hz and a peak voltage of 5V; This signal is used to drive a piezoelectric ceramic actuator, allowing the user to simultaneously feel the physical vibration transmitted through the screen while receiving visual feedback.

[0012] Furthermore, the non-contact proximity processing steps specifically include: A 40kHz PWM signal with the same resonant frequency as the ultrasonic transducer is generated using a microcontroller; The PWM signal is amplified using the L298N driver module; Based on the tracked location of the interactive object, the duty cycle of the PWM signal is dynamically adjusted to control the emission phase of each transducer in the ultrasonic array, so as to achieve the convergence of ultrasonic waves at a specific point in space.

[0013] Furthermore, it also includes a feedback smoothing adjustment step: Real-time monitoring of changes in the position of interactive objects and switching of interactive states dynamically adjusts the output parameters of electrostatic tactile feedback, piezoelectric vibration and ultrasonic tactile feedback to ensure a smooth transition when switching between different modal tactile feedback and maintain a consistent user experience.

[0014] Another object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned multimodal haptic feedback control method.

[0015] The beneficial effects of this invention are as follows: First, it achieves seamless coverage and perceptual continuity across the entire interactive space. Existing technologies typically treat contact (touchscreen) and non-contact (air gestures) separately, causing perceptual gaps when users switch between different interaction modes. This invention, through a heterogeneous hardware architecture and collaborative control logic, utilizes electrostatic technology to achieve dynamic tactile feedback, significantly improving the touch interaction experience. It achieves non-contact tactile feedback through an ultrasonic array, supports spatial interactive operations, and integrates real-time infrared tracking and a multimodal feedback system, comprehensively improving the interaction quality in various application scenarios, eliminating the abruptness of switching between different physical principles, and truly achieving a seamless connection between the physical world and the digital interface.

[0016] Secondly, it significantly reduces audible noise in ultrasonic haptic feedback, improving environmental friendliness. Traditional ultrasonic haptic devices use PWM signals with a fixed 50% duty cycle to drive the transducer, which is prone to generating harsh odd-order harmonic noise due to transducer nonlinearity, limiting its application in quiet environments. This invention, by superimposing low-frequency sinusoidal modulation on a reference duty cycle, disperses and diffuses concentrated harmonic energy across a wider frequency band. This spectrum broadening effect effectively suppresses single-frequency howling, achieving "silent ultrasonic haptic feedback" without increasing the cost of additional filtering hardware. This significantly enhances the interactive experience in fields such as gaming, education, and healthcare, supports new interactive modes such as augmented reality and remote collaboration, and brings a stronger sense of user immersion and satisfaction.

[0017] Third, it provides a high-fidelity sense of confirmation based on physiological perception characteristics. Virtual controls often lead to user errors or repetitive operations due to the lack of physical resistance. This invention deeply integrates human biosensory characteristics through a piezoelectric transient feedback mechanism. By experimentally selecting the feedback frequency and accurately matching the optimal response range, it comprehensively improves the interaction quality in various application scenarios. Combined with audiovisual and tactile synchronous control, it solves the disconnect between "vision first, touch lagging" in virtual interaction, significantly improving the certainty and immersion of the interaction. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating the implementation sequence of the technical principles of the present invention; Figure 2 This is a schematic diagram of the ultrasonic transducer array and driving module according to an embodiment of the present invention; Figure 3 This is an exploded view of the assembly of the multimodal haptic feedback interaction system of the present invention; The labels in the diagram are as follows: 1. Infrared sensor panel; 2. Electrostatic feedback touch layer; 3. Display screen; 4. Ultrasonic generator; 5. Arduino; 6. 8 drivers and 1 contactor; 7. Electrostatic / piezoelectric signal generator; 8. Main unit; 9. Housing. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please refer to Figure 1 This invention provides a multimodal haptic feedback interaction system, comprising: The tactile feedback subsystem includes an electrostatic touchscreen and a piezoelectric vibratory actuator to provide texture and vibration feedback when the user touches the screen; The non-contact tactile feedback subsystem includes an array of ultrasonic transducers arranged in a specific geometric configuration and a drive module, used to generate focused aerial tactile feedback in space. The interactive perception subsystem includes an infrared touch screen and a visual display module, which are used to track the interaction position and synchronize visual feedback. The central processing unit runs the interaction logic engine and is configured to coordinate the output of corresponding tactile signals by the contact and non-contact tactile feedback subsystems according to the user's interaction intent.

[0022] The present invention also provides a multimodal haptic feedback control method, applicable to the aforementioned system, comprising the following steps: Initialization steps: Establish communication between the central processing unit and the signal generator, and initialize the positioning and tracking function of the infrared touch screen; Interaction detection steps: The interactive interface is presented through the visual display module, and the system detects whether the user makes a contact touch or a non-contact approach. Signal processing steps: If contact touch is detected, electrostatic tactile feedback or piezoelectric vibration feedback is triggered to generate the corresponding waveform and voltage; if non-contact proximity is detected, the target focus position is calculated and a PWM signal with the corresponding duty cycle is generated. Drive output steps: The processed signals are sent to the electrostatic touch screen, piezoelectric actuator or ultrasonic array respectively to generate synchronized visual and tactile feedback.

[0023] The technical solution of the present invention will be described in detail below through preferred embodiments.

[0024] Example 1 This embodiment provides a multimodal tactile hardware architecture that can simultaneously support two physical mechanisms: electro-vibration and ultrasonic radiation pressure. It aims to solve the problems of single tactile modality, low driving efficiency, and limited interaction space in the existing technology, and achieve full-link coverage from non-contact air interaction to contact-based fine texture perception.

[0025] The hardware architecture is generally divided into three main hardware pathways and a multimodal collaborative control center: an electrostatic haptic feedback pathway, used to adjust the coefficient of friction by regulating the dielectric force between the skin and the screen when the user's finger touches the screen, thereby simulating the tactile feel of various virtual materials (such as silk, sandpaper, metal, etc.); an ultrasonic haptic feedback pathway, used to generate a perceptible haptic focus in the air through an ultrasonic phased array before the user's finger touches the screen, enabling non-contact interactions such as floating buttons and air sliders; a piezoelectric vibration confirmation pathway, used to provide instantaneous, rigid vibration feedback in contact interactions, simulating the tactile feedback of physical button clicks; and a multimodal collaborative control center, which runs the Unity engine and Python. The Unity engine is used to render the visual interface, while Python is used for tasks including detecting user interactions, calculating phase differences, serial communication, and transmitting commands.

[0026] Preferably, the core of the electrostatic tactile feedback path lies in establishing a controllable alternating electric field on the screen surface, utilizing the dielectric effect to adjust the fingertip friction. Signal generation and impedance matching are performed, including: a signal generator: an arbitrary waveform generator, model FY6300-20M, capable of outputting low-frequency modulated signals in the frequency range of 1Hz–1kHz, and supporting arbitrary waveform editing (sine wave, triangle wave, square wave, etc.). A high-voltage amplifier: an ELECDEMO YK_HA190 high-voltage amplifier, whose input impedance is matched to the AWG output and whose output impedance is matched to the capacitive load of the electrostatic screen, capable of amplifying the signal to a maximum of 1000Vpp, thereby forming a sufficiently strong electric field on the glass surface. An electrostatic touchscreen: a model 497-0472498D capacitive touchscreen, whose ITO conductive layer combines touch sensing and tactile electrode functions, and whose surface is covered with a high-dielectric-constant insulating film to ensure safety and enhance electric field penetration.

[0027] Preferably, when a finger slides across the screen surface, the electric field interacts with the moisture and stratum corneum of the skin surface via dielectrophoresis, generating an attractive force perpendicular to the surface, thereby altering the tangential frictional force. By dynamically modulating the frequency, amplitude, and waveform of the AWG output, the following can be simulated: low frequency (~10Hz): simulating the periodic impact sensation of a rough texture; high frequency (~500Hz): simulating the low friction sensation of a smooth surface; complex envelope waveform: simulating the tactile contours of specific patterns or characters. The goal of the ultrasonic haptic feedback pathway is to generate localized high-intensity tactile points in a non-contact state.

[0028] Preferably, such as Figure 2 As shown, preferably, the transducer array configuration is as follows: a 40kHz piezoelectric ultrasonic transducer with a resonant frequency error ≤ ±0.5kHz and a sound pressure level ≥110dB@10cm. The array configuration is an 8×8 planar matrix arrangement with an element spacing of half a wavelength (approximately 4.3mm) to ensure spatial sampling conforms to the Nyquist criterion and avoids grating lobe generation.

[0029] Preferably, the high-efficiency switch driver circuit includes a microcontroller: Arduino Mega 2560, with enough I / O ports to independently drive 64 array elements. The driver module amplifies the microcontroller's logic-level PWM signal to the transducer's rated operating voltage (±12V), significantly improving energy efficiency and reducing heat loss. Dynamic duty cycle modulation: To avoid audible noise caused by odd harmonics generated by a fixed 50% duty cycle, the system introduces a dynamic duty cycle algorithm: D(t) = D base +k×sin(2πf m t), where D base The reference duty cycle is (approximately 45%), k is the modulation coefficient, and f is the base duty cycle. m The modulation frequency is approximately 1 kHz. This algorithm diffuses harmonic energy across a wider frequency band, reducing single-frequency noise.

[0030] like Figure 3 As shown, the modular structure and connections of this multimodal haptic feedback interaction system are arranged according to the logic of "perception layer - processing layer - execution / presentation layer", mainly including: (1) Processing and Coordination Control Layer (Core Hub) – Host 8, running the operating system environment and software, specifically: the computer runs the Unity engine (responsible for visual rendering and outputting the image to the display screen) and Python scripts (responsible for interaction detection, phase difference calculation, and logic coordination). Module Arduino5: as a lower-level microcontroller, it communicates with the computer (Python) via USB serial port. It receives instructions from the computer and is directly responsible for driving various hardware generators (ultrasonic, electrostatic signals) and vibration drivers.

[0031] (2) Perception and Interaction Input Layer (Detecting User Interaction), specifically: Module infrared sensing panel 1, arranged in front of or at the edge of the system. When the user's finger approaches (before touching the screen), the infrared sensing panel 1 detects the proximity signal and transmits the signal to Arduino5 (or directly via computer processing). Arduino5 controls the module's 8 drivers and 1 contactor 6 / piezoelectric vibration unit, which in turn drives the ultrasonic generator 4 to emit ultrasonic waves, triggering the "non-contact interaction mode".

[0032] (3) Tactile feedback execution layer (three major hardware pathways), specifically: Ultrasonic tactile feedback pathway: composed of the ultrasonic generator 4 in the module and the ultrasonic phased array part / or independent ultrasonic transducer array in the module electrostatic feedback touch layer 2. The Arduino 5 controls the ultrasonic generator 4 to drive the transducer array according to the phase difference calculated by Python, forming a tactile focus in the air (floating interaction). Electrostatic tactile feedback pathway: composed of the module electrostatic / piezoelectric signal generator 7 and the module electrostatic feedback touch layer 2 / conductive touch layer. When the user's finger touches the screen, the Arduino 5 controls the electrostatic / piezoelectric signal generator 7 to apply a specific signal to the touch layer, adjust the dielectric force to change the coefficient of friction, and simulate the touch of virtual materials. Piezoelectric vibration confirmation pathway: composed of the module electrostatic / piezoelectric signal generator 7 and the module display screen 3 (with a piezoelectric ceramic sheet attached to its back). When the Arduino 5 detects a contact confirmation action, it drives the piezoelectric unit at the corresponding position to vibrate instantaneously and rigidly, simulating the feel of a physical button click.

[0033] (4) Visual presentation layer, specifically: Module display screen 3: connected to the Unity engine of host 8 through video interfaces such as HDMI / DP, to render and present the visual interactive interface (buttons, sliders, etc.) in real time.

[0034] (5) Structural integration, module shell 9: As an overall mechanical structure, it integrates and fixes the above-mentioned display screen 3 (surface layer), electrostatic feedback touch layer 2 (attached to the screen), infrared sensing panel 1 (edge / front side), ultrasonic generator 4 / transducer array (may be distributed around the screen or on the back), internally installed driver, Arduino 5, electrostatic / piezoelectric signal generator 7 and host 8 (external computer or built-in host).

[0035] Preferably, the system calculates the acoustic path difference Δdi from the focal point to each array element using the Fresnel diffraction formula based on the three-dimensional coordinates (x, y, z) of the finger provided by the infrared positioning module, and applies a phase delay φ to the PWM signal of each channel. i = 2πΔd i / λ, where λ is the wavelength of the ultrasound in air (approximately 8.6 mm). The signals of each array element constructively interfere at the focal point, forming a high-intensity tactile point.

[0036] Preferably, in contact-based interaction, to provide users with a clear sense of confirmation of operation, the system sets up a piezoelectric vibration feedback path: Actuator: PZT-5H piezoelectric ceramic sheet, polarized in the thickness direction, attached to the back panel of the screen. Drive signal: After the Unity engine detects operations such as button clicks, it controls the signal generator to output a sine wave pulse with a frequency of 200Hz and a peak voltage of 5V, lasting for 20ms. Perception effect: This frequency is within the optimal response range of the Pacini body, which can simulate the "click" sensation of a mechanical microswitch and is strictly synchronized with the visual animation (delay <16ms).

[0037] Preferably, the software architecture uses the Unity engine as the interaction logic layer, responsible for: visual rendering (UI controls, 3D models); and running the Unity engine and Python, where the Unity engine is used to render the visual interface, and Python is used for tasks including detecting user interactions, calculating phase differences, serial communication, and transmitting commands. Python also calls the Serial class to communicate with the AWG and Arduino microcontrollers via serial communication.

[0038] Preferably, the software workflow includes: Initialization phase: Establishing a communication connection with the signal generator via Python, initializing the input tracking function of the infrared touchscreen, and configuring the cursor position monitoring module. Cursor position detection: Continuously monitoring the cursor position using the pyautogui library and mapping the coordinates to a preset signal parameter grid or numerical range. Signal generation: Sending instructions to the signal generator based on the current cursor position, configuring the waveform type (square wave), frequency, and amplitude parameters. Signal amplification: Transmitting the signal output from the generator to the signal amplifier, and boosting it to a driving level suitable for the electrostatic touchscreen after gain processing. Feedback adjustment: Dynamically adjusting the signal output in real time to follow changes in the cursor position, ensuring smooth switching of signal parameters and maintaining a consistent tactile experience.

[0039] Preferably, the system defines four interaction states: HOVER: ultrasonic feedback only, guiding the user to approach the target; APPROACH: ultrasonic + electrostatic fade-in / fade-out transition; CONTACT: electrostatic texture + piezoelectric confirmation; RELEASE: tactile fade-out and return to idle state. State switching is jointly determined by infrared positioning data and capacitive touch data, ensuring seamless multimodal feedback.

[0040] This embodiment integrates three feedback pathways—electrostatic, ultrasonic, and piezoelectric—in a heterogeneous manner, and is uniformly scheduled by a main program written in Python. It achieves the following significant effects: Full interactive space coverage: From hovering in the air to screen contact, there is corresponding tactile feedback. Low-noise ultrasonic output: Dynamic duty cycle modulation effectively suppresses audible noise, improving user comfort. High-fidelity texture reproduction: The wideband signal generation capability of the electrostatic tactile pathway enables detailed virtual material simulation. Instant operation confirmation: Piezoelectric vibration feedback provides a rigid click feel, compensating for the insufficient force of pure electrostatic feedback.

[0041] Example 2 In the field of human-computer interaction, virtual buttons or sliders often suffer from frequent accidental touches or repetitive operations due to a lack of physical resistance or tactile feedback. Research shows that the mechanoreceptors in human fingertips are selectively sensitive to vibrational stimuli of different frequencies: the receptor type is the Pacinian corpuscle, the optimal response frequency range is 200–300 Hz, and the sensory characteristics are deep vibration and impact sensation. This embodiment chooses 200 Hz as the dominant frequency for piezoelectric feedback because this frequency is within the optimal response range of the Pacinian corpuscle, which can simulate the "click" sensation of a rigid object colliding without causing pain or discomfort.

[0042] Preferably, when a user operates controls (such as buttons, knobs, sliders) in the Unity engine interactive interface, the system performs the following steps: (1) Interaction triggering and state detection: The infrared touch screen or capacitive screen detects the finger contact event and transmits the touch point coordinates (x, y) to the central processing unit. The Unity engine maps the coordinates to the preferred UI control and determines the interaction type (single click, double click, long press, swipe, etc.); (2) Visual feedback synchronization: The Unity engine immediately renders the corresponding visual feedback on the screen, such as button indentation, slider movement, knob rotation, etc. The visual refresh rate is set to 60fps, and the single frame duration is about 16.67ms, providing a time reference for haptic synchronization. (3) Tactile signal generation: The Unity engine is only used for UI rendering, while Python uses serial communication to send instructions to the signal generator (AWG). The output parameters are configured as follows: waveform type: sine wave, frequency: 200Hz, peak voltage: 5Vpp, pulse width: 20ms. The reason for choosing a sine wave instead of a square wave is that the sine wave spectrum is pure, containing only the fundamental frequency component, avoiding the harshness caused by high-frequency harmonics, and is more in line with the psychoacoustic characteristics of "soft mechanical click". (4) Piezoelectric coupling and vibration transmission: The signal is transmitted to the PZT-5H piezoelectric ceramic sheet mounted on the back of the screen. Using the inverse piezoelectric effect, the electric field energy is converted into mechanical deformation, causing the screen to produce micron-level bending vibration. The vibration is transmitted to the fingertip through the screen structure, and the user obtains a confirmation touch similar to a micro switch.

[0043] Preferably, to avoid the disconnect caused by "visual feedback preceding tactile feedback," this embodiment introduces a multi-sensory synchronization control strategy, including: Delay control: The system strictly controls the time difference between tactile feedback and visual animation to within ±1 frame (16ms). Hardware interrupt triggering: When the Unity engine renders a specific animation frame, it immediately triggers the signal generator output through a hardware interrupt, rather than relying on software polling, thereby reducing system jitter. Cross-modal consistency: The tactile pulse width (20ms) matches the acceleration-deceleration curve of the visual animation, allowing the user to perceive a unified physical event.

[0044] The piezoelectric transient feedback in this embodiment is not isolated, but complements electrostatic texture feedback and ultrasonic aerial feedback, resulting in the following significant effects: High recognition and confirmation: The 200Hz sine wave precisely stimulates the Pacinian corpuscle, significantly enhancing the sense of confirmation during operation. Low subjective fatigue: The pure spectrum avoids high-frequency noise, making it suitable for prolonged interaction. Multi-sensory consistency: Vision and touch are strictly synchronized, eliminating perceptual disconnect. Deep integration with multimodal systems: It acts as an "anchor" between electrostatic and ultrasonic feedback, enhancing the overall realism of the interaction.

[0045] Example 3 To achieve wide-range, high-precision aerial tactile coverage, this embodiment employs an 8×8 planar matrix array as the basic configuration of the ultrasonic transducer. Transducer selection and parameters: A closed-cell piezoelectric ultrasonic transducer (PZT) with a center frequency of 40kHz is selected. At this frequency, the air attenuation coefficient is moderate, ensuring sufficient sound pressure level to penetrate the air and reach the fingers while remaining away from the most sensitive auditory region (2kHz-5kHz). The sound pressure level (SPL) of a single transducer is no less than 110dB at 10cm. Array arrangement: The element spacing d is designed to be half a wavelength (λ / 2≈4.3mm). This design follows the Nyquist sampling theorem, effectively avoiding grating lobes within the visible area and ensuring that sound energy is concentrated on the main lobe (i.e., the tactile focus) without diffusing outwards, thereby improving tactile clarity.

[0046] Preferably, this embodiment uses the L298N to construct a Class D switching amplifier circuit. Switching mode operation: The Arduino Mega2560 microcontroller outputs a logic-level PWM signal to the input terminals (IN1 / IN2) of the L298N. The power transistors inside the L298N operate between the saturation and cutoff regions, with extremely low on-resistance (approximately 0.3Ω), significantly reducing static power consumption. Compared to linear drive schemes, the energy conversion efficiency is improved by more than 30%, effectively solving the heat accumulation problem when multiple elements operate simultaneously. Voltage matching: The L298N's supply voltage is set to 18V, perfectly matching the rated voltage of the 40kHz transducer, ensuring the transducer operates at its maximum mechanical amplitude point.

[0047] Preferably, this embodiment introduces a dynamic duty cycle modulation algorithm: the system dynamically adjusts the PWM duty cycle D within a range of 30% to 70% according to the current output power demand. Its mathematical model can be expressed as: D(t) = D base +ksin(2πf m t). Wherein: D base The baseline duty cycle is set to 45%; k is the modulation coefficient (ranging from 0.05 to 0.15); and fm is the low-frequency modulation frequency (approximately 1 kHz, higher than the sensitivity threshold of the human ear to a single frequency). Through this low-frequency modulation, the system disperses the harmonic energy originally concentrated at 40 kHz and its harmonics, spreading it across a wider frequency band. This transforms the originally sharp single-frequency noise into a faint background sound similar to white noise, significantly reducing the discomfort perceived by the human ear and achieving "silent touch."

[0048] Preferably, to accurately locate the tactile point in three-dimensional space, the system employs a time-delay control method to achieve beamforming. Focus calculation: The central processing unit calculates the focus based on the three-dimensional coordinates F(x) of the finger obtained from the infrared positioning module. f ,y f ,zf ), calculate the acoustic path difference Δd from each transducer to the focal point. i Phase delay: Based on the speed of sound c (approximately 343 m / s) and the ultrasonic wave length λ, the system calculates the required phase delay φi for each channel: φ i = 2πΔd i / λ. Signal Synchronization: The Arduino microcontroller precisely controls the output timing of 64 PWM signals based on the calculated phase delay. The signal drive array, amplified by the L298N, causes all sound waves to superimpose in phase at the focal point, forming high-intensity radiation pressure. When a user's finger passes through this area, they can clearly feel the physical presence of a "protrusion" or "button".

[0049] Preferably, in the application scenario of this embodiment, the ultrasound array does not work independently, including: proximity guidance: when the user's finger is far from the screen (>5cm), the array generates a large area of ​​weak focus to guide the user's gesture; air interaction: in the 2cm-5cm range, the focus follows the finger's movement, simulating the damping sensation of a slider or knob; mode switching: when the finger is about to touch the screen (<0.5cm), the system gradually reduces the ultrasound duty cycle until it is turned off according to the focusing algorithm, while activating the electrostatic tactile pathway.

[0050] Preferably, the focusing algorithm uses time-delay focusing, and the method is as follows: calculate the distance d from all transducers to the hand position, and find the farthest distance d among them. max Calculate the time t required for the wave from the farthest transducer to reach the focal point. max , t max =d max / c, where c is the velocity of sound in air; calculate the time delay Δt required for other transducers, Δt=(d max - d) / c.

[0051] The technical advantages of this embodiment are: Quiet operation: Dynamic duty cycle modulation effectively suppresses audible noise, improving the device's environmental friendliness. High energy efficiency: The L298N switch driver significantly reduces system power consumption, extending the device's lifespan. High precision: The phased array focusing algorithm achieves millimeter-level tactile positioning accuracy, supporting complex air-to-air interactive operations.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Those skilled in the art should understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal haptic feedback interaction system, characterized in that, include: The contact-based haptic feedback subsystem includes an electrostatic touchscreen and a piezoelectric vibratory actuator, which provides texture and vibration feedback when the user touches the screen; The non-contact tactile feedback subsystem includes an array of ultrasonic transducers arranged in a specific geometric configuration and a drive module, used to generate focused aerial tactile feedback in space. The interactive perception subsystem includes an infrared touch screen and a visual display module, which are used to track the interaction position and synchronize visual feedback. The central processing unit runs the interaction logic engine and is configured to coordinate the output of corresponding tactile signals by the contact and non-contact tactile feedback subsystems according to the user's interaction intent.

2. The system according to claim 1, characterized in that, The driving module in the non-contact tactile feedback subsystem includes: a microcontroller for generating a pulse width modulation (PWM) signal with a frequency of 40 kHz; and an L298N driving module connected to the microcontroller for amplifying the PWM signal to the rated voltage and current levels required by the ultrasonic transducer. The microcontroller is also configured to dynamically adjust the duty cycle of the PWM signal to focus the ultrasonic wave at a specific location and reduce acoustic noise caused by signal modulation.

3. The system according to claim 1, characterized in that, The contact-type haptic feedback subsystem is configured to generate a specific piezoelectric vibration waveform in response to user operations on screen controls; the central processing unit runs the Unity engine and Python, wherein the Unity engine is used to render the visual interface, and Python is used to include detecting user interaction, calculating phase difference, serial communication, transmitting instructions, and, upon detecting a touch operation, sending instructions to a signal generator to generate a sine wave signal with a frequency of 200Hz and a peak voltage of 5V to drive the piezoelectric ceramic actuator to generate haptic feedback.

4. The system according to claim 1, characterized in that, The ultrasonic transducer array achieves focusing in the following way: multiple transducers in the array receive amplified PWM signals and emit ultrasonic waves. By precisely controlling the phase difference of the signals emitted by each transducer, the sound waves generate constructive interference at a specific spatial point above the array, thereby forming a perceptible tactile focus.

5. The system according to claim 1, characterized in that, The interactive sensing subsystem is configured to support the operation of various interactive controls, including but not limited to buttons, knobs, sliders and switches, and the system can call different haptic profiles according to different control types.

6. A multimodal haptic feedback control method, applicable to the system described in any one of claims 1 to 5, characterized in that, Includes the following steps: Initialization steps: Establish communication between the central processing unit and the signal generator, and initialize the positioning and tracking function of the infrared touch screen; Interaction detection steps: The interactive interface is presented through the visual display module, and the system detects whether the user makes a contact touch or a non-contact approach. Signal processing steps: If contact touch is detected, electrostatic tactile feedback or piezoelectric vibration feedback is triggered to generate the corresponding waveform and voltage; If a non-contact approach is detected, the target focus position is calculated, and a PWM signal with the corresponding duty cycle is generated. Drive output steps: The processed signals are sent to the electrostatic touch screen, piezoelectric actuator or ultrasonic array respectively to generate synchronized visual and tactile feedback.

7. The method according to claim 6, characterized in that, The piezoelectric vibration feedback processing steps specifically include: Python recognizes user clicks on controls on a touchscreen; In response to this operation, the control signal generator outputs a sine wave signal with a frequency of 200Hz and a peak voltage of 5V; This signal is used to drive a piezoelectric ceramic actuator, allowing the user to simultaneously feel the physical vibration transmitted through the screen while receiving visual feedback.

8. The method according to claim 6, characterized in that, The non-contact approach processing steps specifically include: A 40kHz PWM signal with the same resonant frequency as the ultrasonic transducer is generated using a microcontroller; The PWM signal is amplified using the L298N driver module; Based on the tracked location of the interactive object, the duty cycle of the PWM signal is dynamically adjusted to control the emission phase of each transducer in the ultrasonic array, so as to achieve the convergence of ultrasonic waves at a specific point in space.

9. The method according to claim 6, characterized in that, It also includes a feedback smoothing adjustment step: Real-time monitoring of changes in the position of interactive objects and switching of interactive states dynamically adjusts the output parameters of electrostatic tactile feedback, piezoelectric vibration and ultrasonic tactile feedback to ensure a smooth transition when switching between different modal tactile feedback and maintain a consistent user experience.

10. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the multimodal haptic feedback control method as described in any one of claims 6 to 9.

Images