Vr interaction method, system and storage medium of multimodal haptic feedback

Abstract

The application discloses a VR interaction method and system of multimodal tactile feedback and a storage medium, relates to the technical field of virtual reality, and comprises the following steps: step one, modeling of physical properties of virtual objects; step two, detection of interaction events and stratification of contact states; step three, generation of layered and progressive multimodal signals; step four, cross-modal collaborative compensation; and step five, output of tactile feedback signals: after smoothing processing is performed on the comprehensive tactile feedback signals, the signals are output to a multimodal tactile feedback device. The application can be widely applied to various scenes such as VR game entertainment, virtual shopping experience, remote medical training, industrial design review, virtual cultural relic display, and the like, and users can truly perceive the material, temperature, weight and surface characteristics of virtual objects in these scenes, thereby significantly expanding the application range and practical value of VR technology.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, specifically to VR interaction methods, systems, and storage media with multimodal haptic feedback. Background Technology

[0002] Virtual reality (VR) technology provides users with an immersive interactive experience through computer-generated 3D virtual environments and has been widely applied in fields such as gaming, education and training, medical rehabilitation, industrial design, and remote collaboration. With the increasing maturity of VR display and spatial tracking technologies, users are demanding higher levels of realism and immersion in VR interactive experiences. In the human sensory system, touch is the second most important sensory channel after vision, and realistic tactile feedback plays an irreplaceable role in enhancing the sense of presence and operational accuracy of VR interactions.

[0003] Existing VR haptic feedback technology suffers from the following shortcomings: Single-modal haptic feedback cannot fully express the multi-dimensional physical properties of virtual objects. Real-world objects possess multiple physical characteristics such as hardness, texture, temperature, weight, and elasticity, and users perceive these attributes comprehensively when touching them. However, existing haptic feedback systems mostly employ a single feedback modality. Force feedback devices can only present mechanical properties and cannot express temperature and texture, while vibration feedback devices can only present impact and roughness and cannot express pressure and weight, resulting in an incomplete haptic experience and insufficient realism. Summary of the Invention

[0004] The purpose of this application is to provide a VR interaction method, system, and storage medium with multimodal haptic feedback to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, this application provides the following technical solution: a VR interaction method with multimodal haptic feedback, characterized by comprising the following steps:

[0006] Step 1, Virtual Object Physical Property Modeling: Establish a set of physical property parameters for interactive objects in the virtual scene. The set of physical property parameters includes material hardness, surface texture parameters, temperature value, mass value, and elastic coefficient.

[0007] Step 2, Interaction Event Detection and Contact State Layering: Based on spatial tracking data, monitor the spatial positional relationship between the user's hand and the virtual object. When a contact event is detected, calculate the contact depth, which is the distance between the user's hand surface and the virtual object surface or the normalized ratio of this distance to the virtual object's feature size. Based on the contact depth, the interaction state is divided into surface contact state, shallow pressing state, and deep interaction state.

[0008] Step 3, Layered progressive multimodal signal generation: Based on the level to which the current interaction state belongs, progressively activate the combination of tactile feedback modes. The tactile feedback modes include force feedback mode, electrical skin stimulation mode, and bone conduction vibration mode. The electrical skin stimulation mode uses transcutaneous electrical nerve stimulation to generate tactile perception by adjusting the amplitude, frequency, and pulse width of the stimulation current. The signal intensity of each mode changes continuously with the increase of contact depth.

[0009] Step 4, cross-modal collaborative compensation: When the actual output capability of any modality is lower than the target output requirement, the tactile information that the modality cannot express is transferred to other modalities with output margin for fusion to generate a comprehensive tactile feedback signal;

[0010] Step 5, tactile feedback signal output: After smoothing the integrated tactile feedback signal, it is output to the multimodal tactile feedback device.

[0011] Preferably, the division of contact states in step two adopts a dynamic boundary mechanism: the contact depth threshold between each state level is determined based on the material hardness of the virtual object, and the material hardness is negatively correlated with the contact depth threshold.

[0012] The contact depth thresholds are dynamically adjusted based on the user's hand contact speed: when the contact speed exceeds the upper speed threshold, the boundaries of each level are shifted outward; when the contact speed is below the lower speed threshold, the boundaries of each level are shifted inward. The upper speed threshold ranges from 0.3 m / s to 1.0 m / s, and the lower speed threshold ranges from 0.05 m / s to 0.15 m / s. The outward and inward shifts are 5% to 30% of the original contact depth thresholds.

[0013] Preferably, in step three, the progressive activation of the tactile feedback modal combination according to the current interaction state includes: in the surface contact state, prioritizing the activation of the skin electrical stimulation modality to present temperature and texture attributes;

[0014] Superimposed activation of bone conduction vibration modes under shallow compression enhances the perception of hardness and roughness;

[0015] In deep interaction states, the superimposed activation force feedback mode exhibits stress resistance and quality properties.

[0016] Preferably, the mapping relationship between each modal signal parameter and physical property parameter in step three includes: the force feedback signal output intensity is positively correlated with material hardness and contact pressure, and negatively correlated with elastic coefficient; when the user's hand is in a multi-finger pinching posture or a palm-wrapping posture, the force feedback signal output intensity is also related to the virtual object mass value and the hand tilt angle.

[0017] The temperature stimulation type of the electrical nerve stimulation signal is determined by the direction of the difference between the virtual object's temperature value and the user's skin temperature. When the virtual object's temperature value is higher than the user's skin temperature, a warm stimulation signal is output; when the virtual object's temperature value is lower than the user's skin temperature, a cool stimulation signal is output. The user's skin temperature is acquired in real time by a temperature sensor or uses a default value within the range of 32°C to 34°C. When the hand is in a relatively sliding state, the texture stimulation frequency of the electrical nerve stimulation signal is positively correlated with the product of the surface texture parameters and the sliding speed.

[0018] The vibration frequency of bone conduction vibration signals is positively correlated with the material hardness, and the vibration amplitude is positively correlated with the contact impact intensity. The contact impact intensity is calculated based on the product of the contact velocity at the moment of contact and the mass value of the virtual object, or based on the rate of change of contact depth. The vibration attenuation mode is related to the elastic coefficient. The larger the elastic coefficient, the longer the vibration attenuation time constant and the closer the attenuation curve is to the oscillating attenuation form. The smaller the elastic coefficient, the shorter the vibration attenuation time constant and the closer the attenuation curve is to the exponential attenuation form.

[0019] Preferably, the cross-modal collaborative compensation in step four includes: when the force feedback mode is limited, converting the missing amount into an amplitude increment of bone conduction vibration signal or a pressure stimulation increment of skin electrical stimulation signal;

[0020] When the modality of electrical skin stimulation is restricted, temperature information is encoded as frequency modulation patterns of bone conduction vibration signals or micro-pressure changes of force feedback signals; texture information is encoded as vibration pulse sequences.

[0021] When bone conduction vibration modes are restricted, short force pulses or pulse sequences of electrical skin stimulation signals are used to convert vibration energy into force feedback signals.

[0022] The conversion gain coefficient of each compensation path is positively correlated with the degree of output loss of the restricted mode, and the degree of output loss is defined as the ratio of the amount of output loss to the target output value.

[0023] Preferably, the smoothing process in step five includes: setting a maximum allowable rate of change for each haptic feedback parameter; when the change in the target parameter between adjacent update cycles exceeds the maximum allowable rate of change, the current cycle only outputs the change that does not exceed the rate, and the excess is accumulated and gradually released in subsequent update cycles.

[0024] When the user's hand is detected to be detached from the surface of the virtual object, the tactile signals of each modality are smoothly decayed from the current value to zero according to the decay curve. The decay time constant is negatively correlated with the detachment speed, and the value ranges from 20ms to 200ms.

[0025] When the user's hand moves along a continuous trajectory from the first virtual object to the second virtual object, the current haptic feedback parameters are processed by linear or nonlinear interpolation based on the physical attribute parameters of the two objects.

[0026] When the hand movement speed exceeds the stable feedback speed threshold, the output strength of the force feedback signal is reduced or the signal smoothness is increased to avoid mechanical interference in high-speed motion.

[0027] Preferably, it also includes a haptic-visual consistency guarantee step: obtaining the visual contact determination result and visual penetration depth value between the user's hand and the virtual object from the current visual rendering frame, wherein the visual penetration depth value is the depth value of the hand model penetrating the virtual object model calculated by the visual rendering module based on the collision detection algorithm;

[0028] The time deviation value is obtained by comparing the tactile feedback activation time with the visual contact determination time, and the intensity deviation value is obtained by comparing the tactile feedback output intensity with the visual penetration depth value.

[0029] When the time deviation value exceeds the time deviation threshold, advance prediction compensation or lag correction is implemented by adjusting the triggering timing of tactile feedback. The advance prediction compensation is achieved by predicting the contact moment based on the hand movement trajectory and speed and triggering tactile feedback in advance. The lag correction is achieved by delaying the output time of the tactile feedback signal. The time deviation threshold ranges from 10ms to 50ms. When the intensity deviation value exceeds the intensity deviation threshold, the intensity of each modal signal is adjusted accordingly to achieve perceptual synchronization between tactile feedback and visual presentation in the intensity dimension.

[0030] Preferably, it also includes a tactile distribution differentiation step based on hand posture: identifying the current posture type of the user's hand through hand skeleton tracking or hand posture recognition algorithm, wherein the posture type includes fingertip touching posture, fingertip pressing posture, multi-finger pinching posture and palm holding posture;

[0031] In the fingertip touch posture, tactile feedback is concentrated and distributed to the fingertip area of ​​the contacting finger to enhance texture and temperature perception; in the fingertip pressing posture, force feedback and bone conduction vibration feedback are extended to the fingertip area to enhance pressure and hardness perception; in the multi-finger pinching posture, the intensity of each modality of tactile feedback is proportionally distributed among the participating fingers according to the contact area ratio of each participating finger; in the palm grip posture, tactile feedback related to weight attributes and grip stability is covered to all contacting fingers and palm areas, and is weighted and distributed between the palm area and fingertip area according to the grip direction;

[0032] The multi-finger pinching posture is a posture in which the fingertips or fingertips of three or more fingers work together to contact the virtual object, and the palm-wrapping posture is a posture in which the fingers and palm together wrap around and envelop the virtual object.

[0033] This application also discloses a multimodal haptic feedback VR interaction system, including:

[0034] The physical property modeling module is used to create a set of physical property parameters for interactive objects in a virtual scene, including material hardness, surface texture parameters, temperature value, mass value, and elastic coefficient.

[0035] The interaction detection module is used to monitor the spatial positional relationship between the user's hand and the virtual object in real time based on spatial tracking data, calculate the contact depth, which is the distance value of the user's hand surface penetrating the surface of the virtual object or the normalized ratio of the distance value relative to the feature size of the virtual object, and divide the interaction state into surface contact state, shallow pressing state and deep interaction state according to the contact depth.

[0036] The signal generation module is used to generate tactile signals of various modalities based on the level to which the current interaction state belongs, and by progressively activating the combination of force feedback mode, electrical skin stimulation mode and bone conduction vibration mode.

[0037] The cross-modal compensation module is used to evaluate the effective output capability of each tactile feedback modality. When the actual output capability of any modality is lower than the target output requirement, the tactile information that cannot be fully expressed in that modality is transferred to other modalities for fusion processing to generate a comprehensive tactile feedback signal.

[0038] The signal output module is used to apply smoothing processing to the integrated tactile feedback signal and then output it to a multimodal tactile feedback device that includes a force feedback actuator, a skin electrical stimulation actuator, and a bone conduction vibration actuator.

[0039] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned VR interaction method with multimodal haptic feedback.

[0040] Compared with the prior art, the beneficial effects of this application are:

[0041] This invention establishes a complete set of physical property parameters for virtual objects, including material hardness, surface texture parameters, temperature value, mass value, and elastic coefficient. By combining the synergistic effect of three tactile feedback modes—force feedback, electrical skin stimulation, and bone conduction vibration—it can comprehensively restore the multi-dimensional physical characteristics of virtual objects, enabling users to obtain a tactile perception experience close to that of the real world in the VR environment. This solves the technical problem that existing single-modal tactile feedback is unable to express complex physical properties.

[0042] This invention employs a layered, progressive multimodal signal generation strategy, which gradually activates different tactile modalities from surface contact to deep interaction based on the contact depth. This simulates the natural process of human hand touching an object in the real world, from surface perception to deep perception, avoiding abrupt activation of tactile feedback and significantly improving the naturalness of VR interaction and user immersion.

[0043] This invention employs a cross-modal collaborative compensation mechanism. When the output of any tactile feedback modality is limited due to hardware constraints or malfunctions, tactile information can be automatically transferred to other modalities for fusion expression. This ensures the integrity and continuity of tactile feedback and improves the robustness and reliability of the system under various operating conditions. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the entire process of this application;

[0045] Figure 2 This is a schematic diagram of the fingertip tapping posture in this application;

[0046] Figure 3 This is a schematic diagram of the fingertip pressing posture in this application;

[0047] Figure 4 This is a schematic diagram of the multi-finger pinching posture in this application;

[0048] Figure 5 This is a schematic diagram of the palm-holding posture in this application;

[0049] Figure 6 This is a schematic diagram of the palm view of this application;

[0050] Figure 7 This is a schematic diagram of the contact state layering and dynamic boundary of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Example

[0052] like Figure 1 As shown, the present invention provides a VR interaction method with multimodal haptic feedback, comprising the following steps:

[0053] Step 1: Modeling the physical properties of virtual objects

[0054] Establish a set of physical property parameters for interactive objects in a virtual scene. This set of physical property parameters includes five core parameters: material hardness, surface texture parameters, temperature value, mass value, and elastic coefficient.

[0055] Let's take a VR virtual store scenario as an example. In this scenario, the shelves display a variety of products such as glass cups, wooden ornaments, silk scarves, and ceramic vases. The system needs to establish a corresponding set of physical attribute parameters for each product in order to provide a differentiated tactile feedback experience when the user touches it.

[0056] For the glass, the system sets its material hardness to a high hardness level, which means that the surface of the glass is hard and the user should feel a noticeable rigidity when touching it; the surface texture parameter is set to extremely smooth, with a very small texture period and a texture depth close to zero; the temperature value is set to slightly below room temperature to simulate the coolness of the glass material; the mass value is set according to the typical weight of an actual glass; and the elastic coefficient is set to be close to that of an ideal elastic body, indicating that the glass will produce a crisp vibration echo when struck.

[0057] For wooden ornaments, the system sets the material hardness to medium hardness, so that users can feel a certain solidity when they touch it, but it is not as hard as glass; the surface texture parameter is set to a rough surface with obvious wood grain period, so that users can feel the undulation of the wood grain when they slide their fingers; the temperature value is set to close to room temperature, so that the low thermal conductivity of wood makes it feel warm to the touch; the elasticity coefficient is set to medium elasticity, so that the vibration decays at a moderate speed when it is struck.

[0058] For silk scarves, the system sets the material hardness to the extremely soft level, so that users can hardly feel any resistance when touching it; the surface texture is set to fine and smooth, with the smooth texture unique to silk; the temperature value can be set to slightly higher than room temperature to simulate the warm touch of the fabric; the mass value is set to relatively light; and the elasticity coefficient is set to low elasticity, indicating that the silk recovers slowly after deformation.

[0059] By establishing the aforementioned set of physical property parameters, the system can distinguish the tactile characteristics of objects made of different materials, providing a complete data foundation for the subsequent generation of tactile feedback signals. In practical applications, the set of physical property parameters can be pre-set by scene designers during the content creation stage, or it can be automatically obtained through material libraries.

[0060] Step 2: Interaction event detection and contact state layering

[0061] The system monitors the spatial relationship between the user's hand and virtual objects based on spatial tracking data. It uses a six-degree-of-freedom spatial tracking device and a hand skeleton tracking sensor to acquire the three-dimensional spatial coordinates of each joint of the user's hand in real time.

[0062] When a user reaches out to touch a glass on a shelf in a virtual store, the system continuously calculates the shortest distance between each joint of the user's hand and the surface of the glass. The system first enters a pre-contact state, and begins to preload relevant tactile parameters as the hand approaches the object's surface.

[0063] When a user's finger is detected to be in contact with the surface of the glass—that is, when the distance between the hand's knuckle and the object's surface is less than or equal to zero—the system begins to calculate the contact depth. Contact depth is defined as the distance the user's hand penetrates into the surface of the virtual object. In some applications, the contact depth can also be a normalized ratio of this distance to the virtual object's feature size. This normalization method facilitates handling the consistency of tactile feedback for objects of different sizes.

[0064] The system divides the interaction state into three levels based on the contact depth. The surface contact state corresponds to the scenario where the user's finger just touches the surface of the object. At this time, the contact depth is small, and the user mainly perceives the surface characteristics of the object. The shallow pressing state corresponds to the scenario where the user's finger applies a certain amount of pressure to the object. The contact depth is at a medium level. The deep interaction state corresponds to the scenario where the user grips or presses deeply. The contact depth is large, and the user needs to perceive the overall physical characteristics of the object.

[0065] The advantage of this layered design lies in its ability to simulate the progressive sensory process of touching objects in the real world. When a person touches an object, they first perceive its surface temperature and texture, then gradually perceive its hardness and elasticity as the pressure increases, and finally perceive its weight when gripping it. The layered mechanism of this invention is precisely a simulation of this natural sensory process.

[0066] Step 3: Layered progressive multimodal signal generation

[0067] Based on the current interaction state's hierarchy, the system progressively activates a combination of tactile feedback modes. The tactile feedback modes employed in this invention include force feedback mode, electrical skin stimulation mode, and bone conduction vibration mode.

[0068] When a user's finger is in surface contact, the system preferentially activates the transdermal electrical stimulation modality to present temperature and texture attributes. Continuing with the example of a user touching a glass, the transdermal electrical stimulation modality begins working as soon as the user's fingertip touches the glass surface. This modality uses transcutaneous electrical nerve stimulation, adjusting the amplitude, frequency, and pulse width of the stimulation current to generate different types of tactile sensations.

[0069] For temperature perception, the system first determines the type of temperature stimulus. The type of temperature stimulus in the electrical skin stimulation signal is determined by the direction of the difference between the virtual object's temperature value and the user's skin temperature. When the virtual object's temperature value is higher than the user's skin temperature, the system outputs a warm stimulus signal; when the virtual object's temperature value is lower than the user's skin temperature, the system outputs a cool stimulus signal. The user's skin temperature can be acquired in real time by a temperature sensor installed on the haptic feedback glove, or a default value within the range of 32°C to 34°C can be used if the sensor is unavailable. For objects such as glass cups whose temperature is lower than skin temperature, the system outputs a cool stimulus signal, allowing the user to perceive the unique cool touch of the glass material.

[0070] Regarding texture perception, when a user's hand is in a relatively sliding state, such as sliding their fingers along the surface of a glass, the texture stimulation frequency of the skin conductance stimulation signal is positively correlated with the product of the surface texture parameters and the sliding speed. Because the glass surface is extremely smooth and the texture period is very small, even if the sliding speed is relatively fast, the stimulation frequency generated is still in the high frequency range, and the user perceives a delicate and smooth tactile sensation.

[0071] When the user's finger enters a superficial indentation state, the system, while maintaining electrical stimulation of the skin, superimposes activation of the bone conduction vibration mode to enhance the perception of hardness and roughness. The bone conduction vibration signal is transmitted through a linear vibration motor installed at the proximal end of the finger, and the vibration wave is transmitted through the finger bones to the user's tactile receptors.

[0072] For high-hardness objects like glass, the vibration frequency of the bone conduction vibration signal is set to a relatively high value because the vibration frequency is positively correlated with the material's hardness. When a user taps the surface of the glass, the vibration amplitude is positively correlated with the contact impact intensity. The impact intensity can be calculated by multiplying the contact velocity at the moment of contact by the mass value of the virtual object, or by calculating the rate of change of contact depth.

[0073] Vibration decay patterns are related to the elastic modulus. Because glass has a relatively high elastic modulus, approaching that of an ideal elastic body, its vibration decay time constant is long, and its decay curve approximates an oscillatory decay form. This means that when a user taps a glass, they will feel a crisp "ding" sound, accompanied by a sustained echo, highly replicating the tactile experience of striking real glass. In contrast, if a user taps a wooden object, because wood has a lower elastic modulus and a shorter vibration decay time constant, its decay curve approximates an exponential decay form, and the user will feel a dull "thud," with the vibration quickly disappearing.

[0074] When a user's finger enters a deep interactive state, the system overlays and activates a force feedback mode, exhibiting pressure resistance and mass properties. The force feedback signal output intensity is positively correlated with material hardness and contact pressure, and negatively correlated with the elastic coefficient. When a user attempts to press down on a glass, due to the high hardness of the glass, the force feedback actuator outputs significant resistance, preventing the user's finger from penetrating further, allowing the user to experience the hardness of the glass.

[0075] When a user grasps the glass with multiple fingers or in a palm-wrapped grip to pick it up, the output intensity of the force feedback signal is also related to the virtual object's mass value and the hand's tilt angle. As the user lifts their hand, the force feedback system applies tactile feedback corresponding to gravity to the palm and individual fingers, allowing the user to perceive the glass's weight. When the hand is tilted, the tactile feedback of gravity is redistributed among different areas of the palm, further enhancing the realism.

[0076] The intensity of each modal signal changes continuously with the increase of contact depth, realizing a natural tactile transition from light touch to deep pressure, and avoiding the abrupt feeling when switching levels.

[0077] Step 4, Cross-modal collaborative compensation

[0078] When the actual output capability of any modality is lower than the target output requirement, the system transfers the tactile information that the modality cannot express to other modalities with output margin for fusion, generating a comprehensive tactile feedback signal.

[0079] In practical applications, each actuator in a haptic feedback device has a physical output limit. For example, the maximum output force of a force feedback actuator is limited by the motor torque and mechanical structure; the output of a skin conduction stimulation actuator is limited by the safe current threshold and electrode contact quality; and the vibration amplitude of a bone conduction vibration actuator is limited by the power of the vibration motor. When the haptic feedback intensity required by a virtual interaction scenario exceeds the output capacity of a certain modality, cross-modal compensation is needed to maintain the integrity of the overall haptic experience.

[0080] When the force feedback mode is limited, such as when a user grips a heavy object forcefully but the force feedback actuator has reached its maximum output, the system converts the missing amount into an amplitude increment of bone conduction vibration signal or a pressure stimulation increment of skin electrical stimulation signal. Enhanced vibration and electrical stimulation can partially replace force feedback, allowing the user to still perceive the tactile effect of "force".

[0081] When the electrical stimulation modality is limited, such as due to poor electrode contact or abnormal skin impedance preventing effective transmission of electrical stimulation, the system encodes temperature information as frequency modulation patterns of bone conduction vibration signals or micro-pressure changes in force feedback signals, and encodes texture information as vibration pulse sequences. Through this information encoding and conversion, even if the electrical stimulation modality fails, the user can still perceive temperature and texture information through the specific modes of vibration and force feedback.

[0082] When bone conduction vibration modes are restricted, the system converts vibrational energy into short force pulses or pulse sequences of electrical stimulation signals. These short force or electrical stimulation pulses can produce a vibration-like sensory effect.

[0083] The conversion gain coefficient of each compensation path is positively correlated with the degree of output loss in the restricted mode. The degree of output loss is defined as the ratio of the amount of output loss to the target output value. When a mode is severely restricted, the compensation gain is increased to ensure the complete transmission of tactile information; when it is only slightly restricted, the compensation gain is smaller to avoid tactile distortion caused by overcompensation.

[0084] Step 5: Output tactile feedback signal

[0085] The integrated haptic feedback signal is smoothed before being output to the multimodal haptic feedback device. Smoothing is a key step in ensuring the comfort and naturalness of the haptic feedback.

[0086] The system sets a maximum allowable rate of change for each haptic feedback parameter. When the change in the target parameter between adjacent update cycles exceeds the maximum allowable rate of change, the current cycle only outputs the change not exceeding that rate, and the excess is accumulated and gradually released in subsequent update cycles. This mechanism avoids abrupt changes in haptic signals. For example, when the user suddenly releases their grip, the force feedback does not disappear instantly, but rather decays smoothly over a short period of time, preventing user discomfort.

[0087] When the user's hand is detected to have left the virtual object's surface, the tactile signals of each modality smoothly decay from their current value to zero according to the decay curve. The decay time constant is negatively correlated with the detachment speed, ranging from 20ms to 200ms. When the user quickly raises their hand, the tactile feedback decays rapidly; when the user slowly leaves the object's surface, the tactile feedback fades slowly, which conforms to the laws of tactile experience in the real world.

[0088] As a user's hand moves along a continuous trajectory from one virtual object to another, such as a finger sliding from a glass to an adjacent wooden ornament, the system performs linear or non-linear interpolation on the current haptic feedback parameters based on the physical properties of the two objects. Within the transition area, the haptic feedback parameters gradually change from the characteristics of the first object to those of the second object, allowing the user to perceive a natural transition in the materials rather than a sudden jump.

[0089] When the hand movement speed exceeds the stable feedback speed threshold, the system reduces the output intensity of the force feedback signal or increases the signal smoothness. In high-speed motion, the mechanical response of the force feedback actuator may not keep up with the hand movement; forced output may cause mechanical interference or user discomfort. By reducing the output intensity and increasing the smoothness, the system can maintain a basic haptic feedback experience while ensuring safety.

[0090] Example 2

[0091] This embodiment describes in detail the dynamic boundary mechanism for contact state division.

[0092] The contact state division adopts a dynamic boundary mechanism, which can adaptively adjust the boundary threshold of the state level according to the object characteristics and user behavior.

[0093] The system determines the contact depth threshold between each state level based on the material hardness of the virtual object. Material hardness and contact depth threshold are negatively correlated. For high-hardness objects such as glass or metal, the contact depth threshold at each level boundary is small because the surface of a hard object is almost incompressible, and the user can feel the object's resistance with a light touch, thus allowing for a quick transition to a deeper interactive state. For low-hardness objects such as sponge or fabric, the contact depth threshold at each level boundary is large, requiring the user's finger to penetrate to a greater depth to perceive substantial resistance.

[0094] The system also dynamically adjusts the contact depth thresholds based on the user's hand contact speed. When a user touches an object quickly, such as by patting or tapping, and the contact speed exceeds the upper speed threshold, the system shifts the boundaries of each level outward. This adjustment makes it easier to trigger tactile feedback in deeper interactive states during rapid contact, aligning with the physical intuition that impact contact should produce a strong tactile response.

[0095] When a user touches an object slowly, such as with a gentle stroke or fine probing motion, and the contact speed is below a minimum threshold, the system shifts the boundaries of each haptic level inward. This adjustment allows for a richer tactile experience within a smaller contact depth range during slow contact, making it suitable for applications requiring fine tactile feedback.

[0096] The upper speed threshold ranges from 0.3 m / s to 1.0 m / s and can be adjusted according to specific application scenarios. The lower speed threshold ranges from 0.05 m / s to 0.15 m / s. The outward and inward offsets are 5% to 30% of the original contact depth thresholds, and the specific offset ratio can be calculated by interpolation based on the degree to which the speed deviates from the threshold.

[0097] like Figure 7As shown, the system sets different contact depth thresholds for objects with different material hardness.

[0098] For soft materials, such as sponge pads or silk fabrics in virtual scenes, due to their low hardness, users can press their fingers deep into the surface without feeling significant resistance. The system sets a relatively large upper boundary threshold Th1 for the surface contact state and a correspondingly large upper boundary threshold Th2 for the shallow pressing state. This means that users need to press their fingers a considerable distance to trigger the next level of tactile feedback modality. For example, when a user touches a virtual sponge, pressing the finger in 3 mm is still in the surface contact state, only activating the electrical skin stimulation modality to present the soft and gentle touch of the sponge; pressing the finger in 7 mm enters the shallow pressing state, superimposed with the activation of the bone conduction vibration modality to present the elastic characteristics of the sponge.

[0099] For medium-hardness materials, such as wooden furniture or plastic products in virtual scenes, the system sets the contact depth threshold to a medium level. When a user touches the virtual wooden table, pressing the finger in 2 millimeters enters a shallow pressing state, and pressing in 5 millimeters enters a deep interaction state. This threshold setting allows users to experience complete three-layer tactile feedback within a relatively shallow contact depth range.

[0100] For hard materials, such as glassware or metal tools in virtual scenes, the surface is almost incompressible due to their extremely high hardness. The system sets a relatively small contact depth threshold, allowing users to quickly enter a deep interaction state with a light touch of their finger. For example, when a user taps a virtual glass, pressing the finger in 1 mm enters a shallow pressing state and triggers high-frequency bone conduction vibration; pressing in 3 mm enters a deep interaction state and triggers force feedback to prevent further penetration, realistically replicating the hard and incompressible physical properties of glass.

[0101] The dynamic boundary adjustment mechanism adjusts the aforementioned thresholds in real time based on the user's hand contact speed. This will be illustrated using an example of user interaction in a virtual kitchen scenario:

[0102] When the user rapidly pats the virtual dough, the system detects a contact speed of 0.6 m / s, exceeding the speed limit threshold of 0.3 m / s. The system then shifts the boundaries of each layer outward by 20%, so that a deep interactive state that previously required pressing in 5 mm now only needs to be triggered by pressing in 4 mm. This adjustment aligns with the intuitive expectation that rapid impact should produce a strong tactile response, allowing the user to immediately feel the elastic feedback of the dough.

[0103] When a user slowly strokes the virtual silk, the system detects a contact speed of 0.08 m / s, which is below the lower speed threshold of 0.1 m / s. The system shifts the boundaries of each layer inward by 15%, allowing the user to experience richer tactile variations within a smaller contact depth. As the user gently slides their finger, they can delicately perceive the temperature, texture, and softness of the silk surface, gaining a refined tactile exploration experience.

[0104] The offset is calculated using a linear interpolation principle. When the contact speed is between the lower speed threshold and the normal speed, the inward offset of the boundary varies linearly between 5% and 15%; when the contact speed is between the normal speed and the upper speed threshold, the outward offset of the boundary varies linearly between 5% and 30%. This continuous boundary adjustment avoids abrupt changes in haptic feedback, ensuring that users receive a smooth and natural haptic experience during interactions at different speeds.

[0105] Example 3

[0106] This embodiment describes a tactile-visual consistency guarantee mechanism.

[0107] In VR applications, the consistency between haptic feedback and visual presentation directly impacts the user's immersive experience. Because haptic feedback systems and visual rendering systems may have different processing latency, a dedicated consistency guarantee mechanism is needed to ensure their coordinated synchronization.

[0108] The system first obtains the visual contact determination result and visual penetration depth value between the user's hand and the virtual object from the current visual rendering frame. The visual penetration depth value is the depth value of the hand model penetrating the virtual object model, calculated by the visual rendering module based on the collision detection algorithm. This value represents the degree of contact between the user's hand and the object at the visual level.

[0109] The system compares the activation time of tactile feedback with the visual contact determination time to obtain a time deviation value. Simultaneously, it compares the tactile feedback output intensity with the visual penetration depth value to obtain an intensity deviation value.

[0110] When the time deviation exceeds the time deviation threshold, the system performs synchronization correction. The time deviation threshold ranges from 10ms to 50ms, and this threshold is set based on the sensitivity of the human perceptual system to visual-tactile asynchrony. If tactile feedback lags behind visual presentation, the user will see that their fingers have touched the object but have not yet felt the touch, which will severely disrupt the sense of immersion. At this time, the system corrects this through advance prediction compensation, predicting the moment of contact based on the hand movement trajectory and speed, and triggering tactile feedback in advance, so that the tactile feedback occurs synchronously with visual contact.

[0111] If haptic feedback precedes visual presentation, the user will experience touch before seeing it. While this slightly disrupts immersion, it still requires correction. The system addresses this through hysteresis correction, appropriately delaying the output of the haptic feedback signal to align it with the visual presentation.

[0112] When the intensity deviation exceeds the intensity deviation threshold, the system adjusts the intensity of each modal signal accordingly to achieve perceptual synchronization between tactile feedback and visual presentation in the intensity dimension. For example, when the visual display shows the depth of a finger penetrating a soft object but the tactile feedback intensity is significantly insufficient, the user will feel a mismatch between touch and vision. In this case, the system increases the tactile output intensity to match the penetration depth presented visually.

[0113] Example 4

[0114] This embodiment describes the tactile distribution differentiation processing based on hand posture.

[0115] The system identifies the user's current hand posture type through hand skeleton tracking or hand posture recognition algorithms. The posture types defined in this invention include four typical types: fingertip touching posture, fingertip pressing posture, multi-finger pinching posture, and palm clasping posture.

[0116] The tactile distribution strategy under different postures is illustrated using a VR pottery making scenario as an example.

[0117] When a user lightly touches and rotates the clay surface with the tip of a single finger to perceive its texture, the system recognizes this as a fingertip touch and concentrates the tactile feedback on the fingertip area to enhance texture and temperature perception. The fingertip is one of the most densely packed areas of tactile receptors in the human body, making it ideal for transmitting fine tactile information. Electrical skin stimulation primarily targets the fingertip electrode array, enabling the user to accurately perceive the moist texture and cool temperature of the clay.

[0118] When a user presses their fingertips against the clay surface, the system recognizes this as a fingertip pressing motion and extends force feedback and bone conduction vibration feedback to the fingertip area to enhance pressure and hardness perception. The fingertip area has a relatively large skin surface area, making it suitable for transmitting holistic tactile information such as pressure and hardness. Users can fully experience the soft, deformable properties of the clay under finger pressure.

[0119] When a user uses their thumb, index finger, and middle finger to pinch and shape the edge of clay, the system recognizes this as a multi-finger pinching posture. A multi-finger pinching posture is defined as a posture in which the fingertips or pads of three or more fingers work together to contact a virtual object. In this posture, the system proportionally distributes the intensity of tactile feedback across the participating fingers based on the contact area ratio. Fingers with larger contact areas receive more tactile feedback, while fingers with smaller contact areas receive correspondingly weaker feedback. This distribution method conforms to the tactile distribution patterns observed in real-world grasping.

[0120] When a user kneads clay with their entire palm, the system recognizes this as a palm-wrapping posture. This posture is defined as the hand and fingers together encircling and wrapping around a virtual object. In this posture, the system applies haptic feedback related to weight and grip stability to all contacting fingers and palm, weighting it between the palm and fingertips based on the grip direction. When the user lifts the clay upwards, the palm receives more weight feedback; when the user pushes the clay forward, the fingertips receive more resistance feedback. This dynamic distribution enhances the realism of the grip interaction.

[0121] This embodiment describes in detail the tactile distribution differentiation strategy and specific parameter configuration based on hand posture recognition.

[0122] like Figure 2-5 As shown, there are four typical hand posture types, and a differentiated haptic feedback distribution scheme is configured for each posture.

[0123] The first posture is the fingertip touch posture, which is characterized by a single finger touching the surface of a virtual object with the fingertip, while the other fingers remain naturally bent or extended. Typical application scenarios include fine interactive tasks such as clicking virtual buttons and touching the surface of an object to explore its texture.

[0124] In a fingertip-touching posture, the system concentrates tactile feedback energy on the fingertip area. Specifically, the fingertip area receives 100% of the skin conduction stimulation feedback intensity to represent the object's temperature and surface texture characteristics; bone conduction vibration feedback is concentrated on the linear vibration motor of that finger, with a vibration intensity of 120% of the standard value to enhance tactile perception; force feedback is output separately through the corresponding chordae tendon actuator. This centralized distribution strategy fully utilizes the high density of tactile receptors in the fingertip area, enabling users to accurately perceive subtle features of the object's surface.

[0125] The second posture is the fingertip pressing posture, which is characterized by pressing the surface of a virtual object with the fingertip area of ​​one or more fingers, with the fingers in a naturally bent state. Typical application scenarios include interactive tasks that require applying pressure, such as pressing soft objects and sensing the hardness of objects.

[0126] Under fingertip pressure, the system extends tactile feedback to the fingertip area. Specifically, both the fingertip and fingertip electrode arrays are activated simultaneously, expanding the skin electrical stimulation coverage to the entire area of ​​the fingertip and fingertip; bone conduction vibration intensity is increased to 150% of the standard value to enhance hardness perception; and force feedback output intensity increases linearly with pressure depth. This large tactile coverage of the fingertip area is suitable for transmitting overall physical property information such as pressure and hardness.

[0127] The third posture is the multi-finger pinching posture, which is characterized by the fingertips or pads of three or more fingers working together to contact the virtual object, with the fingers in an inward-curving state. Typical application scenarios include pinching small objects, fine manipulation of tools, and other interactive tasks that require multi-finger coordination.

[0128] In a multi-finger pinching posture, the system distributes tactile feedback intensity according to the contact area ratio of each participating finger. Taking a pinch with the thumb, index finger, and middle finger as an example, if the detected contact area ratio of the thumb is 45%, the index finger is 35%, and the middle finger is 20%, then the tactile feedback intensity is distributed proportionally: the thumb receives 45% of the total force feedback intensity and skin electrical stimulation intensity, the index finger receives 35%, and the middle finger receives 20%. Bone conduction vibration feedback is also distributed in this proportion among the linear vibration motors of the three fingers. This proportional distribution strategy conforms to the physical law of uneven force distribution among the fingers during multi-finger grasping in the real world, enhancing the realism of the tactile feedback.

[0129] When the number of fingers pinching changes, the system recalculates the contact area ratio of each finger in real time and dynamically adjusts the allocation scheme. For example, when the user changes from pinching with three fingers to pinching with four fingers, the newly added ring finger receives a corresponding proportion of tactile feedback based on its contact area, and the allocation ratio of the other fingers is adjusted accordingly.

[0130] The fourth posture is the palm-wrapping posture, which is characterized by the fingers and palm wrapping around the virtual object. The fingers are in a bent gripping position, and the palm is in contact with the surface of the object. Typical application scenarios include interactive tasks that require the use of the whole hand, such as holding a tool handle, grasping a spherical object, and picking up heavy objects.

[0131] In a palm-wrapped grip posture, the system provides tactile feedback to all contacting fingers and palm areas, and distributes it weighted according to the grip direction. The default distribution is as follows: the palm area receives 60% of the weight tactile feedback intensity, which is output as pressing force through a pneumatic soft actuator; the fingertips of all four fingers receive 30% of the grip force feedback intensity, which is output as gripping resistance through a chord actuator; and the thumb receives 10% of the contralateral stabilizing force feedback intensity.

[0132] The weighting is dynamically adjusted as the user's hand posture changes. If the user lifts an object upwards, the palm bears more weight perception, with the palm area's weight distribution increasing to 70% and the fingertips decreasing to 25%. If the user pushes an object forward, the fingertips bear more resistance perception, with the fingertips' weight distribution increasing to 50% and the palm area decreasing to 45%. This dynamic weighting based on grip direction allows tactile feedback to accurately reflect changes in the spatial distribution of an object's weight.

[0133] The bone conduction vibration feedback in the palm-holding posture is mainly output by the dorsal bone conduction oscillator, which is used to transmit the overall vibration characteristics and impact feedback of the object. When the user holds the virtual hammer to strike the object, the impact vibration is transmitted to the entire palm through the dorsal bone conduction oscillator, which, together with the force feedback of each finger and the electrical stimulation of the skin, forms a complete tool-holding experience.

[0134] Example 5

[0135] like Figure 2 As shown, the present invention also provides a VR interactive system with multimodal haptic feedback, including a physical property modeling module, an interaction detection module, a signal generation module, a cross-modal compensation module, and a signal output module.

[0136] The physical property modeling module is used to establish a set of physical property parameters for interactive objects in a virtual scene. This set includes five core parameters: material hardness, surface texture parameters, temperature value, mass value, and elastic coefficient. The physical property parameter set can be stored in an object property database, read and cached in memory during scene loading for real-time access. This module also supports dynamic modification of object properties at runtime to adapt to scene requirements arising from changes in object state.

[0137] The interaction detection module monitors the spatial relationship between the user's hand and virtual objects in real time based on spatial tracking data. This module receives position data from the head-mounted display and controllers of the six-DOF tracking system, as well as position data from each finger joint of the hand's skeletal tracking sensor. Internally, the module implements an efficient collision detection algorithm capable of determining contact between the hand and all interactive objects in the scene within each rendering frame. When a contact event is detected, the module calculates the contact depth and, based on the contact depth and a dynamic boundary mechanism, classifies the interaction state into three levels: surface contact state, shallow indentation state, and deep interaction state.

[0138] The signal generation module is used to progressively activate combinations of force feedback mode, electrical skin stimulation mode, and bone conduction vibration mode based on the current interaction state level, generating tactile signals for each mode. This module incorporates multiple signal mapping algorithms, capable of converting physical property parameters and interaction state parameters into specific tactile signal parameters, including the force value and direction of force feedback, the current amplitude, frequency, and pulse width of electrical skin stimulation, and the vibration frequency, amplitude, and attenuation mode of bone conduction vibration.

[0139] The cross-modal compensation module is used to evaluate the effective output capability of each tactile feedback modality. This module monitors the working status and output margin of each actuator in real time. When it detects that the actual output capability of any modality is lower than the target output requirement, the cross-modal compensation algorithm is activated to transfer the tactile information that cannot be fully expressed in that modality to other modalities with output margins for fusion processing, and finally generate a comprehensive tactile feedback signal.

[0140] The signal output module smooths the integrated haptic feedback signal before outputting it to the multimodal haptic feedback device. This module implements smoothing algorithms such as rate-of-change limiting, detachment attenuation, cross-object transition, and high-speed motion protection to ensure the continuity and comfort of the output signal. The module transmits the processed signal to the wearable haptic feedback glove via a wired or wireless communication interface.

[0141] Multimodal haptic feedback devices include three types of actuators: force feedback actuators, electrical skin stimulation actuators, and bone conduction vibration actuators. Force feedback actuators can employ chordal drives, using a motor to pull chords to generate resistance at the finger joints; or they can use pneumatic soft actuators to generate pressure in the palm area through inflation and deflation. Electrical skin stimulation actuators use flexible electrode arrays distributed at the fingertips and fingertips, capable of generating various tactile sensations such as temperature and texture. Bone conduction vibration actuators use linear vibration motors or bone conduction oscillators installed at the proximal ends of the fingers and the back of the hand, with vibration signals transmitted through the bones to the skin's tactile receptors.

[0142] This embodiment describes in detail the hardware structure configuration of the multimodal haptic feedback glove.

[0143] like Figure 6 As shown, the multimodal haptic feedback glove consists of two parts: a back-side structure and a palm-side structure.

[0144] The glove is equipped with a bone conduction vibration actuator and a force feedback drive mechanism in the back of the hand structure.

[0145] The bone conduction transducer is installed in the center of the back of the hand and uses a high-power bone conduction transducer with a vibration frequency range of 20Hz to 500Hz and a maximum vibration amplitude of 3G. This bone conduction transducer is used to transmit overall impact feedback and low-frequency vibration signals. When the user taps or bumps a virtual object, the vibration wave is transmitted through the metacarpal bones to the entire hand, producing a realistic impact sensation.

[0146] Linear vibration motors are installed on the dorsal side of the proximal phalanges of the index, middle, ring, and little fingers, as well as the proximal part of the thumb, totaling five independent vibration units. Each linear vibration motor has a vibration frequency range of 50Hz to 300Hz and a maximum vibration amplitude of 2G. These distributed vibration motors can output differentiated vibration feedback based on the independent contact state of each finger, achieving a refined local tactile presentation.

[0147] The force feedback drive mechanism employs a chord traction method. Five miniature servo motors are mounted inside the wrist shell of the glove, each connected to a high-strength chord. The chords run along the back of the hand, passing through guide loops on the back of each finger, and finally connecting to the distal phalanx of each finger. When the servo motors tighten the chords, the tension prevents the fingers from bending, thus generating gripping resistance feedback. The maximum traction force of a single motor is 15N, and when all five fingers work together, they can provide a total gripping resistance of 75N, sufficient to simulate the tactile experience of gripping heavy or hard objects.

[0148] The glove is equipped with a skin electrical stimulation actuator, a palm force feedback mechanism, and various sensors in the palm side structure.

[0149] The electrical stimulation (ESS) actuator comprises two sets of electrodes: a fingertip electrode array and a fingertip electrode array. The fingertip electrode arrays are installed at the tips of the index, middle, ring, and little fingers, and at the fingertip of the thumb. Each array contains 16 individual microelectrodes arranged in a 4×4 matrix, with an electrode diameter of 2 mm and an electrode spacing of 3 mm. The fingertip electrode arrays are installed in the middle of the fingertip of each finger, with each array containing 12 individual microelectrodes arranged in a 3×4 matrix. All electrodes are made of flexible conductive silicone, conforming to the curved surface of the finger and maintaining stable skin contact. The electrical stimulation signal uses a biphasic symmetrical pulse waveform, with a current amplitude ranging from 0 to 8 mA, a frequency range from 1 Hz to 250 Hz, and a pulse width range from 50 microseconds to 500 microseconds. By adjusting the activation sequence and current parameters of adjacent electrodes, the system can simulate temperature sensing and texture slip sensing.

[0150] A pneumatic soft actuator, employing a multi-chamber airbag structure made of silicone, is installed in the palm area. The airbag is connected to a miniature air pump and solenoid valve assembly in the wrist. By precisely controlling the inflation and deflation of each chamber, it can generate pressing pressure feedback of 0 to 8N in the palm. This actuator is mainly used to present the palm pressure sensation when holding an object and the tactile expression of the object's weight.

[0151] The sensor array comprises three categories: pressure sensors, temperature sensors, and an inertial measurement unit (IMU). Five thin-film pressure sensors, each with a range of 0 to 50 N and a resolution of 0.1 N, are mounted at the fingertips of each finger and are used to detect the actual pressure applied by the user to achieve closed-loop control of force feedback. Temperature sensors are mounted in the palm to collect real-time data on the user's skin temperature, providing a reference value for tactile temperature feedback. The IMU, located on the back of the wrist, includes a three-axis accelerometer and a three-axis gyroscope, used to detect the hand's movement speed and posture angle, providing motion data support for dynamic boundary adjustment and weight-based tactile feedback.

[0152] Example 6

[0153] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps of the above-described multimodal haptic feedback VR interaction method.

[0154] The storage medium can be any medium capable of storing program code, such as read-only memory, random access memory, hard disk, optical disk, flash memory, etc. The processor can be a central processing unit, graphics processing unit, digital signal processor, application-specific integrated circuit, field-programmable gate array, or other computing unit capable of executing programs.

[0155] In a typical system deployment, the computer program is stored in the memory of the VR headset or a connected host computer and is loaded and executed by the device's processor. During runtime, the program continuously receives spatial tracking data and hand posture data, performs the aforementioned interaction detection, signal generation, cross-modal compensation, and signal output processes, and sends haptic feedback signals to the wearable haptic feedback device via a communication interface.

[0156] The embodiments of this application are subject to the understanding of those skilled in the art, and various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application. The scope of this application is defined by the appended claims and their equivalents.

Claims

1. A VR interaction method with multimodal haptic feedback, characterized in that, Includes the following steps: Step 1, Virtual Object Physical Property Modeling: Establish a set of physical property parameters for interactive objects in the virtual scene. The set of physical property parameters includes material hardness, surface texture parameters, temperature value, mass value, and elastic coefficient. Step 2, Interaction Event Detection and Contact State Layering: Based on spatial tracking data, monitor the spatial positional relationship between the user's hand and the virtual object. When a contact event is detected, calculate the contact depth, which is the distance between the user's hand surface and the virtual object surface or the normalized ratio of this distance to the virtual object's feature size. Based on the contact depth, the interaction state is divided into surface contact state, shallow pressing state, and deep interaction state. Step 3, Layered progressive multimodal signal generation: Based on the level to which the current interaction state belongs, progressively activate the combination of tactile feedback modes. The tactile feedback modes include force feedback mode, electrical skin stimulation mode, and bone conduction vibration mode. The electrical skin stimulation mode uses transcutaneous electrical nerve stimulation to generate tactile perception by adjusting the amplitude, frequency, and pulse width of the stimulation current. The signal intensity of each mode changes continuously with the increase of contact depth. Step 4, cross-modal collaborative compensation: When the actual output capability of any modality is lower than the target output requirement, the tactile information that the modality cannot express is transferred to other modalities with output margin for fusion to generate a comprehensive tactile feedback signal; Step 5, tactile feedback signal output: After smoothing the integrated tactile feedback signal, it is output to the multimodal tactile feedback device; It also includes a haptic-visual consistency guarantee step: obtaining the visual contact determination result and visual penetration depth value between the user's hand and the virtual object from the current visual rendering frame. The visual penetration depth value is the depth value of the hand model penetrating the virtual object model calculated by the visual rendering module based on the collision detection algorithm. The time deviation value is obtained by comparing the tactile feedback activation time with the visual contact determination time, and the intensity deviation value is obtained by comparing the tactile feedback output intensity with the visual penetration depth value. When the time deviation value exceeds the time deviation threshold, advance prediction compensation or lag correction is implemented by adjusting the triggering timing of tactile feedback. The advance prediction compensation is achieved by predicting the contact moment based on the hand movement trajectory and speed and triggering tactile feedback in advance. The lag correction is achieved by delaying the output time of the tactile feedback signal. The time deviation threshold ranges from 10ms to 50ms. When the intensity deviation value exceeds the intensity deviation threshold, the intensity of each modal signal is adjusted accordingly to achieve perceptual synchronization between tactile feedback and visual presentation in the intensity dimension.

2. The VR interaction method with multimodal haptic feedback according to claim 1, characterized in that, The second step of the contact state division adopts a dynamic boundary mechanism: the contact depth threshold between each state level is determined based on the material hardness of the virtual object, and the material hardness is negatively correlated with the contact depth threshold. The contact depth thresholds are dynamically adjusted based on the user's hand contact speed: when the contact speed exceeds the upper speed threshold, the boundaries of each level are shifted outward; when the contact speed is below the lower speed threshold, the boundaries of each level are shifted inward. The upper speed threshold ranges from 0.3 m / s to 1.0 m / s, and the lower speed threshold ranges from 0.05 m / s to 0.15 m / s; the outward and inward offsets are 5% to 30% of the original contact depth threshold.

3. The VR interaction method with multimodal haptic feedback according to claim 1, characterized in that, In step three, the progressive activation of tactile feedback modal combinations based on the current interaction state level includes: prioritizing the activation of the skin electrical stimulation modality to present temperature and texture attributes in the surface contact state; Superimposed activation of bone conduction vibration modes under shallow compression enhances the perception of hardness and roughness; In deep interaction states, the superimposed activation force feedback mode exhibits stress resistance and quality properties.

4. The VR interaction method with multimodal haptic feedback according to claim 3, characterized in that, The mapping relationship between each modal signal parameter and physical property parameter in step three includes: the force feedback signal output intensity is positively correlated with material hardness and contact pressure, and negatively correlated with elastic coefficient; when the user's hand is in a multi-finger pinching posture or a palm-wrapping posture, the force feedback signal output intensity is also related to the virtual object mass value and the hand tilt angle. The temperature stimulation type of the electrical nerve stimulation signal is determined by the direction of the difference between the virtual object's temperature value and the user's skin temperature. When the virtual object's temperature value is higher than the user's skin temperature, a warm stimulation signal is output; when the virtual object's temperature value is lower than the user's skin temperature, a cool stimulation signal is output. The user's skin temperature is acquired in real time by a temperature sensor or uses a default value within the range of 32°C to 34°C. When the hand is in a relatively sliding state, the texture stimulation frequency of the electrical nerve stimulation signal is positively correlated with the product of the surface texture parameters and the sliding speed. The vibration frequency of bone conduction vibration signals is positively correlated with the material hardness, and the vibration amplitude is positively correlated with the contact impact intensity. The contact impact intensity is calculated based on the product of the contact velocity at the moment of contact and the mass value of the virtual object, or based on the rate of change of contact depth. The vibration attenuation mode is related to the elastic coefficient. The larger the elastic coefficient, the longer the vibration attenuation time constant and the closer the attenuation curve is to the oscillating attenuation form. The smaller the elastic coefficient, the shorter the vibration attenuation time constant and the closer the attenuation curve is to the exponential attenuation form.

5. The VR interaction method with multimodal haptic feedback according to claim 1, characterized in that, The cross-modal collaborative compensation in step four includes: when the force feedback mode is limited, converting the missing amount into an amplitude increment of bone conduction vibration signal or a pressure stimulation increment of skin electrical stimulation signal; When the modality of electrical skin stimulation is restricted, temperature information is encoded as frequency modulation patterns of bone conduction vibration signals or micro-pressure changes of force feedback signals; texture information is encoded as vibration pulse sequences. When bone conduction vibration modes are restricted, short force pulses or pulse sequences of electrical skin stimulation signals are used to convert vibration energy into force feedback signals. The conversion gain coefficient of each compensation path is positively correlated with the degree of output loss of the restricted mode, and the degree of output loss is defined as the ratio of the amount of output loss to the target output value.

6. The VR interaction method with multimodal haptic feedback according to claim 1, characterized in that, The smoothing process in step five includes: setting a maximum allowable rate of change for each haptic feedback parameter; when the change in the target parameter between adjacent update cycles exceeds the maximum allowable rate of change, the current cycle only outputs the change amount not exceeding the rate of change, and the excess portion is accumulated and gradually released in subsequent update cycles. When the user's hand is detected to be detached from the surface of the virtual object, the tactile signals of each modality are smoothly decayed from the current value to zero according to the decay curve. The decay time constant is negatively correlated with the detachment speed, and the value ranges from 20ms to 200ms. When the user's hand moves along a continuous trajectory from the first virtual object to the second virtual object, the current haptic feedback parameters are processed by linear or nonlinear interpolation based on the physical attribute parameters of the two objects. When the hand movement speed exceeds the stable feedback speed threshold, the output strength of the force feedback signal is reduced or the signal smoothness is increased to avoid mechanical interference in high-speed motion.

7. The VR interaction method with multimodal haptic feedback according to claim 1, characterized in that, It also includes a tactile distribution differentiation step based on hand posture: identifying the current posture type of the user's hand through hand skeleton tracking or hand posture recognition algorithm, the posture type including fingertip touching posture, fingertip pressing posture, multi-finger pinching posture and palm holding posture; In the fingertip touch posture, tactile feedback is concentrated and distributed to the fingertip area of ​​the contacting finger to enhance texture perception and temperature perception; in the fingertip pressing posture, force feedback and bone conduction vibration feedback are extended to cover the fingertip area to enhance pressure perception and hardness perception; in the multi-finger pinching posture, the intensity of each modal tactile feedback is proportionally distributed among the participating fingers according to the contact area ratio of each participating finger. In the palm-wrapping posture, tactile feedback related to weight attributes and grip stability is covered to all contact fingers and palm areas, and weighted distribution is made between the palm area and fingertip area according to the grip direction. The multi-finger pinching posture is a posture in which the fingertips or fingertips of three or more fingers work together to contact the virtual object, and the palm-wrapping posture is a posture in which the fingers and palm together wrap around and envelop the virtual object.

8. The VR interaction method with multimodal haptic feedback according to claim 1, characterized in that, Also includes: The physical property modeling module is used to create a set of physical property parameters for interactive objects in a virtual scene, including material hardness, surface texture parameters, temperature value, mass value, and elastic coefficient. The interaction detection module is used to monitor the spatial positional relationship between the user's hand and the virtual object in real time based on spatial tracking data, calculate the contact depth, which is the distance value of the user's hand surface penetrating the surface of the virtual object or the normalized ratio of the distance value relative to the feature size of the virtual object, and divide the interaction state into surface contact state, shallow pressing state and deep interaction state according to the contact depth. The signal generation module is used to generate tactile signals of various modalities based on the level to which the current interaction state belongs, and by progressively activating the combination of force feedback mode, electrical skin stimulation mode and bone conduction vibration mode. The cross-modal compensation module is used to evaluate the effective output capability of each tactile feedback modality. When the actual output capability of any modality is lower than the target output requirement, the tactile information that cannot be fully expressed in that modality is transferred to other modalities for fusion processing to generate a comprehensive tactile feedback signal. The signal output module is used to apply smoothing processing to the integrated tactile feedback signal and then output it to a multimodal tactile feedback device that includes a force feedback actuator, a skin electrical stimulation actuator, and a bone conduction vibration actuator.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the VR interaction method of multimodal haptic feedback as described in any one of claims 1 to 8.

Images