Vr haptic feedback system with multimodal sensory fusion and method thereof

Abstract

The application relates to the technical field of virtual reality, in particular to a VR tactile feedback system based on multi-modal perception fusion and a method thereof, which comprises a tactile interaction terminal, a distributed tactile feedback terminal, a distributed signal acquisition unit, a computer system, a tactile presentation device, a visual device and an auditory device; the core of the application is a tactile prediction unit in the distributed tactile feedback terminal; a 3D content tactile prediction algorithm based on matrix theory is constructed; tactile texture matrix and tactile material matrix are used to realize accurate mapping of visual features and tactile features; a high-dimensional matrix is innovatively used to represent a tactile feature space; a matrix spectrum analysis technology is used to realize adaptive adjustment of tactile features; a multi-scale matrix decomposition and fusion algorithm is used to realize tactile texture reconstruction from a macroscopic view to a microscopic view; the application solves the problems of limited tactile feature expression capability, lack of adaptability and insufficient multi-modal perception cooperation in a traditional VR tactile feedback system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of virtual reality technology, in particular to a multi-modal perception fusion VR haptic feedback system and method thereof, and more particularly to a haptic prediction algorithm based on matrix theory and a multi-sensory collaborative enhancement technical solution. BACKGROUND

[0002] In recent years, with the rapid development of virtual reality technology, the immersion experience of VR systems has become a research hotspot. However, existing VR systems mainly focus on visual and auditory experiences, and the research on haptic feedback is relatively insufficient, resulting in a lack of realism in user interaction experience in virtual environments.

[0003] Most of the haptic feedback devices on the market currently use simple vibration or force feedback mechanisms, which cannot accurately simulate complex haptic characteristics. Traditional haptic feedback systems usually rely on pre-set haptic templates or simple mapping functions, making it difficult to accurately express the micro-haptic characteristics of virtual objects such as material and texture. In addition, existing technologies lack effective multi-modal perception fusion mechanisms, and cannot achieve collaborative enhancement of vision, hearing and touch, thereby limiting the immersion and naturalness of VR system interaction.

[0004] In particular, traditional haptic feedback systems have three main problems: first, the haptic feature expression ability is limited, making it difficult to simultaneously simulate full-spectrum haptic characteristics from macro-shape to micro-texture; second, there is a lack of adaptive ability for individual differences and interaction modes, and the haptic feedback is single and fixed; third, multi-modal perception collaboration is insufficient, and the fusion degree of haptic and audio-visual perception is low, making it difficult to form effective multi-sensory enhancement effects.

[0005] Therefore, there is an urgent need for a VR haptic feedback system that can accurately simulate complex haptic characteristics, has adaptive ability, and supports multi-modal perception fusion, to improve user immersion and interaction experience in virtual environments. SUMMARY

[0006] The purpose of the present application is to provide a multi-modal perception fusion VR haptic feedback system and method thereof, by innovatively applying matrix theory to construct a haptic prediction algorithm, solving the problems of limited haptic feature expression ability, lack of adaptability, and insufficient multi-modal perception collaboration in traditional haptic feedback systems, and realizing high-precision, personalized, and multi-modal collaborative haptic feedback experience.

[0007] The present application proposes a multi-modal perception fusion VR haptic feedback system, comprising:

[0008] A haptic interaction terminal for providing haptic interaction and collecting haptic interaction data;

[0009] The distributed tactile feedback end is connected with the tactile interaction end in communication, and is configured to predict tactile texture based on visual texture information and provide tactile feedback.

[0010] The distributed signal acquisition unit is connected with the tactile interaction end and the distributed tactile feedback end in communication, and is configured to acquire the tactile interaction data and provide tactile signals.

[0011] The computer system is connected with the tactile interaction end, the distributed tactile feedback end and the distributed signal acquisition unit in communication, and is configured to collect the tactile interaction data, realize tactile prediction by using a multi-modal perception fusion method, perform control of a tactile presentation device, and generate visual images and voice signals.

[0012] The tactile presentation device is connected with the distributed tactile feedback end, and is configured to generate tactile feedback according to the tactile signals, simulate tactile characteristics and texture.

[0013] The visual device is connected with the distributed tactile feedback end in communication, and is configured to present the visual images and realize tactile-visual perception collaborative enhancement in combination with the tactile presentation device.

[0014] The auditory device is connected with the tactile interaction end, and is configured to synchronously play the voice signals and realize tactile-auditory perception collaborative enhancement.

[0015] The distributed tactile feedback end comprises a tactile prediction unit, the tactile prediction unit constructs a 3D content tactile prediction algorithm based on matrix theory, the tactile prediction unit realizes mapping of visual features and tactile features by using a predefined tactile texture matrix and a tactile texture matrix, the tactile texture matrix and the tactile texture matrix represent a tactile feature space by using a high-dimensional matrix, the tactile prediction unit realizes adaptive adjustment of tactile features based on a matrix spectrum analysis technology, and the tactile prediction unit realizes tactile texture reconstruction from a macroscopic view to a microscopic view by using a multi-scale matrix decomposition and fusion algorithm.

[0016] Preferably, the tactile interaction end comprises a wearable actuator array and a vibration glove, wherein the actuator array comprises a micro actuator array and a thermal stimulation coil; the micro actuator array can accurately reflect a contact position with an object, and has a single-point tactile feedback precision of 2 mm; the thermal stimulation coil generates skin thermal feedback by temperature change, improves the perception accuracy of a tactile object surface, and simulates hardness and elasticity characteristics of different textures.

[0017] Preferably, the tactile prediction unit predefines a two-dimensional tactile texture matrix J and a two-dimensional tactile texture matrix M, which record tactile information at different positions in a tactile space; the tactile texture matrix J ∈ R (m×n)wherein m represents spatial resolution, n represents tactile texture attribute dimension; the tactile material matrix M ∈ R (m×k) wherein k represents material attribute dimension; the tactile prediction unit maps the tactile texture matrix J and the tactile material matrix M with texture data delivered by the visual device, realizing material prediction and tactile feedback.

[0018] As preferred, the tactile prediction unit realizes adaptive adjustment of tactile features by the following way: collecting tactile interaction data, constructing interaction mode vector P, the interaction mode vector P contains the size of applied force F ∈ R m , contact duration T ∈ R m and force direction vector D ∈ R (m×3) ; applying spectral clustering algorithm to map the interaction mode vector P to four basic tactile mode spaces: contact, sliding, knocking, vibration; according to the identified tactile mode, adjusting the tactile texture matrix J and the tactile material matrix M through error correction function of matrix spectral analysis.

[0019] As preferred, the tactile prediction unit realizes multi-scale tactile texture reconstruction by the following way: decomposing the tactile texture matrix J into multiple scale components wherein, represents macro-scale tactile features, represents the most microscopic scale tactile features; constructing mode-scale mapping function, activating tactile features of different scales according to different tactile interaction modes; calculating scale weight according to the current interaction mode, generating weighted fusion tactile features ; real-time monitoring of interaction state changes, dynamically adjusting scale weight, constructing smooth transition function, ensuring natural switching between different scales.

[0020] As preferred, the distributed signal acquisition unit includes signal acquisition circuit, tactile interaction device interface and tactile presentation device interface; the signal acquisition circuit acquires tactile interaction data, which is delivered to the computer system through the tactile interaction device interface, and to the tactile presentation device through the tactile presentation device interface; the distributed signal acquisition unit acquires tactile interaction data from the tactile interaction end and the distributed tactile feedback end, feeds back tactile signals of the tactile interaction end and the tactile presentation device, and transmits to the computer system through the communication network.

[0021] As preferred, the computer system comprises a processor, a memory and a communication interface, wherein the memory stores a haptic perception collaborative enhancement algorithm; the processor executes the haptic perception collaborative enhancement algorithm, connects with the haptic interaction terminal, the distributed haptic feedback terminal and the distributed signal acquisition unit through the communication interface, and communicates with the visual device and the auditory device using a communication network; the processor executes the haptic perception collaborative enhancement algorithm in cooperation with the visual device and the auditory device, generates haptic feedback according to haptic patterns and materials, and realizes the fusion of touch-visual and touch-auditory perception.

[0022] As preferred, the distributed haptic feedback terminal, the visual device and the haptic presentation device are jointly presented, providing the fusion of visual images and haptic feedback, and realizing the touch-visual perception collaborative enhancement; wherein the haptic presentation device simulates haptic characteristics, transmits haptic features such as haptic materials and textures of virtual object surfaces through tactile simulation, torque simulation and temperature simulation.

[0023] As preferred, the auditory device plays voice signals synchronized with the haptic feedback, realizing the touch-auditory perception collaborative enhancement; the voice signals of the auditory device are played synchronously with the haptic feedback, improving the fineness of the haptic feedback.

[0024] The VR haptic feedback method of multi-modal perception fusion comprises the following steps:

[0025] The haptic interaction is provided through the haptic interaction terminal, and haptic interaction data is collected;

[0026] The haptic feedback is provided through the distributed haptic feedback terminal based on visual material information to predict haptic materials;

[0027] The haptic signal is provided through the distributed signal acquisition unit to collect the haptic interaction data;

[0028] The haptic interaction data is collected through the computer system, the haptic prediction is realized using the multi-modal perception fusion method, the control of the haptic presentation device is executed, and the visual images and voice signals are generated;

[0029] The haptic feedback is generated according to the haptic signal through the haptic presentation device, and the haptic characteristics and materials are simulated;

[0030] The visual images are presented through the visual device, and the haptic presentation device is jointly presented, realizing the touch-visual perception collaborative enhancement;

[0031] The voice signals are played synchronously through the auditory device, realizing the touch-auditory perception collaborative enhancement;

[0032] In the step of predicting haptic material at the distributed haptic feedback end, a 3D content haptic prediction algorithm based on matrix theory is constructed, a predefined haptic texture matrix and a haptic material matrix are used to realize the mapping of visual features and haptic features, the haptic texture matrix and the haptic material matrix adopt high-dimensional matrix to represent the haptic feature space, adaptive adjustment of the haptic features is realized based on matrix spectral analysis technology, and the haptic texture reconstruction from macro to micro is realized through a multi-scale matrix decomposition and fusion algorithm.

[0033] The beneficial effects of the present application include:

[0034] 1. The haptic feature space is represented by a high-dimensional matrix, which improves the expression ability of the haptic features, enables the system to accurately capture and express the micro haptic characteristics of complex surfaces, and improves the haptic feedback accuracy from 5-8mm of the traditional system to 2mm, and the material identification accuracy of the user from 65% to 92%.

[0035] 2. Adaptive adjustment of the haptic features is realized based on matrix spectral analysis technology, the system can learn and optimize the haptic parameters from the actual interaction of the user, adapt to the haptic perception characteristics of different users, and improve the user satisfaction by 47% and the haptic immersion score by 38%.

[0036] 3. Through a multi-scale matrix decomposition and fusion algorithm, haptic texture reconstruction from macro to micro is realized, the system can simulate full-spectrum haptic characteristics from rough surface to fine texture at the same time, greatly enhancing the sense of hierarchy and realism of haptic experience.

[0037] 4. The synergistic enhancement of touch-visual and touch-auditory perception is realized, which significantly improves the user's immersion and naturalness of interaction. In the multi-modal perception test, the touch-visual-auditory synergistic enhancement mechanism of the present system makes the user report a 52% increase in the presence score.

[0038] 5. The system response delay is reduced from 25-30ms of the traditional scheme to 8-12ms, close to the human touch perception threshold (about 10ms), in high-intensity interaction scenarios, the system can still maintain a stable update rate of 60Hz, ensuring the continuity and fluency of haptic feedback. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 It is the overall architecture schematic diagram of the VR haptic feedback system of the present application multi-modal perception fusion;

[0040] Figure 2 It is the structure schematic diagram of the haptic interaction end of the present application;

[0041] Figure 3 It is the structure schematic diagram of the distributed haptic feedback end of the present application;

[0042] Figure 4Flow chart of adaptive adjustment of haptic features based on matrix spectrum analysis of the application;

[0043] Figure 5 Schematic diagram of multi-scale matrix decomposition and fusion algorithm of the application;

[0044] Figure 6 Schematic diagram of implementation mechanism of touch-visual perception collaborative enhancement of the application;

[0045] Figure 7 Schematic diagram of implementation mechanism of touch-auditory perception collaborative enhancement of the application;

[0046] Figure 8 Flow chart of VR haptic feedback method of multi-modal perception fusion of the application. DETAILED DESCRIPTION

[0047] Please refer to Figure 1 - Figure 8 The technical solutions of the application will be described in detail below with reference to the drawings and embodiments.

[0048] Refer to Figure 1 The multi-modal perception fusion VR haptic feedback system provided by the application comprises a haptic interaction end 1, a distributed haptic feedback end 2, a distributed signal acquisition unit 3, a computer system 4, a haptic presentation device 5, a visual device 6, and an auditory device 7.

[0049] The haptic interaction end 1 is in communication connection with the computer system 4, and is used for providing haptic interaction and collecting haptic interaction data. The distributed haptic feedback end 2 is in communication connection with the computer system 4, and is used for predicting haptic materials based on visual material information and providing haptic feedback. The distributed signal acquisition unit 3 is in communication connection with the haptic interaction end 1 and the distributed haptic feedback end 2, and is in communication connection with the computer system 4 through a communication network, and is used for collecting haptic interaction data and providing haptic signals. The computer system 4 collects haptic interaction data, realizes haptic prediction by using a multi-modal perception fusion method, executes control of the haptic presentation device 5, and generates visual images and voice signals. The haptic presentation device 5 is connected with the distributed haptic feedback end 2, and is used for generating haptic feedback according to the haptic signals, simulating haptic characteristics and materials. The visual device 6 is in communication connection with the distributed haptic feedback end 2, and is used for presenting visual images, and is combined with the haptic presentation device 5 to realize touch-visual perception collaborative enhancement. The auditory device 7 is connected with the haptic interaction end 1, and is used for synchronously playing voice signals to realize touch-auditory perception collaborative enhancement.

[0050] Refer to Figure 2 In a preferred embodiment of the application, the haptic interaction end 1 comprises a wearable actuator array 11 and a vibration glove 12. The actuator array 11 comprises a micro actuator array and a thermal stimulation coil.

[0051] The micro-actuator array employs piezoelectric actuators and linear motor technology, arranged in a hexagonal grid structure, capable of accurately reflecting the contact position with the object, with single-point tactile feedback precision reaching 2mm. This high-precision design enables the system to simulate subtle surface texture changes, significantly enhancing the realism of the tactile experience. For example, when a user touches a wooden table in a virtual environment, the system can accurately simulate the subtle bumps and lines of the wood grain, allowing the user to clearly perceive the realistic texture of the material. The actuator array 11 and the vibration glove 12 constitute the tactile interaction end 1, which provides real-time precise tactile feedback through the actuators on the actuator array 11, including contact, sliding, tapping, vibration, texture, and material properties, and exchanges tactile interaction data with the computer system 4 in real time.

[0052] The thermal stimulation coil uses a combination of micro Peltier elements and temperature sensors, with a temperature range controllable between -5°C and +45°C, and a temperature rise / fall response time less than 2 seconds. The skin thermal feedback generated by temperature change improves the perception of the tactile object surface, simulating the hardness and elasticity characteristics of different materials. For example, when simulating metal material, the system will produce a slight cold feeling (about 18°C), while simulating wood will produce a mild warm feeling (about 25°C). In a virtual education scenario, students can touch different virtual material samples and feel the real temperature difference, thereby gaining a deeper understanding of the material physical properties.

[0053] In practical applications, the power consumption of the tactile interaction end 1 is controlled within 5W, which can be powered through a standard USB interface, ensuring comfort and portability for long-term use.

[0054] The distributed tactile feedback end 2 includes a tactile prediction unit, which constructs a 3D content tactile prediction algorithm based on matrix theory, and is one of the core innovations of the invention.

[0055] The tactile prediction unit realizes the mapping of visual features and tactile features through a predefined tactile texture matrix J and a tactile material matrix M. These two matrices are two-dimensional matrices that record tactile information at different positions in the tactile space. Specifically, the tactile texture matrix J ∈ R (m×n) where m represents the spatial resolution (number of position points), and n represents the tactile texture attribute dimension; the tactile material matrix M ∈ R (m×k) where k represents the material attribute dimension.

[0056] In one specific embodiment of the present invention, the typical value of the spatial resolution m is 64×64, that is, 4096 spatial location points, corresponding to the layout of the haptic actuator array; the typical value of the haptic texture attribute dimension n is 16, including characteristics such as roughness, hardness, and elasticity; the typical value of the material attribute dimension k is 12, including characteristics such as temperature conductivity and coefficient of friction. For example, when a user explores the surface of an ancient artifact in a VR environment, the system can accurately simulate the fine texture of pottery and the smoothness of porcelain, helping the user obtain richer information about the artifact through touch.

[0057] The haptic prediction unit maps the haptic matrices J and M to the texture data transmitted by the vision device 6, thereby achieving material prediction and haptic feedback. The specific mapping process is as follows:

[0058] First, key features are extracted from the visual texture features V, and a tactile texture matrix J is generated through the mapping function f(V):

[0059] ,

[0060] in, This is the tactile texture matrix, representing the texture characteristics of each point in the tactile space; The visual texture feature vector is extracted from the visual device. This is a visual feature transformation function that maps visual features to tactile feature space; This is the weight matrix, which controls the strength of the mapping; This is a bias term used to adjust the baseline tactile sensation.

[0061] Similarly, from the perspective of visual material characteristics Through mapping function Generate haptic material matrix :

[0062] ,

[0063] in, This is the tactile material matrix, representing the material properties of each point in the tactile space; The visual material feature vector is extracted from the visual device. This is a material feature transformation function that maps visual material features to tactile material space; This is the weight matrix, which controls the strength of the mapping; This is a bias term used to adjust the feel of the base material.

[0064] To improve computational efficiency, the haptic prediction unit employs singular value decomposition (SVD) of matrices to optimize the computation process. This is based on the haptic texture matrix. For example, its SVD decomposition is as follows:

[0065] ,

[0066] in, For tactile texture matrix (dimension m×n); It is a left singular vector matrix (dimension m×m) containing feature information of spatial location; It is a singular value diagonal matrix (dimension m×n), where the elements on the diagonal represent the importance of each feature; It is a right singular vector matrix (dimension n×n) containing feature information of texture attributes; Representation matrix The transpose of .

[0067] By retaining 90% of the total information... With singular values, we can obtain an approximate matrix after dimensionality reduction:

[0068] ,

[0069] in, To retain the previous The left singular vector matrix of the column (dimension m×r); To retain the previous A diagonal matrix with singular values ​​(dimension r×r); To retain the previous The transpose of the right singular vector matrix of the row (dimension r×n). The number of singular values ​​to be preserved is typically much smaller than In this embodiment The typical value is 20.

[0070] This dimensionality reduction significantly reduces computational complexity while preserving the key features of the matrix. In practical VR applications, such as virtual sculpture creation environments, this optimization allows the system to maintain haptic feedback quality while increasing the refresh rate to 60Hz, ensuring smooth haptic feedback for the user during the sculpting process.

[0071] In one important embodiment of the present invention, the tactile prediction unit achieves adaptive adjustment of tactile features based on matrix spectral analysis technology, thereby improving the personalization and accuracy of the system.

[0072] The specific implementation method is as follows: First, tactile interaction data is collected through the distributed signal acquisition unit 3 to construct an interaction mode vector. This vector contains the magnitude of the applied force. Duration of contact and the direction vector of the force

[0073] ,

[0074] Wherein, P is the interaction mode vector, containing complete tactile interaction information; F represents the magnitude of the force applied at m spatial locations, in Newtons (N), typically ranging from 0 to 10 N; T represents the contact duration, in seconds (s), typically ranging from 0 to 2 s; D represents the force direction vector, containing three directional components: x, y, and z, and is an m×3 matrix, with each directional component ranging from [-1, 1], representing a unit vector.

[0075] Then, the system applies a spectral clustering algorithm to map the interaction pattern vector P to four basic tactile pattern spaces: contact (C), gliding (S), tapping (K), and vibration (V). The pattern discrimination function h(P) is defined as:

[0076] ,

[0077] Where h(P) is the pattern discrimination function, and the output is the type of tactile pattern recognized; The feature vectors for each pattern are determined in advance using training data; arg max represents the vector inner product operation, which calculates the similarity between P and the feature vectors of each pattern; arg max represents the pattern type corresponding to the maximum inner product value.

[0078] For example, in a VR medical training system, when trainees practice palpation skills, the system can identify their touch patterns (gentle touch, gliding exploration, dot tapping, or vibration detection) and provide corresponding tactile feedback based on different palpation patterns, allowing trainees to feel the tactile differences of different tissues.

[0079] Based on the identified tactile patterns, the tactile texture matrix J and the tactile material matrix M are adjusted using an error correction function derived from matrix spectral analysis.

[0080] ,

[0081] ,

[0082] in, and These are the current tactile texture matrix and tactile material matrix, respectively; and This is the updated matrix; and It is an error correction function based on matrix spectral analysis; and This is an adaptive learning rate, initially set to 0.05 and 0.03 respectively, and gradually decreased as the number of interactions increases to ensure the convergence of the algorithm. The learning rate update formula is as follows: ,in The initial learning rate, This is the attenuation coefficient (typically 0.01). This represents the number of interactions.

[0083] Error correction function The specific definition is:

[0084] ,

[0085] in, This represents the error correction amount for the texture matrix; The ideal tactile texture matrix is ​​inferred based on interaction data; This is a weighting coefficient (typically 0.8), which controls the strength of the correction. The feature selection matrix is ​​a diagonal matrix with diagonal elements of 0 or 1, used to determine the feature dimensions that need to be updated.

[0086] Similarly, The function is defined as:

[0087] ,

[0088] in, This represents the error correction amount for the material matrix; An ideal tactile material matrix inferred from interaction data; This is the weighting coefficient (typically 0.7). Select a matrix for material features, structure and similar.

[0089] In practical applications, such as VR gaming environments, when players interact with the same virtual object (such as leather surfaces with different textures) multiple times, the system gradually adapts to the player's tactile perception preferences. For example, if a player tends to lightly touch the surface to perceive subtle textures, the system will enhance the feedback intensity of the micro-textures; if a player is accustomed to pressing hard to perceive the elasticity of the material, the system will adjust the elasticity feedback parameters to provide a tactile experience that better suits their personal habits.

[0090] To ensure the stability and convergence of matrix adjustments, the system uses eigenvalue decomposition to extract principal feature directions and eliminate redundant information. If the correlation coefficient between two feature dimensions is greater than 0.85 (this threshold was determined through extensive user testing and effectively balances dimensionality reduction and information preservation), they are merged to reduce the dimensionality of the feature space and improve computational efficiency. The formula for calculating the feature correlation coefficient is:

[0091] ,

[0092] in, Features and The correlation coefficient between them; Let be the covariance of the two features; and denoted as the standard deviations of the two features.

[0093] Reference Figure 6 In another innovative embodiment of the present invention, the tactile prediction unit realizes tactile texture reconstruction from macro to micro through a multi-scale matrix decomposition and fusion algorithm, which solves the problem of single scale and loss of details in traditional tactile simulation.

[0094] First, the tactile texture matrix J is decomposed into multiple scale components:

[0095] ,

[0096] in, For the complete tactile texture matrix; For component matrices of different scales, the dimension of each component matrix is... same; It represents macroscopic tactile features (such as overall shape and outline). It represents the most microscopic tactile features (such as fine textures and surface microstructures). is the scale, representing the number of levels in the decomposition.

[0097] This decomposition is based on the wavelet transform of the matrix, ensuring the orthogonality of features at different scales. In practical applications, the typical value of the number of scales, 's', is 5, setting 5 levels from macroscopic to microscopic to meet the expression needs of tactile features at different granularities. The value of 's' can be adjusted for different application scenarios: in medical palpation simulation, the value of 's' can be set to 6-7 to capture more subtle tissue differences; while in industrial design applications, the value of 's' can be set to 4-5 to focus on macroscopic tactile characteristics.

[0098] The specific wavelet transform decomposition process is as follows:

[0099] ,

[0100] in, For the first A component matrix of several scales; For the first Wavelet transform matrices of various scales; for The transpose of the matrix. Wavelet transform matrices at different scales. Wavelet basis functions corresponding to different frequency ranges.

[0101] Then, construct the pattern-scale mapping function. Mapping tactile patterns to corresponding scale weights:

[0102] ,

[0103] in, This is the pattern-scale mapping function; The output of the tactile pattern discrimination function defined above is the type of tactile pattern that is identified; the mapping result is the most active scale index, ranging from 1 to s.

[0104] Different tactile features are activated based on different interaction modes:

[0105] Contact mode (C) is the primary activation mode. and (Macroeconomic characteristics);

[0106] Slide mode (S) activated arrive (Mesoscopic characteristics);

[0107] Tap mode (K) activated arrive (Macro to meso-level characteristics);

[0108] Vibration mode (V) activation arrive (Mesoscopic to microscopic features);

[0109] This model-scale mapping is based on psychophysical research into human tactile perception. For example, when humans lightly touch an object's surface, they primarily perceive macroscopic shapes, while when sliding their fingers, they are more sensitive to mesoscopic texture features. In VR art creation platforms, when artists interact with a virtual canvas using different touch methods, the system correspondingly activates tactile features at different scales, enabling artists to accurately perceive the full spectrum of tactile information, from the overall texture of the canvas to subtle brushstrokes.

[0110] Based on the current interaction mode Calculate the weights for each scale. :

[0111] ,

[0112] in, Let be the weight of the i-th scale, ranging from [0,1], and the sum of all weights is 1; The center location of the current major activation scale is determined by the mode-scale mapping function. Sure; The standard deviation of the weight distribution (typically 1.2) controls the concentration of the weight distribution; It is a natural exponential function; the denominator is a normalization factor to ensure that the sum of all weights is 1.

[0113] This Gaussian-based weighting ensures a smooth transition between different scales. For example, for the contact mode, a μ value of 1.5 (between scales 1 and 2) and a σ value of 1.2 will produce a set of smooth weight distributions dominated by scales 1 and 2.

[0114] Finally, weighted fused tactile features are generated:

[0115] ,

[0116] in, The resulting tactile texture matrix; Let be the weight of the i-th scale; Let be the component matrix at the i-th scale; This represents a summation operation, which sums up all weighted scale components.

[0117] Similarly, the material matrix M also undergoes similar multi-scale decomposition and fusion processing to obtain... .

[0118] To ensure a natural transition between different scales and avoid abrupt changes in tactile feedback, the system monitors changes in the interaction state in real time, dynamically adjusts the scale weights, and constructs a smooth transition function:

[0119] ,

[0120] in, The weight of the i-th scale at the current time t; The weight for the next time step; The target weight is calculated from the current interaction mode; τ is the smoothness coefficient, with a value of 0.15, which controls the smoothness of scale switching. The smaller the τ value, the smoother the transition but the slower the response; the larger the τ value, the faster the response but may produce a sudden feeling. 0.15 is a balance value determined through user experience testing.

[0121] In VR surgical simulation systems, this multi-scale tactile reconstruction technology enables surgeons to simultaneously perceive the overall elasticity and micro-texture of tissues. When the surgeon transitions from lightly touching the tissue to applying pressure or sliding, the system can smoothly transition tactile feedback, providing a consistent and realistic tactile experience.

[0122] Reference Figure 1 In an embodiment of the present invention, the distributed signal acquisition unit 3 includes a signal acquisition circuit, a tactile interaction device interface, and a tactile presentation device interface.

[0123] The signal acquisition circuit collects tactile interaction data at a sampling rate of 1000Hz to ensure high-frequency tactile interaction capture. This data is transmitted to the computer system 4 via the tactile interaction device interface and to the tactile presentation device 5 via the tactile presentation device interface. The distributed signal acquisition unit 3 collects tactile interaction data from the tactile interaction terminal 1 and the distributed tactile feedback terminal 2, provides feedback on the tactile signals from the tactile interaction terminal 1 and the tactile presentation device 5, and transmits them to the computer system 4 via a communication network.

[0124] In VR industrial training applications, when trainees operate virtual mechanical equipment, the signal acquisition circuit can accurately capture different operating forces and contact methods, providing the system with precise tactile interaction data, thereby generating realistic mechanical operation tactile sensations and helping trainees master the correct operating skills.

[0125] Signal processing latency is controlled within 5ms, ensuring the overall system response latency is less than 10ms, close to the threshold of human tactile perception. Furthermore, wireless communication utilizes the low-latency Bluetooth 5.0 protocol, with a data transmission rate of 3Mbps, meeting real-time interaction requirements.

[0126] The computer system 4 includes a processor, memory, and a communication interface, wherein the memory stores a tactile perception collaborative enhancement algorithm.

[0127] The processor executes a tactile perception co-enhancement algorithm, connecting to the tactile interaction terminal 1, the distributed tactile feedback terminal 2, and the distributed signal acquisition unit 3 via a communication interface, and communicating with the vision device 6 and the hearing device 7 via a communication network. The processor, in collaboration with the vision device 6 and the hearing device 7, executes the tactile perception co-enhancement algorithm to generate tactile feedback based on tactile patterns and materials, achieving fusion of touch-visual and touch-auditory perception.

[0128] In practical applications, computer system 4 can be a dedicated VR processor or a general-purpose computer platform. The processor requires at least 4 cores and 8 threads with a clock speed of 2.5 GHz or higher; the memory capacity must be at least 8 GB; and the communication interface must support multiple communication protocols such as USB 3.0, Bluetooth 5.0, and Wi-Fi 6.

[0129] In VR education applications, such as virtual astronomy classrooms, the computer system 4 can coordinate the visual presentation, tactile feedback, and audio explanations of planetary surfaces, allowing students to experience the synergistic stimulation of sight, touch, and hearing by touching different planetary surfaces, thereby enhancing their understanding and memory of celestial characteristics.

[0130] Reference Figure 6 In an embodiment of the present invention, the distributed haptic feedback terminal 2, the visual device 6 and the haptic presentation device 5 are presented together to provide the fusion of visual images and haptic feedback, thereby achieving synergistic enhancement of touch-visual perception.

[0131] The haptic presentation device 5 simulates tactile characteristics, conveying the tactile features of the virtual object's surface, such as its material and texture, through touch simulation, torque simulation, and temperature simulation. When a user interacts with the virtual object through the haptic interaction terminal 1, the 3D model in the vision device 6 will simulate a corresponding response. For example, when the haptic interaction device simulates a hard object, the surface of the 3D model in the visual image will show indentation when subjected to force. The haptic feedback terminal controls the vibration device to vibrate, simulating the indentation characteristic of the 3D model's surface.

[0132] In virtual architectural design applications, when designers touch different building material surfaces, the system not only provides corresponding tactile feedback but also visually presents the material's deformation and changes in light and shadow. For example, pressing a wood surface will produce a slight indentation and a corresponding elastic feedback, while pressing a concrete surface will produce almost no deformation but will feel hard to the touch. This tactile-visual collaboration greatly enhances the perceptual realism of material properties, helping designers make more accurate material choices.

[0133] This haptic-visual co-enhancement mechanism significantly enhances the user's sense of immersion. In experiments, compared to single haptic feedback, haptic-visual co-enhancement increased the user's reported presence by 43%, with particularly noticeable effects in material recognition tasks.

[0134] Reference Figure 7 In an embodiment of the present invention, the auditory device 7 plays a voice signal synchronized with the tactile feedback to achieve synergistic enhancement of tactile and auditory perception.

[0135] When a user interacts with a virtual object, the system plays corresponding sound effects in sync. For example, tapping wood produces a crisp "thump," and tapping metal produces a distinctive "clang." These sound effects are precisely synchronized with the haptic feedback, with a delay of less than 5ms, ensuring the consistency and coordination of auditory and tactile perception.

[0136] In virtual music teaching applications, when students touch different virtual instruments, the system simultaneously provides tactile feedback (such as the damping sensation of piano keys or the tension of violin strings) and corresponding timbre, helping students establish a connection between tactile actions and timbre, thus accelerating the mastery of instrument playing skills. This tactile-auditory synergy makes the learning process more intuitive and efficient.

[0137] The audio signal from the auditory device 7 is played synchronously with the tactile feedback, improving the precision of the tactile feedback. Experiments show that the enhanced tactile-auditory synergy improves the user's tactile discrimination ability by about 25% and reduces the perception threshold for subtle tactile differences by about 30%.

[0138] Reference Figure 8 The present invention also provides a VR haptic feedback method based on multimodal perception fusion, comprising the following steps:

[0139] The first step involves providing tactile interaction through the tactile interaction terminal 1 and collecting tactile interaction data. Specifically, users interact with objects in the virtual environment through a wearable actuator array 11 and vibrating gloves 12, and the system collects interaction data such as force, position, and duration in real time. In VR cultural relic restoration training, trainees can use this system to accurately perceive the texture and brittleness of ancient ceramics. The system collects the trainees' touch force and gestures, providing appropriate tactile feedback to prevent damage to cultural relics due to improper force during actual restoration.

[0140] The second step involves using the distributed haptic feedback terminal 2 to predict the haptic material based on visual material information and provide haptic feedback. In this step, the haptic prediction unit constructs a 3D content haptic prediction algorithm based on matrix theory, and predefines the haptic texture matrix J and the haptic material matrix M to realize the mapping between visual features and haptic features.

[0141] The third step involves collecting tactile interaction data through the distributed signal acquisition unit 3 to provide tactile signals. The signal acquisition circuit collects interaction data in real time and transmits relevant signals through the tactile interaction device interface and the tactile presentation device interface.

[0142] The fourth step involves collecting tactile interaction data through computer system 4, using multimodal perception fusion methods to achieve tactile prediction, executing control of tactile presentation device 5, and generating visual images and voice signals.

[0143] The fifth step involves generating tactile feedback based on tactile signals using the tactile presentation device 5, simulating tactile characteristics and materials.

[0144] The sixth step involves presenting a visual image through the visual device 6, which is then presented in conjunction with the tactile presentation device 5 to achieve synergistic enhancement of touch-visual perception.

[0145] The seventh step is to synchronously play voice signals through the hearing device 7 to achieve enhanced tactile and auditory perception.

[0146] In the step of predicting tactile material at the distributed tactile feedback end 2, the tactile prediction unit achieves three core innovations: first, it represents the tactile feature space through a high-dimensional matrix; second, it achieves adaptive adjustment of tactile features based on matrix spectral analysis technology; and third, it achieves tactile texture reconstruction from macroscopic to microscopic through multi-scale matrix decomposition and fusion algorithms.

[0147] In VR rehabilitation applications, when patients are undergoing hand rehabilitation training, the system works in tandem through these seven steps to provide visual guidance, tactile feedback, and auditory cues, helping patients complete precise rehabilitation movements and promoting neuroplasticity through multimodal sensory stimulation, thereby accelerating the rehabilitation process.

[0148] In the actual implementation of this invention, to ensure effective fusion of data from different sources, the system normalizes all input data. For tactile interaction data, the force F is normalized to the range [0,1] and then transformed linearly. ,in and These are the preset minimum force (ON) and maximum force (10N); time TI is normalized to the [0,1] range using a similar linear transformation; direction vector D is normalized to a unit vector, through... Calculation, where Let represent the Euclidean norm of vector D.

[0149] For visual feature data, texture features are normalized to the range [-1, 1], and color features are normalized to the range [0, 1]. This uniform data scale processing ensures the accuracy and effectiveness of multimodal data fusion. For example, in virtual textile design applications, the system needs to fuse visual texture, color, and tactile feel. Normalization ensures that these different types of data can be compared and fused at the same scale, enabling designers to accurately preview the visual and tactile effects of fabrics.

[0150] Furthermore, the system implements a data synchronization mechanism to ensure the time alignment of tactile, visual, and auditory data. The visual update rate is set at 90Hz, the tactile feedback update rate at 1000Hz, and the auditory signal update rate at 48kHz. Through interpolation and prediction algorithms, the system can achieve a smooth transition between different update rates, ensuring the coordinated consistency of multimodal perception. The interpolation algorithm uses cubic spline interpolation to ensure a smooth transition between high-frequency tactile data and relatively low-frequency visual data, avoiding perceptual jumps or delays.

[0151] This invention provides a multimodal perception fusion VR haptic feedback system and method. By innovatively applying matrix theory to construct a 3D content haptic prediction algorithm, it achieves a high-precision, personalized, and multimodal collaborative haptic feedback experience. The system uses a high-dimensional matrix to represent the haptic feature space, achieves adaptive adjustment of haptic features based on matrix spectral analysis technology, reconstructs haptic textures from macro to micro levels through multi-scale matrix decomposition and fusion algorithms, and combines touch-visual and touch-auditory perception collaborative enhancement mechanisms to significantly improve the immersion and naturalness of interaction in the VR system.

[0152] This invention solves the problems of limited tactile feature expression ability, lack of adaptability, and insufficient multimodal perception coordination in traditional tactile feedback systems, and provides a new development path for VR tactile feedback technology, which has important theoretical value and application prospects.

[0153] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of protection of the present invention.

Claims

1. A VR haptic feedback system based on multimodal perception fusion, characterized in that, include: The haptic interaction terminal is used to provide haptic interaction and collect haptic interaction data; A distributed haptic feedback terminal, which is communicatively connected to the haptic interaction terminal, is used to predict haptic materials based on visual material information and provide haptic feedback; A distributed signal acquisition unit, which is communicatively connected to the tactile interaction terminal and the distributed tactile feedback terminal, is used to acquire the tactile interaction data and provide tactile signals; The computer system is communicatively connected to the tactile interaction terminal, the distributed tactile feedback terminal, and the distributed signal acquisition unit. It is used to collect the tactile interaction data, realize tactile prediction using a multimodal perception fusion method, execute the control of the tactile presentation device, and generate visual images and voice signals. A tactile presentation device, connected to the distributed tactile feedback end, is used to generate tactile feedback based on the tactile signals to simulate tactile characteristics and materials; A visual device, which is communicatively connected to the distributed haptic feedback terminal, is used to present the visual image and is presented in conjunction with the haptic presentation device to achieve synergistic enhancement of tactile and visual perception. An auditory device, connected to the tactile interaction terminal, is used to synchronously play the voice signal to achieve synergistic enhancement of tactile and auditory perception; The distributed haptic feedback terminal includes a haptic prediction unit, which constructs a 3D content haptic prediction algorithm based on matrix theory. The haptic prediction unit realizes the mapping between visual features and haptic features through a predefined haptic texture matrix and haptic material matrix. The haptic texture matrix and the haptic material matrix use high-dimensional matrices to represent the haptic feature space. The haptic prediction unit realizes adaptive adjustment of haptic features based on matrix spectral analysis technology. The haptic prediction unit realizes haptic texture reconstruction from macroscopic to microscopic through a multi-scale matrix decomposition and fusion algorithm. The tactile prediction unit predefines a tactile texture matrix J and a tactile material matrix M, which are two-dimensional matrices that record tactile information at different locations in the tactile space; the tactile texture matrix J ∈ R (m×n) Where m represents spatial resolution and n represents the dimension of tactile texture attributes; the tactile material matrix M∈R (m×k) , where k represents the material attribute dimension; the tactile prediction unit maps the tactile texture matrix J and the tactile material matrix M to the texture data transmitted by the visual device to realize material prediction and tactile feedback; The tactile prediction unit achieves adaptive adjustment of tactile features by: acquiring tactile interaction data and constructing an interaction pattern vector P, wherein the interaction pattern vector P contains the magnitude of the applied force F∈R. m Contact duration T∈R m The direction vector of the force D∈R (m×3) The interaction pattern vector P is mapped to four basic tactile pattern spaces using a spectral clustering algorithm: contact, sliding, tapping, and vibration. Based on the identified tactile patterns, the tactile texture matrix J and the tactile material matrix M are adjusted using an error correction function based on matrix spectral analysis. The tactile prediction unit achieves multi-scale tactile texture reconstruction by decomposing the tactile texture matrix J into multiple scale components. ,in, Indicates macroscopic tactile characteristics. Representing the most microscopic tactile features; constructing a mode-scale mapping function to activate tactile features at different scales according to different tactile interaction modes; calculating the weights of each scale based on the current interaction mode. Generate weighted fusion of tactile features Real-time monitoring of interaction state changes, dynamic adjustment of scale weights, and construction of smooth transition functions ensure natural switching between different scales.

2. The VR haptic feedback system with multimodal perception fusion according to claim 1, characterized in that: The tactile interaction terminal includes a wearable actuator array and a vibrating glove. The actuator array includes a micro-actuator array and a thermal stimulation coil. The micro-actuator array can accurately reflect the contact position with the object and has a single-point tactile feedback accuracy of up to 2 mm. The thermal stimulation coil improves the perception precision of the tactile object surface through skin thermal feedback generated by temperature changes and simulates the hardness and elasticity characteristics of different materials.

3. The VR haptic feedback system with multimodal perception fusion according to claim 1, characterized in that: The distributed signal acquisition unit includes a signal acquisition circuit, a tactile interaction device interface, and a tactile presentation device interface. The signal acquisition circuit acquires tactile interaction data, transmits it to the computer system through the tactile interaction device interface, and transmits it to the tactile presentation device through the tactile presentation device interface. The distributed signal acquisition unit acquires tactile interaction data from the tactile interaction terminal and the distributed tactile feedback terminal, feeds back tactile signals from the tactile interaction terminal and the tactile presentation device, and transmits them to the computer system through a communication network.

4. The VR haptic feedback system with multimodal perception fusion according to claim 1, characterized in that: The computer system includes a processor, a memory, and a communication interface, wherein the memory stores a tactile perception collaborative enhancement algorithm; the processor executes the tactile perception collaborative enhancement algorithm, connects to the tactile interaction terminal, the distributed tactile feedback terminal, and the distributed signal acquisition unit through the communication interface, and communicates with the visual device and the auditory device using a communication network; the processor executes the tactile perception collaborative enhancement algorithm in collaboration with the visual device and the auditory device, generates tactile feedback based on tactile patterns and materials, and realizes the fusion of touch-visual and touch-auditory perception.

5. The VR haptic feedback system with multimodal perception fusion according to claim 1, characterized in that: The distributed haptic feedback terminal, the visual device, and the haptic presentation device work together to provide a fusion of visual images and haptic feedback, thereby achieving synergistic enhancement of touch-visual perception. The haptic presentation device simulates haptic characteristics and transmits haptic features such as tactile material and texture of virtual object surfaces through tactile simulation, torque simulation, and temperature simulation.

6. The VR haptic feedback system with multimodal perception fusion according to claim 1, characterized in that: The auditory device plays a voice signal synchronized with the tactile feedback to achieve enhanced tactile-auditory perception; the voice signal of the auditory device is played synchronously with the tactile feedback to improve the precision of the tactile feedback.

7. A VR haptic feedback method based on multimodal perception fusion, employing the VR haptic feedback system based on multimodal perception fusion as described in any one of claims 1-6, characterized in that, Includes the following steps: Provide tactile interaction through a tactile interaction terminal and collect tactile interaction data; Tactile feedback is provided by predicting tactile materials based on visual material information through a distributed tactile feedback terminal; The tactile interaction data is collected by a distributed signal acquisition unit to provide tactile signals; The tactile interaction data is collected by a computer system, tactile prediction is achieved using a multimodal perception fusion method, the control of the tactile presentation device is executed, and visual images and voice signals are generated. The tactile presentation device generates tactile feedback based on the tactile signals to simulate tactile characteristics and materials; The visual image is presented by a visual device and presented in conjunction with the tactile presentation device to achieve synergistic enhancement of tactile and visual perception. The audio signal is played synchronously through an auditory device to achieve enhanced tactile and auditory perception. In the step of predicting tactile material at the distributed tactile feedback end, a 3D content tactile prediction algorithm based on matrix theory is constructed. A predefined tactile texture matrix and tactile material matrix are used to realize the mapping between visual features and tactile features. The tactile texture matrix and the tactile material matrix are represented by high-dimensional matrices to represent the tactile feature space. Adaptive adjustment of tactile features is realized based on matrix spectral analysis technology. Tactile texture reconstruction from macro to micro is realized through multi-scale matrix decomposition and fusion algorithms.

Images