Multi-mode perception fusion VR tactile feedback system and method thereof

Abstract

The invention relates to the technical field of virtual reality, in particular to a multi-mode perception fusion VR tactile feedback system and method, and the system comprises a tactile interaction end, a distributed tactile feedback end, a distributed signal collection unit, a computer system, a tactile presentation device, a visual device and an auditory device. The core of the invention lies in that a tactile prediction unit in a distributed tactile feedback end constructs a 3D content tactile prediction algorithm based on a matrix theory, and utilizes a tactile texture matrix and a tactile material matrix to realize accurate mapping between visual features and tactile features; a high-dimensional matrix is creatively adopted to represent a tactile feature space, adaptive adjustment of tactile features is realized based on a matrix spectrum analysis technology, and tactile texture reconstruction from macroscopic to microscopic is realized through a multi-scale matrix decomposition and fusion algorithm. The problems that in a traditional VR tactile feedback system, the tactile feature expression ability is limited, self-adaptability is lacked, and multi-mode perception collaboration is insufficient are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of virtual reality technology, in particular to a multi-modal perception fusion VR haptic feedback system and method thereof, and more particularly to a haptic prediction algorithm based on matrix theory and a multi-sensory collaborative enhancement technical solution. BACKGROUND

[0002] In recent years, with the rapid development of virtual reality technology, the immersion experience of VR systems has become a research hotspot. However, existing VR systems mainly focus on visual and auditory experiences, and the research on haptic feedback is relatively insufficient, resulting in a lack of realism in user interaction experience in virtual environments.

[0003] Most of the haptic feedback devices on the market currently use simple vibration or force feedback mechanisms, which cannot accurately simulate complex haptic characteristics. Traditional haptic feedback systems usually rely on pre-set haptic templates or simple mapping functions, making it difficult to accurately express the micro-haptic characteristics of virtual objects such as material and texture. In addition, existing technologies lack effective multi-modal perception fusion mechanisms, and cannot achieve collaborative enhancement of vision, hearing and touch, thereby limiting the immersion and naturalness of VR system interaction.

[0004] In particular, traditional haptic feedback systems have three main problems: first, the haptic feature expression ability is limited, making it difficult to simultaneously simulate full-spectrum haptic characteristics from macro-shape to micro-texture; second, there is a lack of adaptive ability for individual differences and interaction modes, and the haptic feedback is single and fixed; third, multi-modal perception collaboration is insufficient, and the fusion degree of haptic and audio-visual perception is low, making it difficult to form effective multi-sensory enhancement effects.

[0005] Therefore, there is an urgent need for a VR haptic feedback system that can accurately simulate complex haptic characteristics, has adaptive ability, and supports multi-modal perception fusion, to improve user immersion and interaction experience in virtual environments. SUMMARY

[0006] The purpose of the present application is to provide a multi-modal perception fusion VR haptic feedback system and method thereof, by innovatively applying matrix theory to construct a haptic prediction algorithm, solving the problems of limited haptic feature expression ability, lack of adaptability, and insufficient multi-modal perception collaboration in traditional haptic feedback systems, and realizing high-precision, personalized, and multi-modal collaborative haptic feedback experience.

[0007] The present application proposes a multi-modal perception fusion VR haptic feedback system, comprising:

[0008] A haptic interaction terminal for providing haptic interaction and collecting haptic interaction data;

[0009] The distributed tactile feedback end is connected with the tactile interaction end in communication, and is configured to predict tactile texture based on visual texture information and provide tactile feedback.

[0010] The distributed signal acquisition unit is connected with the tactile interaction end and the distributed tactile feedback end in communication, and is configured to acquire the tactile interaction data and provide tactile signals.

[0011] The computer system is connected with the tactile interaction end, the distributed tactile feedback end and the distributed signal acquisition unit in communication, and is configured to collect the tactile interaction data, realize tactile prediction by using a multi-modal perception fusion method, perform control of a tactile presentation device, and generate visual images and voice signals.

[0012] The tactile presentation device is connected with the distributed tactile feedback end, and is configured to generate tactile feedback according to the tactile signals, simulate tactile characteristics and texture.

[0013] The visual device is connected with the distributed tactile feedback end in communication, and is configured to present the visual images and realize tactile-visual perception collaborative enhancement in combination with the tactile presentation device.

[0014] The auditory device is connected with the tactile interaction end, and is configured to synchronously play the voice signals and realize tactile-auditory perception collaborative enhancement.

[0015] The distributed tactile feedback end comprises a tactile prediction unit, the tactile prediction unit constructs a 3D content tactile prediction algorithm based on matrix theory, the tactile prediction unit realizes mapping of visual features and tactile features by using a predefined tactile texture matrix and a tactile texture matrix, the tactile texture matrix and the tactile texture matrix represent a tactile feature space by using a high-dimensional matrix, the tactile prediction unit realizes adaptive adjustment of tactile features based on a matrix spectrum analysis technology, and the tactile prediction unit realizes tactile texture reconstruction from a macroscopic view to a microscopic view by using a multi-scale matrix decomposition and fusion algorithm.

[0016] Preferably, the tactile interaction end comprises a wearable actuator array and a vibration glove, wherein the actuator array comprises a micro actuator array and a thermal stimulation coil; the micro actuator array can accurately reflect a contact position with an object, and has a single-point tactile feedback precision of 2 mm; the thermal stimulation coil generates skin thermal feedback by temperature change, improves the perception accuracy of a tactile object surface, and simulates hardness and elasticity characteristics of different textures.

[0017] Preferably, the tactile prediction unit predefines a two-dimensional tactile texture matrix J and a two-dimensional tactile texture matrix M, which record tactile information at different positions in a tactile space; the tactile texture matrix J ∈ R (m×n)wherein m represents spatial resolution, n represents tactile texture attribute dimension; the tactile material matrix M ∈ R (m×k) wherein k represents material attribute dimension; the tactile prediction unit maps the tactile texture matrix J and the tactile material matrix M with texture data delivered by the visual device, realizing material prediction and tactile feedback.

[0018] As preferred, the tactile prediction unit realizes adaptive adjustment of tactile features by the following way: collecting tactile interaction data, constructing interaction mode vector P, the interaction mode vector P contains the size of applied force F ∈ R m , contact duration T ∈ R m and force direction vector D ∈ R (m×3) ; applying spectral clustering algorithm to map the interaction mode vector P to four basic tactile mode spaces: contact, sliding, knocking, vibration; according to the identified tactile mode, adjusting the tactile texture matrix J and the tactile material matrix M through error correction function of matrix spectral analysis.

[0019] As preferred, the tactile prediction unit realizes multi-scale tactile texture reconstruction by the following way: decomposing the tactile texture matrix J into multiple scale components wherein, represents macro-scale tactile features, represents the most microscopic scale tactile features; constructing mode-scale mapping function, activating tactile features of different scales according to different tactile interaction modes; calculating scale weight according to the current interaction mode, generating weighted fusion tactile features ; real-time monitoring of interaction state changes, dynamically adjusting scale weight, constructing smooth transition function, ensuring natural switching between different scales.

[0020] As preferred, the distributed signal acquisition unit includes signal acquisition circuit, tactile interaction device interface and tactile presentation device interface; the signal acquisition circuit acquires tactile interaction data, which is delivered to the computer system through the tactile interaction device interface, and to the tactile presentation device through the tactile presentation device interface; the distributed signal acquisition unit acquires tactile interaction data from the tactile interaction end and the distributed tactile feedback end, feeds back tactile signals of the tactile interaction end and the tactile presentation device, and transmits to the computer system through the communication network.

[0021] As preferred, the computer system comprises a processor, a memory and a communication interface, wherein the memory stores a haptic perception collaborative enhancement algorithm; the processor executes the haptic perception collaborative enhancement algorithm, connects with the haptic interaction terminal, the distributed haptic feedback terminal and the distributed signal acquisition unit through the communication interface, and communicates with the visual device and the auditory device using a communication network; the processor executes the haptic perception collaborative enhancement algorithm in cooperation with the visual device and the auditory device, generates haptic feedback according to haptic patterns and materials, and realizes the fusion of touch-visual and touch-auditory perception.

[0022] As preferred, the distributed haptic feedback terminal, the visual device and the haptic presentation device are jointly presented, providing the fusion of visual images and haptic feedback, and realizing the touch-visual perception collaborative enhancement; wherein the haptic presentation device simulates haptic characteristics, transmits haptic features such as haptic materials and textures of virtual object surfaces through tactile simulation, torque simulation and temperature simulation.

[0023] As preferred, the auditory device plays voice signals synchronized with the haptic feedback, realizing the touch-auditory perception collaborative enhancement; the voice signals of the auditory device are played synchronously with the haptic feedback, improving the fineness of the haptic feedback.

[0024] The VR haptic feedback method of multi-modal perception fusion comprises the following steps:

[0025] The haptic interaction terminal provides haptic interaction and collects haptic interaction data;

[0026] The distributed haptic feedback terminal predicts haptic materials based on visual material information and provides haptic feedback;

[0027] The distributed signal acquisition unit collects the haptic interaction data and provides haptic signals;

[0028] The computer system collects the haptic interaction data, uses the multi-modal perception fusion method to realize haptic prediction, executes the control of the haptic presentation device, and generates visual images and voice signals;

[0029] The haptic presentation device generates haptic feedback according to the haptic signals, simulates haptic characteristics and materials;

[0030] The visual device presents the visual images in cooperation with the haptic presentation device, realizing the touch-visual perception collaborative enhancement;

[0031] The auditory device plays the voice signals synchronously, realizing the touch-auditory perception collaborative enhancement;

[0032] In the step of predicting haptic material at the distributed haptic feedback end, a 3D content haptic prediction algorithm based on matrix theory is constructed, a predefined haptic texture matrix and a haptic material matrix are used to realize the mapping of visual features and haptic features, the haptic texture matrix and the haptic material matrix adopt high-dimensional matrix to represent the haptic feature space, adaptive adjustment of the haptic features is realized based on matrix spectral analysis technology, and the haptic texture reconstruction from macro to micro is realized through a multi-scale matrix decomposition and fusion algorithm.

[0033] The beneficial effects of the present application include:

[0034] 1. The haptic feature space is represented by a high-dimensional matrix, which improves the expression ability of the haptic features, enables the system to accurately capture and express the micro haptic characteristics of complex surfaces, and improves the haptic feedback accuracy from 5-8mm of the traditional system to 2mm, and the material identification accuracy of the user from 65% to 92%.

[0035] 2. Adaptive adjustment of the haptic features is realized based on matrix spectral analysis technology, the system can learn and optimize the haptic parameters from the actual interaction of the user, adapt to the haptic perception characteristics of different users, and improve the user satisfaction by 47% and the haptic immersion score by 38%.

[0036] 3. Through a multi-scale matrix decomposition and fusion algorithm, haptic texture reconstruction from macro to micro is realized, the system can simulate full-spectrum haptic characteristics from rough surface to fine texture at the same time, greatly enhancing the sense of hierarchy and realism of haptic experience.

[0037] 4. The synergistic enhancement of touch-visual and touch-auditory perception is realized, which significantly improves the user's immersion and naturalness of interaction. In the multi-modal perception test, the touch-visual-auditory synergistic enhancement mechanism of the present system makes the user report a 52% increase in the presence score.

[0038] 5. The system response delay is reduced from 25-30ms of the traditional scheme to 8-12ms, close to the human touch perception threshold (about 10ms), in high-intensity interaction scenarios, the system can still maintain a stable update rate of 60Hz, ensuring the continuity and fluency of haptic feedback. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 It is the overall architecture schematic diagram of the VR haptic feedback system of the multi-modal perception fusion of the present application;

[0040] Figure 2 It is the structure schematic diagram of the haptic interaction end of the present application;

[0041] Figure 3 It is the structure schematic diagram of the distributed haptic feedback end of the present application;

[0042] Figure 4Flow chart of adaptive adjustment of haptic features based on matrix spectrum analysis of the application;

[0043] Figure 5 Schematic diagram of multi-scale matrix decomposition and fusion algorithm of the application;

[0044] Figure 6 Schematic diagram of implementation mechanism of touch-visual perception collaborative enhancement of the application;

[0045] Figure 7 Schematic diagram of implementation mechanism of touch-auditory perception collaborative enhancement of the application;

[0046] Figure 8 Flow chart of VR haptic feedback method of multi-modal perception fusion of the application. DETAILED DESCRIPTION

[0047] Please refer to Figure 1 - Figure 8 The technical solutions of the application will be described in detail below with reference to the drawings and embodiments.

[0048] Refer to Figure 1 The multi-modal perception fusion VR haptic feedback system provided by the application comprises a haptic interaction end 1, a distributed haptic feedback end 2, a distributed signal acquisition unit 3, a computer system 4, a haptic presentation device 5, a visual device 6, and an auditory device 7.

[0049] The haptic interaction end 1 is in communication connection with the computer system 4, and is used for providing haptic interaction and collecting haptic interaction data. The distributed haptic feedback end 2 is in communication connection with the computer system 4, and is used for predicting haptic materials based on visual material information and providing haptic feedback. The distributed signal acquisition unit 3 is in communication connection with the haptic interaction end 1 and the distributed haptic feedback end 2, and is in communication connection with the computer system 4 through a communication network, and is used for collecting haptic interaction data and providing haptic signals. The computer system 4 collects haptic interaction data, realizes haptic prediction by using a multi-modal perception fusion method, executes control of the haptic presentation device 5, and generates visual images and voice signals. The haptic presentation device 5 is connected with the distributed haptic feedback end 2, and is used for generating haptic feedback according to the haptic signals, simulating haptic characteristics and materials. The visual device 6 is in communication connection with the distributed haptic feedback end 2, and is used for presenting visual images, and is combined with the haptic presentation device 5 to realize touch-visual perception collaborative enhancement. The auditory device 7 is connected with the haptic interaction end 1, and is used for synchronously playing voice signals to realize touch-auditory perception collaborative enhancement.

[0050] Refer to Figure 2 In a preferred embodiment of the application, the haptic interaction end 1 comprises a wearable actuator array 11 and a vibration glove 12. The actuator array 11 comprises a micro actuator array and a thermal stimulation coil.

[0051] The micro-actuator array employs piezoelectric actuators and linear motor technology, arranged in a hexagonal grid structure, capable of accurately reflecting the contact position with the object, with single-point tactile feedback precision reaching 2mm. This high-precision design enables the system to simulate subtle surface texture changes, significantly enhancing the realism of the tactile experience. For example, when a user touches a wooden table in a virtual environment, the system can accurately simulate the subtle bumps and lines of the wood grain, allowing the user to clearly perceive the realistic texture of the material. The actuator array 11 and the vibration glove 12 constitute the tactile interaction end 1, which provides real-time precise tactile feedback through the actuators on the actuator array 11, including contact, sliding, tapping, vibration, texture, and material properties, and exchanges tactile interaction data with the computer system 4 in real time.

[0052] The thermal stimulation coil uses a combination of micro Peltier elements and temperature sensors, with a temperature range controllable between -5°C and +45°C, and a temperature rise / fall response time less than 2 seconds. The skin thermal feedback generated by temperature change improves the perception of the tactile object surface, simulating the hardness and elasticity characteristics of different materials. For example, when simulating metal material, the system will produce a slight cold feeling (about 18°C), while simulating wood will produce a mild warm feeling (about 25°C). In a virtual education scenario, students can touch different virtual material samples and feel the real temperature difference, thereby gaining a deeper understanding of the material physical properties.

[0053] In practical applications, the power consumption of the tactile interaction end 1 is controlled within 5W, which can be powered through a standard USB interface, ensuring comfort and portability for long-term use.

[0054] The distributed tactile feedback end 2 includes a tactile prediction unit, which constructs a 3D content tactile prediction algorithm based on matrix theory, and is one of the core innovations of the invention.

[0055] The tactile prediction unit realizes the mapping of visual features and tactile features through a predefined tactile texture matrix J and a tactile material matrix M. These two matrices are two-dimensional matrices that record tactile information at different positions in the tactile space. Specifically, the tactile texture matrix J ∈ R (m×n) where m represents the spatial resolution (number of position points), and n represents the tactile texture attribute dimension; the tactile material matrix M ∈ R (m×k) where k represents the material attribute dimension.

[0056] In a specific embodiment of the present application, the typical value of the spatial resolution m is 64x64, i.e. 4096 spatial position points, corresponding to the layout of the haptic actuator array; the typical value of the haptic texture attribute dimension n is 16, including roughness, hardness, elasticity, etc. characteristics; the typical value of the material attribute dimension k is 12, including temperature conductivity, friction coefficient, etc. characteristics. For example, when the user explores the surface of ancient artifacts in the VR environment, the system can accurately simulate the subtle texture of the pottery surface and the smooth feeling of the porcelain, helping the user to obtain more rich artifact information through touch.

[0057] The haptic prediction unit maps the haptic matrix J and M with the texture data delivered by the visual device 6, realizes material prediction and haptic feedback. The specific mapping process is as follows:

[0058] First, the key features are extracted from the visual texture features V, and the haptic texture matrix J is generated through the mapping function f(V):

[0059]

[0060] wherein, is the haptic texture matrix, representing the texture characteristics of each position point in the haptic space; is the visual texture feature vector, extracted from the visual device; is the visual feature transformation function, which maps the visual features to the haptic feature space; is the weight matrix, which controls the strength of the mapping; is the bias term, which is used to adjust the baseline haptic perception.

[0061] Similarly, the visual material features are mapped to the haptic material space through the mapping function to generate the haptic material matrix :

[0062]

[0063] wherein, is the haptic material matrix, representing the material characteristics of each position point in the haptic space; is the visual material feature vector, extracted from the visual device; is the material feature transformation function, which maps the visual material features to the haptic material space; is the weight matrix, which controls the strength of the mapping; is the bias term, which is used to adjust the baseline material perception.

[0064] In order to improve the calculation efficiency, the haptic prediction unit applies the singular value decomposition (SVD) technique of matrix to optimize the calculation process. Taking the haptic texture matrix as an example, its SVD decomposition is: ​​

[0065] ,

[0066] wherein, is the tactile texture matrix (dimension m x n); is the left singular vector matrix (dimension m x m), containing the feature information of spatial positions; is the singular value diagonal matrix (dimension m x n), the elements on the diagonal line represent the importance of each feature; is the right singular vector matrix (dimension n x n), containing the feature information of texture attributes; denotes the transpose of the matrix .

[0067] By retaining the first singular values accounting for 90% of the total information amount, the reduced dimension approximate matrix can be obtained:

[0068] ,

[0069] wherein, is the left singular vector matrix (dimension m x r) retaining the first columns; is the diagonal matrix (dimension r x r) retaining the first singular values; is the transpose of the right singular vector matrix (dimension r x n) retaining the first rows; is the number of retained singular values, which is usually much smaller than , in the embodiment the typical value of

[0070] This dimension reduction processing significantly reduces the computational complexity while retaining the main feature information of the matrix. In actual VR applications, such as a virtual sculpture creation environment, this optimization enables the system to increase the refresh rate to 60 Hz while maintaining the quality of tactile feedback, ensuring that users obtain smooth tactile feedback during the shaping process.

[0071] In an important embodiment of the present application, the tactile prediction unit realizes adaptive adjustment of tactile features based on matrix spectral analysis technology, improving the individualization degree and accuracy of the system.

[0072] The specific implementation is as follows: first, the tactile interaction data is collected by the distributed signal acquisition unit 3 to construct the interaction mode vector , which contains the magnitude of the applied force, the contact duration and the direction vector of the force

[0073] ,

[0074] where P is the interaction mode vector, containing complete haptic interaction information; F represents the force magnitude applied at m spatial position points, with units of Newton (N) and a typical range of 0-10 N; T represents the contact duration, with units of seconds (s) and a typical range of 0-2 s; D represents the force direction vector, containing x, y, z direction components, which is an m x 3 matrix, and each direction component ranges from [-1, 1], representing a unit vector.

[0075] Then, the system applies a spectral clustering algorithm to map the interaction mode vector P to four basic haptic mode spaces: contact (C), sliding (S), tapping (K), and vibration (V). The mode discrimination function h(P) is defined as:

[0076] ,

[0077] where h(P) is the mode discrimination function, and the output is the recognized haptic mode type; is the feature vector of each mode, which is determined in advance through training data; represents the vector inner product operation, which calculates the similarity between P and each mode feature vector; arg max represents the mode type corresponding to the maximum inner product value.

[0078] For example, in a VR medical training system, when trainees practice palpation skills, the system can recognize their touch patterns (gentle contact, sliding exploration, point tapping, or vibration detection) and provide corresponding haptic feedback according to different palpation patterns, allowing trainees to feel the tactile differences of different tissues.

[0079] According to the recognized haptic mode, the haptic texture matrix J and the haptic material matrix M are adjusted through the error correction function of matrix spectral analysis:

[0080] ,

[0081] ,

[0082] where and are the current haptic texture matrix and haptic material matrix, respectively; and are the updated matrices; and are the error correction functions based on matrix spectral analysis; and are the adaptive learning rates, with initial values of 0.05 and 0.03, respectively, gradually decreasing with the increase of interaction times to ensure the convergence of the algorithm. The update formula for the learning rate is where is the initial learning rate, is the decay coefficient (typical value is 0.01), is the number of interactions.

[0083] Error correction function The specific definition is:

[0084] ,

[0085] Where, is the error correction amount of the texture matrix; is the ideal haptic texture matrix inferred based on interaction data; is the weight coefficient (typical value is 0.8), which controls the strength of the correction; is the feature selection matrix, which is a diagonal matrix with diagonal elements of 0 or 1, used to determine the feature dimensions that need to be updated.

[0086] Similarly, The function is defined as:

[0087] ,

[0088] Where, is the error correction amount of the material matrix; is the ideal haptic material matrix inferred based on interaction data; is the weight coefficient (typical value is 0.7); is the material feature selection matrix, similar in structure to .

[0089] In practical applications, such as in VR game environments, when the player interacts with the same virtual object (such as a leather surface with different textures) multiple times, the system will gradually adapt to the player's haptic perception preferences. For example, if the player tends to lightly touch the surface to perceive fine textures, the system will enhance the feedback strength of the microscopic textures; if the player is used to pressing hard to perceive material elasticity, the system will adjust the elasticity feedback parameters to provide a haptic experience that is more in line with individual habits.

[0090] In order to ensure the stability and convergence of matrix adjustment, the system applies eigenvalue decomposition to extract the principal feature direction and eliminate redundant information. If the correlation coefficient between two feature dimensions is greater than 0.85 (this threshold is determined through a large number of user tests, which can effectively balance the dimension reduction effect and information preservation), they will be combined for processing to reduce the dimension of the feature space and improve computational efficiency. The formula for calculating the feature correlation coefficient is:

[0091] ,

[0092] Where, is the feature and the correlation coefficient between two features; the covariance of two features; and the standard deviation of two features, respectively.

[0093] Referring to Figure 6 In another innovative embodiment of the present application, the tactile prediction unit realizes tactile texture reconstruction from macro to micro through multi-scale matrix decomposition and fusion algorithm, solving the problem of single scale and loss of details in traditional tactile simulation.

[0094] First, the tactile texture matrix J is decomposed into multiple scale components:

[0095] ,

[0096] where, is the complete tactile texture matrix; is the component matrix of different scales, and the dimension of each component matrix is the same as ; represents macro-scale tactile features (such as overall shape and contour), represents the most micro-scale tactile features (such as fine texture and surface microstructure); is the number of scales, representing the number of decomposition levels.

[0097] This decomposition is based on the wavelet transform of the matrix, which guarantees the orthogonality of different scale features. In practical applications, the typical value of the scale number s is 5, setting 5 levels from macro to micro to meet the expression needs of different granularity tactile features. For different application scenarios, the value of s can be adjusted: in medical palpation simulation, the value of s can be set to 6-7 to capture finer tissue differences; while in industrial design applications, the value of s can be set to 4-5, focusing on macro tactile features.

[0098] The specific wavelet transform decomposition process is:

[0099] ,

[0100] where, is the component matrix of the th scale; is the wavelet transform matrix of the th scale; is the transpose matrix of . The wavelet transform matrix of different scales corresponds to wavelet basis functions of different frequency ranges.

[0101] Then, the mode-scale mapping function is constructed to map the tactile mode to the corresponding scale weight:

[0102] ,

[0103] where, is the mode-scale mapping function; is the output of the tactile mode discrimination function defined above, i.e., the recognized tactile mode type; the mapping result is the index of the most active scale, ranging from 1 to s.

[0104] Different scales of tactile features are activated according to different interaction modes:

[0105] Contact mode (C) mainly activates and (macro features);

[0106] Sliding mode (S) activates to (meso features);

[0107] Knocking mode (K) activates to (macro to meso features);

[0108] Vibrating mode (V) activates to (meso to micro features);

[0109] This mode-scale mapping is based on psychophysical studies of human tactile perception, for example, humans mainly perceive macro shapes when lightly touching object surfaces, while being more sensitive to meso texture features when sliding fingers. In the VR art creation platform, when artists use different touch methods to interact with the virtual canvas, the system will accordingly activate tactile features of different scales, enabling artists to accurately perceive full-spectrum tactile information from the overall texture of the canvas to subtle brush strokes.

[0110] According to the current interaction mode , the weight of each scale is calculated:

[0111] ,

[0112] where, is the weight of the i-th scale, ranging from [0, 1], and the sum of all weights is 1; is the center position of the currently mainly activated scale, determined by the mode-scale mapping function ; is the standard deviation of the weight distribution (typical value is 1.2), controlling the concentration of the weight distribution; is the natural exponential function; the denominator is the normalization factor, ensuring that the sum of all weights is 1.

[0113] This Gaussian distribution-based weight assignment ensures smooth transition between different scales. For example, for the contact mode, the value of μ is 1.5 (between scales 1 and 2), and the value of σ is 1.2, which will produce a set of smooth weight distribution dominated by scales 1 and 2.

[0114] Finally, the weighted fused haptic feature is generated:

[0115]

[0116] where, is the fused haptic texture matrix; is the weight of the i-th scale; is the component matrix of the i-th scale; denotes the summation operation, which accumulates all weighted scale components.

[0117] Similarly, the material matrix M also undergoes similar multi-scale decomposition and fusion processing, obtaining .

[0118] In order to ensure natural switching between different scales and avoid haptic jump, the system monitors the interactive state changes in real time, dynamically adjusts the scale weight, and constructs a smooth transition function:

[0119]

[0120] where, is the i-th scale weight at the current time t; is the weight at the next time; is the target weight, which is calculated by the current interactive mode; τ is the smoothing coefficient, taking the value of 0.15, which controls the smoothness of scale switching. The smaller the value of τ, the smoother the transition but the slower the response; the larger the value of τ, the faster the response but it may produce a sudden feeling. 0.15 is the balance value determined through user experience test.

[0121] In the VR surgery simulation system, this multi-scale haptic reconstruction technology enables surgeons to simultaneously perceive the overall elasticity and microscopic texture of the tissue, and when the doctor changes from lightly touching the tissue to pressing or sliding operation, the system can smoothly transition the haptic feedback and provide a coherent and consistent real touch feeling.

[0122] Referring to Figure 1 , in the embodiment of the present application, the distributed signal acquisition unit 3 includes a signal acquisition circuit, a haptic interaction device interface, and a haptic presentation device interface.

[0123] ​​The signal acquisition circuit collects haptic interaction data, and the sampling rate is set to 1000 Hz to ensure that high-frequency haptic interaction can be captured. The haptic interaction data is transmitted to the computer system 4 through the haptic interaction device interface, and to the haptic presentation device 5 through the haptic presentation device interface. The distributed signal acquisition unit 3 collects haptic interaction data from the haptic interaction end 1 and the distributed haptic feedback end 2, and feeds back the haptic signals of the haptic interaction end 1 and the haptic presentation device 5 to the computer system 4 through the communication network.

[0124] In the VR industrial training application, when the trainee operates the virtual mechanical device, the signal acquisition circuit can accurately capture different operation forces and contact methods to provide accurate haptic interaction data for the system, thereby generating a real mechanical operation touch feeling to help the trainee master the correct operation skills.

[0125] The signal processing delay is controlled within 5ms, ensuring that the response delay of the entire system is less than 10ms, close to the human tactile perception threshold. In addition, the wireless communication adopts a low-delay Bluetooth 5.0 protocol, with a data transmission rate of 3Mbps, meeting the real-time interaction requirements.

[0126] The computer system 4 includes a processor, a memory and a communication interface, wherein the memory stores a haptic perception collaborative enhancement algorithm.

[0127] The processor executes the haptic perception collaborative enhancement algorithm, connects with the haptic interaction end 1, the distributed haptic feedback end 2 and the distributed signal acquisition unit 3 through the communication interface, and communicates with the visual device 6 and the auditory device 7 through the communication network. The processor executes the haptic perception collaborative enhancement algorithm in cooperation with the visual device 6 and the auditory device 7, generates haptic feedback according to the haptic mode and material, and realizes the fusion of touch-visual and touch-auditory perception.

[0128] In actual application, the computer system 4 can be a dedicated VR processor or a general-purpose computer platform. The computing power requirement of the processor is at least 4 cores and 8 threads, with a main frequency of 2.5GHz or above; the memory capacity is at least 8GB; and the communication interface supports multiple communication protocols such as USB3.0, Bluetooth5.0 and Wi-Fi6.

[0129] In the VR education application, such as a virtual astronomy classroom, the computer system 4 can coordinate the visual presentation, haptic feedback and audio description of the surface of the star, so that students can feel the coordinated stimulation of vision, touch and hearing by touching the surface of different stars, thereby enhancing their understanding and memory of celestial characteristics.

[0130] Reference Figure 6 In the embodiment of the present application, the distributed haptic feedback end 2, the visual device 6 and the haptic presentation device 5 are jointly presented to provide the fusion of visual images and haptic feedback, and realize the touch-visual perception collaborative enhancement.

[0131] The haptic rendering device 5 simulates haptic properties, delivering haptic features such as the surface texture and material of virtual objects through tactile simulation, force-torque simulation, and temperature simulation. When the user interacts with virtual objects through the haptic interaction terminal 1, the three-dimensional model in the visual device 6 will simulate a corresponding response, for example, when the haptic interaction device simulates a hard object, the surface of the three-dimensional model in the visual image will appear to be indented when subjected to force, the haptic feedback terminal controls the vibration of the vibration device, simulating the characteristics of the indentation of the surface of the three-dimensional model.

[0132] In the application of virtual architectural design, when the designer touches the surface of different building materials, the system not only provides corresponding haptic feedback, but also visually presents the deformation and light and shadow changes of the materials, for example, pressing the surface of wood will see a slight indentation and feel the corresponding elastic feedback, while pressing the surface of concrete will hardly see deformation but feel a hard touch. This touch-visual synergy greatly enhances the perceived realism of material properties, helping designers make more accurate material selections.

[0133] This visual-haptic synergy enhancement mechanism significantly improves the user's sense of immersion. In experiments, compared with single haptic feedback, touch-visual synergy enhancement makes users report a 43% increase in presence, especially in material identification tasks.

[0134] Referring to Figure 7 In embodiments of the present application, the auditory device 7 plays voice signals synchronized with haptic feedback, realizing touch-auditory perception synergy enhancement.

[0135] When the user interacts with virtual objects, the system will simultaneously play corresponding sound effects, for example, knocking on wood will produce a crisp sound, and knocking on metal will produce a metallic sound. These sound effects are precisely synchronized with haptic feedback, with a delay of less than 5ms, ensuring the consistency of auditory and haptic perception synergy.

[0136] In the application of virtual music teaching, when students touch different virtual musical instruments, the system will provide haptic feedback (such as the damping of piano keys and the tension of violin strings) and the corresponding timbre at the same time, helping students establish the connection between haptic action and timbre, and speeding up the mastery of musical instrument playing skills. This touch-auditory synergy makes the learning process more intuitive and efficient.

[0137] The voice signals of the auditory device 7 are played synchronously with the haptic feedback, improving the fineness of the haptic feedback. Experiments show that touch-auditory synergy enhancement improves the user's haptic discrimination ability by about 25%, and reduces the perception threshold of subtle haptic differences by about 30%.

[0138] Referring to Figure 8 The present application also provides a VR haptic feedback method for multi-modal perception fusion, comprising the following steps:

[0139] In the first step, haptic interaction is provided through the haptic interaction terminal 1, and haptic interaction data is collected. Specifically, the user interacts with objects in the virtual environment through the wearable actuator array 11 and the vibration glove 12, and the system collects interaction data such as force, position, and duration in real time. In VR cultural relic restoration training, students can accurately perceive the texture and fragility of ancient ceramics through this system, and the system collects the touch force and gestures of the students, providing appropriate haptic feedback to prevent damage to cultural relics due to improper force in actual restoration.

[0140] In the second step, the distributed haptic feedback terminal 2 predicts the haptic material based on the visual material information and provides haptic feedback. In this step, the haptic prediction unit constructs a 3D content haptic prediction algorithm based on matrix theory, and predefines the haptic texture matrix J and the haptic material matrix M to realize the mapping of visual features and haptic features.

[0141] In the third step, the distributed signal collection unit 3 collects haptic interaction data and provides haptic signals. The signal collection circuit collects interaction data in real time, and transmits related signals through the haptic interaction device interface and the haptic presentation device interface.

[0142] In the fourth step, the computer system 4 collects haptic interaction data, realizes haptic prediction using a multi-modal perception fusion method, executes control of the haptic presentation device 5, and generates visual images and voice signals.

[0143] In the fifth step, the haptic presentation device 5 generates haptic feedback according to the haptic signals, simulating haptic characteristics and materials.

[0144] In the sixth step, the visual device 6 presents visual images in conjunction with the haptic presentation device 5, realizing touch-visual perception collaborative enhancement.

[0145] In the seventh step, the auditory device 7 synchronously plays voice signals, realizing touch-auditory perception collaborative enhancement.

[0146] In the step of predicting haptic materials in the distributed haptic feedback terminal 2, the haptic prediction unit realizes three core innovations: first, high-dimensional matrix is used to represent the haptic feature space; second, adaptive adjustment of haptic features is realized based on matrix spectral analysis technology; third, multi-scale matrix decomposition and fusion algorithm is used to realize haptic texture reconstruction from macro to micro.

[0147] In VR rehabilitation treatment applications, when patients perform hand rehabilitation training, the system works collaboratively through the seven steps to provide visual guidance, haptic feedback, and auditory cues, helping patients complete precise rehabilitation movements, and promoting neural plasticity through multi-modal sensory stimulation to accelerate the rehabilitation process.

[0148] In the practical implementation of the present application, in order to ensure that the data from different sources can be effectively fused, the system normalizes all input data. For tactile interaction data, the force F is normalized to the range [0, 1] by a linear transformation , wherein and are the preset minimum force (ON) and maximum force (10N) respectively; the time TI is normalized to the range [0, 1] using a similar linear transformation; the direction vector D is normalized to a unit vector by , wherein represents the Euclidean norm of the vector D.

[0149] For visual feature data, texture features are normalized to the range [-1, 1] and color features are normalized to the range [0, 1]. This unified data scale processing ensures the accuracy and effectiveness of multi-modal data fusion. For example, in the virtual textile design application, the system needs to fuse visual texture, color and tactile sensation, and the normalization processing ensures that these different types of data can be compared and fused on the same scale, enabling designers to accurately preview the visual and tactile effects of the fabric.

[0150] In addition, the system also implements a data synchronization mechanism to ensure the time alignment of tactile, visual and auditory data. The visual update rate is set to 90Hz, the tactile feedback update rate is 1000Hz, and the auditory signal update rate is 48kHz. Through interpolation and prediction algorithms, the system can achieve smooth transition between different update rates, ensuring the collaborative consistency of multi-modal perception. The interpolation algorithm uses cubic spline interpolation to ensure smooth transition between high-frequency tactile data and relatively low-frequency visual data, avoiding perceptual jumps or delays.

[0151] The VR tactile feedback system and method for multi-modal perception fusion provided by the present application realizes high-precision, personalized and multi-modal collaborative tactile feedback experience by innovatively applying matrix theory to construct a 3D content tactile prediction algorithm. The system uses high-dimensional matrices to represent the tactile feature space, realizes adaptive adjustment of tactile features based on matrix spectral analysis technology, realizes tactile texture reconstruction from macro to micro through multi-scale matrix decomposition and fusion algorithm, and combines tactile-visual and tactile-auditory perception collaborative enhancement mechanism, significantly improving the immersion and naturalness of VR system interaction.

[0152] The present application solves the problems of limited tactile feature expression, lack of adaptability and insufficient multi-modal perception collaboration in traditional tactile feedback systems, providing a new development path for VR tactile feedback technology, and having important theoretical value and application prospect.

[0153] The above merely provides the preferred embodiments of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall fall into the scope of protection of the present application.

Claims

1. A multimodal perceptual fusion VR haptic feedback system, characterized in that, The application relates to a distributed tactile interaction system, comprising: a tactile interaction terminal for providing tactile interaction and collecting tactile interaction data; a distributed tactile feedback terminal in communication connection with the tactile interaction terminal, for predicting tactile texture based on visual texture information, and providing tactile feedback; a distributed signal acquisition unit in communication connection with the tactile interaction terminal and the distributed tactile feedback terminal, for collecting the tactile interaction data and providing tactile signals; a computer system in communication connection with the tactile interaction terminal, the distributed tactile feedback terminal and the distributed signal acquisition unit, for collecting the tactile interaction data, realizing tactile prediction by using a multi-modal perception fusion method, executing control of a tactile presentation device, and generating visual images and voice signals; a tactile presentation device connected with the distributed tactile feedback terminal, for generating tactile feedback according to the tactile signals, simulating tactile characteristics and texture; a visual device in communication connection with the distributed tactile feedback terminal, for presenting the visual images, and realizing touch-visual perception synergistic enhancement in combination with the tactile presentation device; an auditory device connected with the tactile interaction terminal, for synchronously playing the voice signals, and realizing touch-auditory perception synergistic enhancement; wherein the distributed tactile feedback terminal comprises a tactile prediction unit, the tactile prediction unit constructs a 3D content tactile prediction algorithm based on matrix theory, the tactile prediction unit realizes mapping of visual features and tactile features by using a predefined tactile texture matrix and a tactile texture matrix, the tactile texture matrix and the tactile texture matrix adopt high-dimensional matrices to represent a tactile feature space, the tactile prediction unit realizes adaptive adjustment of tactile features based on a matrix spectrum analysis technology, and the tactile prediction unit realizes tactile texture reconstruction from macro to micro by using a multi-scale matrix decomposition and fusion algorithm.

2. The multimodal perceptual fusion based VR haptic feedback system of claim 1, wherein: The tactile interaction terminal comprises wearable actuator arrays and vibration gloves, wherein the actuator arrays comprise micro actuator arrays and thermal stimulation coils; the micro actuator arrays can accurately reflect contact positions with objects, and have single-point tactile feedback precision of 2 mm; the thermal stimulation coils generate skin thermal feedback through temperature changes, improve the perception accuracy of the surface of a tactile object, and simulate the hardness and elasticity characteristics of different textures.

3. The multimodal perceptual fusion based VR haptic feedback system of claim 1, wherein: The haptic texture matrix J and the haptic material matrix M predefined by the haptic prediction unit are two-dimensional matrices recording haptic information of different positions in a haptic space; the haptic texture matrix J ∈ R (m×n) , wherein m represents a spatial resolution, and n represents a haptic texture attribute dimension; the haptic material matrix M ∈ R (m×k) , wherein k represents a material attribute dimension; the haptic prediction unit maps the haptic texture matrix J and the haptic material matrix M with texture data delivered by the visual device to realize material prediction and haptic feedback.

4. The multimodal perceptual fusion based VR haptic feedback system of claim 3, wherein: The haptic prediction unit realizes adaptive adjustment of haptic features by collecting haptic interaction data, constructing an interaction mode vector P, which contains the magnitude of the applied force F∈R m , the contact duration T∈R m , and the direction vector of the force D∈R (m ×3) ; applying a spectral clustering algorithm to map the interaction mode vector P to four basic haptic mode spaces: contact, sliding, tapping, and vibration; and adjusting the haptic texture matrix J and the haptic material matrix M through an error correction function of matrix spectral analysis according to the identified haptic mode.

5. The multimodal perceptual fusion based VR haptic feedback system of claim 4, wherein: The haptic prediction unit realizes multi-scale haptic texture reconstruction by decomposing the haptic texture matrix J into multiple scale components , wherein represents macro-scale haptic features, represents the most micro-scale haptic features; a mode-scale mapping function is constructed to activate haptic features of different scales according to different haptic interaction modes; according to the current interaction mode, the scale weight is calculated , and a weighted fused haptic feature is generated ; the interaction state change is monitored in real time, the scale weight is dynamically adjusted, a smooth transition function is constructed, and natural switching between different scales is ensured.

6. The multimodal perceptual fusion based VR haptic feedback system of claim 1, wherein: The distributed signal acquisition unit comprises a signal acquisition circuit, a tactile interaction device interface and a tactile presentation device interface; the signal acquisition circuit collects tactile interaction data, transmits the tactile interaction data to the computer system through the tactile interaction device interface, and transmits the tactile interaction data to the tactile presentation device through the tactile presentation device interface; the distributed signal acquisition unit collects tactile interaction data from the tactile interaction terminal and the distributed tactile feedback terminal, feeds back tactile signals of the tactile interaction terminal and the tactile presentation device, and transmits the tactile signals to the computer system through a communication network.

7. The multimodal perceptual fusion based VR haptic feedback system of claim 1, wherein: The computer system comprises a processor, a memory and a communication interface, wherein the memory stores a haptic perception collaborative enhancement algorithm; the processor executes the haptic perception collaborative enhancement algorithm, connects with the haptic interaction terminal, the distributed haptic feedback terminal and the distributed signal acquisition unit through the communication interface, and communicates with the visual device and the auditory device by using a communication network; the processor executes the haptic perception collaborative enhancement algorithm in cooperation with the visual device and the auditory device, generates haptic feedback according to a haptic mode and a material, and realizes fusion of touch-visual and touch-auditory perception.

8. The multimodal perceptual fusion based VR haptic feedback system of claim 1, wherein: The distributed haptic feedback terminal, the visual device and the haptic presentation device are jointly presented to provide fusion of visual images and haptic feedback, and realize touch-visual perception collaborative enhancement; wherein the haptic presentation device simulates haptic characteristics, and transmits haptic features such as haptic material and texture of a virtual object surface through tactile simulation, torque simulation and temperature simulation.

9. The multimodal perceptual fusion based VR haptic feedback system of claim 1, wherein: The auditory device plays a voice signal synchronized with the haptic feedback to realize touch-auditory perception collaborative enhancement; the voice signal of the auditory device is played synchronously with the haptic feedback to improve the fineness of the haptic feedback.

10. A method of multimodal perceptual fusion VR haptic feedback using the multimodal perceptual fusion VR haptic feedback system of any of claims 1-9, characterized in that, The method comprises the following steps: providing haptic interaction through a haptic interaction terminal to collect haptic interaction data; predicting haptic material based on visual material information through a distributed haptic feedback terminal to provide haptic feedback; acquiring the haptic interaction data through a distributed signal acquisition unit to provide haptic signals; collecting the haptic interaction data through a computer system, realizing haptic prediction by using a multi-modal perception fusion method, executing control of a haptic presentation device, and generating visual images and voice signals; generating haptic feedback according to the haptic signals through a haptic presentation device to simulate haptic characteristics and material; presenting the visual images through a visual device to jointly present with the haptic presentation device, and realize touch-visual perception collaborative enhancement; synchronously playing the voice signals through an auditory device to realize touch-auditory perception collaborative enhancement; In the step of predicting haptic material in the distributed haptic feedback terminal, a 3D content haptic prediction algorithm based on matrix theory is constructed, a haptic texture matrix and a haptic material matrix are predefined to realize mapping of visual features and haptic features, the haptic texture matrix and the haptic material matrix use high-dimensional matrices to represent a haptic feature space, adaptive adjustment of haptic features is realized based on matrix spectral analysis technology, and haptic texture reconstruction from macro to micro is realized through a multi-scale matrix decomposition and fusion algorithm.

Images