Virtual interaction system for realizing multi-dimensional perception

Abstract

The invention discloses a virtual interaction system for realizing multi-dimensional perception, which relates to the technical field of virtual interaction, and comprises a man-machine interaction unit, a mapping matching unit, a data processing unit and a data processing unit, and the coordinate matching module is responsible for establishing a coordinate matching relation of the virtual object in the virtual space according to the size, shape and attitude parameters of the virtual object. The interaction device is operated to control the intelligent mark point in the virtual interaction system to interact with the target object, according to the position relation in the interaction process, the position posture of the interaction object is collected to determine whether contact occurs, event triggering and target interaction are achieved through the triggering control unit, a user senses the event, and the interaction efficiency is improved. Interaction requirements can be met, multi-target interaction control requirements are met, and the purpose of saving system resources is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of virtual interaction technology, in particular to a multi-dimensional perception virtual interaction system. BACKGROUND

[0002] In the Chinese patent with the application number 201310085951.9, a multi-dimensional perception virtual interaction system and implementation method are disclosed, and a function expansion system and implementation method based on the system are also disclosed. The system of the present application comprises a core processing unit, a user auxiliary unit, a 3D camera unit, a 3D projection unit, and can further comprise an expansion unit. The multi-dimensional perception virtual interaction system and implementation method of the present application can realize multi-dimensional perception virtual interaction, and can realize virtual interaction between users and the system from multiple dimensions. The virtual interaction scenarios include virtual playing of musical instruments, virtual meetings, virtual gatherings, etc. Based on multiple systems described in the present application, multi-dimensional perception virtual interaction of multiple users in space-time synchronization can be realized.

[0003] The above document realizes multi-dimensional perception virtual interaction. However, in the existing interaction system, the interaction process of the system is usually sampled in a periodic sampling manner. In the case of multi-target interaction, a large amount of node resources need to be occupied, and the calculation amount of the system is also increased. In addition, it is difficult to define the contact effect of force perception in virtual space, and it is difficult to convert virtual force according to the position and motion state of the object in the virtual environment and map it to the physical interaction device, so it is difficult to realize the force perception and interaction of the operator. SUMMARY

[0004] The present application aims to provide a multi-dimensional perception virtual interaction system to solve the problems in the background art.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solution: a multi-dimensional perception virtual interaction system, comprising:

[0006] A human-computer interaction unit is responsible for receiving instruction information sent by the user through the interaction device during the operation of the virtual device, and sending force feedback data generated in the computer simulated virtual environment, so that the user can perceive the tactile properties of the virtual object;

[0007] A mapping matching unit is responsible for establishing the coordinate pairing relationship of the virtual object in the virtual space according to the size, shape and posture parameters of the virtual object, and establishing the modal mapping model between different senses;

[0008] A behavior decision unit is responsible for obtaining the intelligent marker points and the position, posture and coordinate information of the contacted object fused with the virtual space positioning and force feedback interaction from the human-computer interaction unit and the mapping matching unit, and controlling the motion position of the object in the virtual space;

[0009] a contact detection unit for detecting the interference between the smart marker point and the object in real time, and when the object in the virtual space is in contact, the contact detection is performed, the force perception is calculated according to the contact data, the constraint data is formed and sent to the human-computer interaction unit, the human-computer interaction unit sends the constraint data to the interaction device to form real-time constraints, and the user perceives the virtual force through the interaction device;

[0010] a trigger control unit for establishing a distributed subsystem, the interaction state of each subsystem is sampled by a node sampler, and it is judged whether the trigger condition is met: the maximum depth of the overlapping area of the two contacted objects exceeds the preset value, if yes, the controller of the subsystem is controlled by the trigger, and the interaction state of the subsystem is sent to the controller, otherwise, the sampled value of the state is discarded;

[0011] a visualization unit for capturing external conditions in the virtual environment and converting them into object parameter information, and performing real-time rendering processing via GPU and OpenGL.

[0012] Preferably, the modal mapping model realizes a kind of mapping relationship, including visual and tactile mapping and auditory and tactile mapping, the visual and tactile mapping adjusts the vibration frequency and amplitude of the tactile feedback according to the object material, the auditory and tactile mapping synchronously adjusts the force of the tactile through the tone and loudness of the sound, and the mapping process involves tactile, visual, auditory multi-modal data acquisition, multi-modal data fusion, and multi-modal time synchronization and spatial alignment.

[0013] Preferably, the multi-modal data fusion process is based on machine learning and priority rules.

[0014] The priority rule is defined as follows: when the tactile, visual and auditory conflict, the tactile data is used as the reference;

[0015] The machine learning-based fusion: collect the interaction data of the user and the real object, and label the multi-modal tags of the interaction data, then use them to train the multi-modal fusion model, learn the implicit correlation between different sensory signals, and realize the three-modal collaboration.

[0016] Preferably, the multi-modal fusion model adopts a hierarchical architecture design, first receives multi-source heterogeneous data through the input layer, then each modal data flows into the special modal-specific encoder in parallel, then dynamically calculates the cross-modal interaction weight through the attention fusion layer, realizes the adaptive alignment and semantic integration between features, inputs the fused multi-modal data into the cross-modal LSTM for time-dependent modeling, and finally generates the prediction results related to the task through the output layer.

[0017] Preferably, the tactile force feedback is generated by calculating the reaction force based on the physical properties of the virtual object and outputting it through a force feedback device, and then modifying the fusion parameters in real time according to user behavior or environmental changes.

[0018] Preferably, the force sensing calculates the magnitude and direction of the virtual force based on the contact data to simulate real physical interaction. The constraint data aims to convert the tactile data into an instruction format that the interactive device can understand. For the contact point, the normal direction of the contact surface is determined by calculating the maximum depth of the overlapping area of ​​the two objects, and the global coordinates of the contact point are recorded.

[0019] Preferably, the constraint data generation includes the following limitations:

[0020] a. Perform data encapsulation to determine the magnitude and direction of the force in 3D space, the position of the contact point relative to the end of the device, and ensure data real-time performance;

[0021] b. Define constraint types, including fixing object position, allowing finite deformation, and limiting sliding speed;

[0022] c. Establish a data protocol and use the HID protocol to assign high transmission weights to high priority data.

[0023] Preferably, the controller of the interactive subsystem of the trigger control unit is controlled by a trigger mechanism. Let the sampling time of the system sampling node be ΔT, and the time series of system state sampling be S. s ,satisfy:

[0024] S s ={ΔT, 2ΔT, ..., KΔT}, K ∈ positive integers

[0025] Let the triggering time of the i-th subsystem be t. k When sending control information to the controller, the trigger sequence is E. t Then E t ={t1,t2,t3,···,t k}, where k is a positive integer, and for a single subsystem, t is taken as t. k and t k+1 The states at time t are x(t) k ) and x(t k+1 Error state quantity e i (t)=x(t k+1 )-x(t k ),t∈(t k ,t k+1 To ensure the stability of the entire interactive system, the following must be met:

[0026]

[0027] Wherein, Lambda is a symmetric positive definite matrix, e(t) is an error state matrix, e T (t) is the transpose matrix of e(t), is an adjustment factor, x(t) is a state matrix, x T (t) is the transpose matrix of x(t);

[0028] If the letter Lambda in the above formula is taken as a unit matrix, the following can be obtained:

[0029] After decoupling, for any subsystem, the following is true: N belongs to a positive integer, indicating the number of subsystems, i is in [1, N];

[0030] The time occupied by the system each time is a constant c i , and the distributed triggering condition satisfies:

[0031] |e i (t)|>|x i (t)|+c i

[0032] Then the triggering interval of the i-th system satisfies: t k+1 =t k +min|NDeltaT|;

[0033] The subsystem can only be triggered when its event time and the triggering time of the neighbor node occur at the same time, and then the controller is updated according to the state of itself and the neighbor.

[0034] Compared with the prior art, the beneficial effects of the present application are:

[0035] The present application controls the interaction between the intelligent marker point and the target object in the virtual interaction system through the operation of the interactive device, determines whether contact occurs according to the position relationship in the interaction process, and if contact occurs, triggers the corresponding interaction subsystem, on the one hand, forms constraint data and sends it to the man-machine interaction unit, and completes virtual force perception on the interactive device, on the other hand, visualizes the display on the visualization unit, through the triggering control unit, when interaction occurs each time, the interactive object sends the sampling value in the interaction process to the controller according to its own node information, realizes event triggering and target interaction, and feeds back the interaction force to the interactive device, which is perceived by the user, can meet the interaction demand, meets the multi-target interaction control demand, and achieves the purpose of saving system resources. BRIEF DESCRIPTION OF DRAWINGS

[0036] Fig. 1 The overall system structure schematic diagram provided by the embodiment of the present application;

[0037] Fig. 2A multi-modal fusion model construction schematic diagram provided for an embodiment of the present application is shown in the figure.

[0038] Fig. 3 A subsystem trigger control structure schematic diagram provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0040] Please refer to Figs. 1 to 3 The present application provides a technical solution: a multi-dimensional perception virtual interaction system, comprising:

[0041] A human-computer interaction unit is responsible for receiving instruction information sent by an interaction device in the process of operating a virtual device by a user, and sending force feedback data generated in a computer simulated virtual environment, so that the user perceives the tactile properties of virtual objects;

[0042] A mapping matching unit is responsible for establishing coordinate pairing relationships of virtual objects in a virtual space according to size, shape and posture parameters of the virtual objects, and establishing a modal mapping model between different senses, i.e. the sense information of vision and hearing is kept synchronous in time and space logic, when the user touches the virtual object, the delay of tactile feedback needs to be less than 50 milliseconds, and the contact effect of the hand and the object is displayed synchronously in vision;

[0043] A behavior decision unit is responsible for obtaining intelligent marker points (which can mark positions in a virtual environment, and real-time perceive, simulate or transfer mechanical interaction) and the position, posture and coordinate information of the contacted object, which are fused with virtual space positioning and force feedback interaction, from the human-computer interaction unit and the mapping matching unit, and controlling the motion position of the object in the virtual space;

[0044] A contact detection unit is used for real-time detection of the interference between the intelligent marker points and the object, so as to realize contact detection. When the object in the virtual space is contacted, contact detection is performed, force perception is calculated according to the contact data, constraint data is formed and sent to the human-computer interaction unit, the human-computer interaction unit sends the constraint data to the interaction device, real-time constraints are formed, and the user perceives the virtual force through the interaction device;

[0045] A trigger control unit is configured to establish a distributed subsystem, each subsystem interacting with a node sampler to collect information and determine whether a trigger condition is met, i.e., the maximum depth of the overlapping area of the two contact objects exceeds a preset value, if the trigger condition is met, the controller of the subsystem is controlled by a trigger, and the interaction state of the subsystem is sent to the controller, otherwise, the sampled value of the state is discarded.

[0046] A visualization unit is configured to capture external conditions in a virtual environment and convert them into object parameter information, and perform real-time rendering processing via a GPU and OpenGL.

[0047] The application will be further described below in conjunction with Examples 1 to 3:

[0048] Example 1

[0049] This example is used to describe the kind of mapping relationship realized by the modal mapping model, including visual and tactile mapping and auditory and tactile mapping, the visual and tactile mapping adjusts the vibration frequency and amplitude of the tactile feedback according to the object material, and the auditory and tactile mapping synchronously adjusts the force of the tactile feedback through the pitch and loudness of the sound, for example, when knocking a hard object, the tactile feedback is more intense and high-frequency, the mapping process involves tactile, visual, auditory multi-modal data acquisition, multi-modal data fusion, and multi-modal time synchronization and spatial alignment;

[0050] When collecting tactile data, a force feedback device is used to capture the force, torque and contact position applied by the user, the reaction force on the surface of the virtual object is measured by a piezoelectric sensor, and the sampling rate needs to be not less than 1 kHz to capture dynamic changes;

[0051] When collecting visual data, a 3D camera or a VR headset is used to track the position of the user's hand or tool in real time, and a SLAM algorithm is used to construct a virtual environment spatial coordinate system to ensure spatial alignment of the tactile feedback and the visual scene;

[0052] When collecting auditory data, a microphone array is used to capture environmental sounds, or synthesized sound effects matching the interactive action, and a HRTF model is used to realize 3D spatial audio, so that the user can determine the orientation of the virtual object through the sound;

[0053] The multi-modal data fusion process is based on machine learning and priority rules;

[0054] The priority rules are defined as follows: when the tactile and visual, auditory conflict, the tactile data is used as the reference, for example, in a virtual grasping task, if the visual display shows that the object is held but the tactile does not detect pressure, the system will trigger a visual correction because the tactile is more directly responsive to physical interaction;

[0055] Machine learning-based fusion: Collect interaction data between users and real objects, such as grip strength, object deformation, and sound, and label these data with multi-modal labels, then use them to train a multi-modal fusion model to learn the implicit relationship between different sensory signals, such as in virtual piano playing, the model can predict the corresponding tactile vibration frequency of the piano key pressing force and the sound pitch, realizing three modalities collaboration;

[0056] The multi-modal fusion model adopts a hierarchical architecture design. First, the input layer receives multi-source heterogeneous data, then the modal data flows into the modal-specific encoder in parallel (touch uses BiLSTM to extract vibration sequence features, vision uses CNN+LSTM to fuse space-time features, and audio uses LSTM to process time-series patterns), then the attention fusion layer dynamically calculates the cross-modal interaction weight to realize adaptive alignment and semantic integration between features, and finally the multi-modal data after fusion is input into the cross-modal LSTM for time-dependent modeling, and the output layer generates the prediction results related to the task;

[0057] The multi-modal fusion model is used to integrate data from different modalities to improve the perception and understanding ability of the model. The spatial reference system of the touch device, the visual camera, and the virtual environment is unified through the coordinate system transformation matrix, the timestamp is embedded in the data packet, and the receiving end compensates for the transmission delay through the interpolation algorithm to ensure that the time error of the touch, visual, and auditory data is less than 5 milliseconds;

[0058] Haptic force feedback: According to the physical properties of virtual objects, the reaction force is calculated and output through the force feedback device. Then, the fusion parameters are modified in real time according to user behavior or environmental changes, for example, when the user accelerates the movement, the vibration amplitude of the haptic feedback is reduced to compensate for the lack of auditory information, and when the environmental noise increases, the vibration amplitude of the haptic feedback is increased to compensate for the lack of auditory information.

[0059] Embodiment 2:

[0060] This embodiment is used to describe the generation of constraint data. First, force perception calculates the size and direction of virtual force according to contact data to simulate real physical interaction. Constraint data aims to convert haptic data into a format that interaction devices can understand. For contact points, the normal direction of the contact surface is determined by calculating the maximum depth of the overlapping area of the two objects, and the global coordinates of the contact point are recorded.

[0061] When generating constraint data, the following restrictions are included:

[0062] a. Data encapsulation: determine the size and direction of the force in 3D space, the position of the contact point relative to the end of the device, and ensure the real-time nature of the data;

[0063] b. Define constraint types, including fixed object position, allow limited deformation, and limit sliding speed;

[0064] c. Establish a data protocol and use the HID protocol to assign high transmission weights to high priority data.

[0065] Example 3:

[0066] This embodiment describes the control mechanism of the controller of the interactive subsystem of the trigger control unit being controlled by a trigger. Let the sampling time of the system sampling node be ΔT, and the time series of system state sampling be S. s ,satisfy:

[0067] S s ={ΔT, 2ΔT, ..., KΔT}, K ∈ positive integers

[0068] Let the triggering time of the i-th subsystem be t. k When sending control information to the controller, the trigger sequence is E. t Then E t ={t1,t2,t3,···,t k}, where k is a positive integer, and for a single subsystem, t is taken as t. k and t k+1 The states at time t are x(t) k ) and x(t k+1 Error state quantity e i (t)=x(t k+1 )-x(t k ),t∈(t k ,t k+1 To ensure the stability of the entire interactive system, the following must be met:

[0069]

[0070] Where Λ is a symmetric positive definite matrix, e(t) is the error state matrix, and e T (t) is the transpose of e(t). Let x(t) be the adjustment factor, and x(t) be the state matrix. T (t) is the transpose of x(t);

[0071] If we take the letter Λ in the above formula as the identity matrix, we can obtain:

[0072] After decoupling, for any subsystem, we have: N is a positive integer, representing the number of subsystems, i∈[1,N];

[0073] The time taken for each system trigger is a constant c. i The distributed triggering conditions are met:

[0074] |e i (t)|>|xi (t)|+c i

[0075] then the triggering interval of the ith system satisfies: t k+1 = t k + min|NΔT|;

[0076] Therefore, each subsystem can only be triggered when its event time and the triggering time of its neighbor nodes occur simultaneously, and then the controller is updated according to the states of itself and its neighbors, i.e. only in the interval of (t k ,t k+1 ], the controller can be updated when the next triggering time t k+1 comes or at least one of its neighbors is triggered, thus the update rate of the controller can be greatly reduced.

[0077] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one from another entity or action, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0078] Although the embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A multi-dimensional perception virtual interaction system, comprising: The system comprises: a human-computer interaction unit responsible for receiving instruction information sent by the user through the interactive device during the operation of the virtual device, and sending force feedback data generated in the computer simulated virtual environment, so that the user can perceive the tactile properties of the virtual object; a mapping matching unit responsible for establishing the coordinate pairing relationship of the virtual object in the virtual space according to the size, shape and attitude parameters of the virtual object, and establishing a modal mapping model between different senses; a behavior decision unit responsible for obtaining the intelligent marker points and the position, attitude and coordinate information of the contacted object from the human-computer interaction unit and the mapping matching unit, and controlling the motion position of the object in the virtual space; a contact detection unit for real-time detection of the interference between the intelligent marker points and the object, and when the object in the virtual space is contacted, the contact detection is carried out, the force perception is calculated according to the contact data, the constraint data is formed and sent to the human-computer interaction unit, and the human-computer interaction unit sends the constraint data to the interactive device to form real-time constraints, and the user perceives the virtual force through the interactive device; a trigger control unit for establishing a distributed subsystem, and each subsystem interacts with the information collector to judge whether the trigger condition is met: the maximum depth of the overlapping area of the two contacted objects exceeds the preset value, if yes, the controller of the subsystem is controlled by the trigger, and the interaction state of the subsystem is sent to the controller, otherwise, the sampling value of the state is discarded; a visualization unit for capturing external conditions in the virtual environment and converting them into object parameter information, and performing real-time rendering processing through GPU and OpenGL.

2. The multi-dimensional perception virtual interaction system according to claim 1, wherein: The modal mapping model realizes the mapping relationship of different categories, including visual and tactile mapping and auditory and tactile mapping. The visual and tactile mapping adjusts the vibration frequency and amplitude of the tactile feedback according to the object material. The auditory and tactile mapping synchronously adjusts the force of the tactile feedback through the tone and loudness of the sound. The mapping process involves multi-modal data acquisition, multi-modal data fusion, and multi-modal time synchronization and spatial alignment.

3. The multi-dimensional perception virtual interaction system according to claim 2, wherein: The multi-modal data fusion process is based on machine learning and priority rules; The priority rules are defined as follows: when the tactile and visual and auditory data conflict, the tactile data is used as the reference; The machine learning based fusion process involves collecting user interaction data with real objects, labeling the multi-modal labels of the interaction data, and then using the multi-modal fusion model to learn the implicit correlation between different sensory signals to realize the three-modal collaboration.

4. The multi-dimensional perception virtual interaction system according to claim 3, wherein: The multi-modal fusion model adopts a hierarchical architecture design. First, the multi-source heterogeneous data is received through the input layer, then the modal data flows into the special modal specific encoder in parallel, then the cross-modal interaction weight is dynamically calculated through the attention fusion layer to realize the adaptive alignment and semantic integration between features, the fused multi-modal data is input into the cross-modal LSTM for time-dependent modeling, and finally the output layer generates the prediction results related to the task.

5. The multi-dimensional perception virtual interaction system according to claim 1, wherein: The tactile generated force feedback calculates the reaction force according to the physical properties of the virtual object, and outputs it through the force feedback device, and then modifies the fusion parameters in real time according to the user behavior or environmental changes.

6. The multi-dimensional perception virtual interaction system according to claim 1, wherein: The force perception calculates the size and direction of virtual force according to the contact data, simulates real physical interaction, and the constraint data is intended to convert the haptic data into a command format that the interaction device can understand. For the contact point, the normal direction of the contact surface is determined by calculating the maximum depth of the overlapping area of the two objects, and the global coordinates of the contact point are recorded.

7. The multi-dimensional perception virtual interaction system according to claim 1, wherein: The constraint data generation includes the following steps: a. Data encapsulation, determine the size and direction of force in 3D space, the position of the contact point relative to the end of the device, and ensure data real-time; b. Define the constraint type, including fixing the position of the object, allowing limited deformation and limiting the sliding speed; c. Achieve data protocol, give high transmission weight to high priority force by using HID protocol.

8. The multi-dimensional perception virtual interaction system according to claim 1, wherein: The controller of the interactive subsystem of the trigger control unit is controlled by the trigger control mechanism, and the sampling time of the system sampling node is ΔT, and the time sequence of the system state sampling is S s , which satisfies: S s = {ΔT, 2ΔT, ···, KΔT}, K ∈ positive integer Let the triggering time of the i-th subsystem be t. k When sending control information to the controller, the trigger sequence is E. t Then E t ={t1,t2,t3,···,t k }, where k is a positive integer, and for a single subsystem, t is taken as t. k and t k+1 The states at time t are x(t) k ) and x(t k+1 Error state quantity e i (t)=x(t k+1 )-x(t k ),t∈(t k ,t k+1 To ensure the stability of the entire interactive system, the following must be met: where Λ is a symmetric positive definite matrix, e(t) is an error state matrix, e T (t) is the transpose matrix of e(t), is an adjustment factor, x(t) is a state matrix, x T (t) is the transpose matrix of x(t); If the letter A in the above formula is taken as a unit matrix, we have: After decoupling, for any subsystem has: N is a positive integer, representing the number of subsystems, i∈[1,N]; The time taken by the system to trigger each time is a constant c i The distributed trigger condition is satisfied: |e i (t)|>|x i (t)|+c i The ith system's trigger interval satisfies: t k+1 = t k + min | N ΔT | ; The subsystem can only be triggered when its event time and the trigger time of the neighbor node occur at the same time, and then the controller is updated according to the state of itself and the neighbor.

Images