Multi-perception interaction method and system based on virtual reality

Abstract

The invention discloses a multi-perception interaction method and system based on virtual reality. The method comprises the following steps: creating a virtual scene and an intelligent agent based on an interaction demand, and constructing a perception model of the virtual scene and the intelligent agent; the intelligent agent perceives visual, auditory and tactile environments, extracts perceived feature time sequence data, divides instructions according to the perceived feature time sequence data, and calculates instruction priorities. And generating a decision network according to a preset knowledge graph, and forming an instruction chain in combination with the instruction priority, so that the intelligent agent obtains a feedback strategy. A dynamic weight value is distributed to a feedback strategy, a virtual scene is updated through fusion of an attention mechanism and a perception model, an interaction mode is obtained, meanwhile, physical and psychological parameters of a user are recorded in real time, abnormal fluctuation segments are recognized, and emotional tendency of the user is learned from corresponding interaction, so that the interaction mode is continuously optimized. According to the method, the interaction experience can be improved, the intelligent agent can better meet the user requirement, the interaction effectiveness and adaptability are enhanced, and meanwhile good interpretability is achieved.

Description

Technical Field

[0001] The present invention relates to the field of virtual interaction, and in particular to a multi-sensory interaction method and system based on virtual reality. Background Art

[0002] With the development of science and technology, virtual reality technology has been widely used in many fields. However, traditional virtual reality interaction methods have many limitations, which has prompted the emergence of multi-sensory interaction methods and systems based on virtual reality.

[0003] In traditional virtual reality, the interaction method is often relatively simple, mainly relying on simple devices such as handles for operation, and the user experience is relatively stiff and lacks realism. At the same time, human perception in the real world is multi-dimensional, including the synergy of multiple senses such as vision, hearing, and touch. However, existing virtual reality technology has shortcomings in simulating these multi-sensory interactions, making it difficult for users to immerse themselves in it in all aspects.

[0004] With the continuous progress in artificial intelligence, sensor technology and other related fields, it is possible to achieve more complex and intelligent multi-sensory interactions. An innovative multi-sensory interaction method and system based on virtual reality fully integrates multiple sensing technologies to enhance the user's interactive experience and immersion in the virtual environment, meet the growing application needs in different fields, such as medical training, industrial design, entertainment games, etc., and promote the development of virtual reality technology to a higher level. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-sensory interaction method and system based on virtual reality.

[0006] To achieve the above object, the present invention is implemented according to the following technical solutions:

[0007] A first aspect of the present invention provides a multi-sensory interaction method based on virtual reality, comprising:

[0008] S100 creates a virtual scene and an intelligent agent based on interaction requirements, and builds a perception model for the intelligent agent;

[0009] S200 performs visual, auditory and tactile environmental perception through the agent, extracts time series data of perception features, divides instructions based on the time series data, and calculates instruction priorities;

[0010] S300 generates a decision network according to a preset knowledge graph, generates an instruction chain according to the priority of the instruction, and the intelligent agent obtains a feedback strategy based on the decision network and the instruction chain;

[0011] S400 allocates a dynamic weight to the feedback strategy, fuses the dynamic weight with the perception model through an attention mechanism, updates the virtual scene, and combines it with the feedback strategy to obtain an interaction mode;

[0012] S500 records the user's physical and mental parameters in real time, identifies abnormal fluctuation segments, learns the user's emotional tendencies based on the interactions corresponding to the abnormal fluctuation segments, and continuously optimizes the feedback strategy.

[0013] As a further method, the method of creating a virtual scene and an intelligent agent based on interaction requirements and constructing a perception model for the intelligent agent includes:

[0014] Create object and scene models based on the elements, functions and characteristics required for interaction to form a virtual scene;

[0015] Based on the purpose of interaction requirements, the role and function of the intelligent agent are determined, and the internal architecture is constructed, including: knowledge storage and decision-making modules, where the state transfer function of the intelligent agent is expressed as:

[0016]

[0017] Among them, s is the current state of the agent, e is the input event, z is the decision state taken by the agent, f is the environmental factor, t is the time variable, and s * is the next state that the agent transfers to in a given state, P(s * |s,e,z,f,t) is to transfer to state s in a given state * The probability of, μ(s,e,z,f,t) is the expected value of the next state based on a given state, and σ is the time step;

[0018] A multi-sensory self-attention model is introduced to add multimodal fusion function to the intelligent agent and obtain a perception model. Specifically, for visual data, the image frame is downsampled to a preset size and the area is randomly cropped during training; for audio data, the signal is resampled at a preset frequency and the short-time Fourier transform is calculated to generate a mel spectrum; for tactile data, a tactile image is generated based on optical tracking and deformation detection;

[0019] A multi-head attention mechanism is implemented in the perception model to process data of different modalities. The expression of the multi-head attention mechanism is:

[0020]

[0021] Among them, Q, K and V are query matrix, key matrix and value matrix respectively, W 0 is the linear transformation matrix of the output layer, Concat represents the concatenation operation, n is the number of attention heads, and W iis the weight matrix of the ith attention head, T represents the transposition operation, d i is the size of the i-th attention head dimension, A i is the bias term of the i-th attention head.

[0022] As a further method, the method of dividing instructions based on the time series data and calculating instruction priorities includes:

[0023] The time series data is divided into different instructions according to the preset interaction rules, which specifically includes: obtaining a threshold from the preset interaction rules, and if the time series data exceeds the threshold, it is judged as interaction and an instruction is obtained;

[0024] Duplicate instructions and combined instructions are identified based on the similarity metric and information metric of the instructions, where the calculation formulas of the similarity metric and the information metric are as follows:

[0025]

[0026]

[0027] Where W(P, Q) is the Wasserstein distance between the probability distributions P and Q of two instructions, inf is the infimum, which means finding the value with the smallest expected value among all possible γ, γ is the joint distribution, Π(P, Q) is the set of transitions from the probability distribution of instruction P to instruction Q, It is expressed as the expected distance between random variables X and Y under the joint distribution γ, ||XY|| is the Euclidean distance between X and Y, I(P, Q) is the mutual information between the probability distributions P and Q of the two instructions, p(x, y) is the joint probability distribution of X and Y, p(x) and p(y) are the marginal probability distributions of X and Y respectively;

[0028] After normalizing the similarity metric and information metric from 0 to 1, the instructions with similarity metric greater than 0.7 and information metric greater than 0.8 are combined instructions, and the instructions with similarity metric greater than 0.7 and information metric less than or equal to 0.8 are repeated instructions. For repeated instructions, the instruction with the largest joint probability distribution is selected as the final instruction;

[0029] Calculate the priority of each instruction, the expression is:

[0030]

[0031] Among them, P i is the priority of the ith instruction, t c is the current time, t i is the association time of the ith instruction, t t is the total duration of the entire task, η iis the number of times the i-th instruction appears, η t is the total number of instructions, n i，s is the number of operation steps involved in executing the i-th instruction, n i，p is the number of parameters involved in the ith instruction, t i，r The estimated response time required to complete the i-th instruction, d i is the degree of dependence of the ith instruction on other instructions, μ i is the urgency of the ith instruction.

[0032] As a further method, the method of generating a decision network according to a preset knowledge graph and generating an instruction chain according to the priority of the instructions includes:

[0033] Build a decision network based on the decision tree, and concatenate the vector representations of entities and relationships in the preset knowledge graph with the output vector of the perception model as the input of the decision network;

[0034] The rules in the preset knowledge graph are used as the branch rules of the decision tree in the decision network. The decision network is trained to learn the rules in the knowledge graph by minimizing the loss function. The expression of the loss function is:

[0035]

[0036] Among them, |T| is the total number of leaf nodes in the decision tree T, K is the total number of categories of nodes in the decision tree, and N tk is the number of samples belonging to category k that fall on leaf node t, N t is the total number of samples falling on leaf node t, α is the regularization parameter;

[0037] Instructions are sorted according to their priority, with instructions with higher priority placed in front, forming an instruction chain.

[0038] As a further method, the method for the intelligent agent to obtain a feedback strategy based on a decision network and a command chain includes:

[0039] The instruction chain is passed as input to the decision network, including the feature vector and priority of the instruction. The decision path with the shortest time required to complete the task is searched in the decision network as a feedback strategy. The probability distribution expression of different decision paths is:

[0040]

[0041] Where δ is the number of decision paths, x i is the feature vector of the i-th decision path, m is x i, ∑ is the covariance matrix of the decision path distribution, μ is the mean vector of the decision path distribution, and T is the transpose operation.

[0042] As a further method, the method of assigning dynamic weights to the feedback strategy and fusing the dynamic weights with the perception model through an attention mechanism includes:

[0043] Obtain visual, auditory, and tactile feedback decisions from the feedback strategy, and assign a weight value of the corresponding number of instructions in the proportion of all instructions to each feedback decision;

[0044] The feature representations of different perceptual channels are extracted from the perceptual model, the assigned weight values ​​are weightedly fused with the features of each perceptual channel, the attention weight of each channel is calculated, and according to the attention weights, the feature maps of different perceptual channels are weightedly fused by channel-by-channel multiplication.

[0045] As a further method, the method for updating the virtual scene includes:

[0046] Constructing a scene graph of a virtual scene, wherein nodes include physical nodes, role nodes and environment nodes, and edges include spatial relationship edges, interaction relationship edges and causal relationship edges;

[0047] The information in the fused feature graph is associated with the nodes and edges in the scene graph, the attributes of the nodes and the relationship between the edges in the scene graph are updated, and then the virtual scene is updated.

[0048] As a further method, the method for interactively learning the user's emotional tendency based on the abnormal fluctuation segment includes:

[0049] According to the interaction corresponding to the abnormal fluctuation segment, the state value function and action value function of the abnormal fluctuation segment are obtained:

[0050] The expression of the state value function is:

[0051]

[0052] Where h is the number of characteristic functions, ω i is the weight of the i-th feature function, β i is the linear coefficient of the i-th characteristic function, α i is the nonlinear coefficient of the i-th characteristic function, s is the current state of the agent, and f i (s) is the output of the i-th feature function extracted from state s, Φ is the state space, p(θ|S) is the probability of transitioning from state s to state θ, V(θ) is the value of the next state θ, ∈ is the adjustment term, and ζ is the threshold parameter used to control the center point of the tanh function;

[0053] The expression of the action value function is:

[0054]

[0055] Where s is the current state of the agent, r(s, a) is the immediate reward for performing action a in state s, γ is the discount factor, Φ is the state space, P(θ|s, a) is the probability of transferring to the next state θ after taking action a in state s, V(θ) is the value of the next state θ, m is the number of dimensions of the action, and v j is the weight of the jth dimension, δ j is the linear coefficient of the jth dimension, g j (θ, a) is the additional value calculated based on state θ, action a and the jth dimension, ξ j is the attenuation coefficient of the jth dimension, λ j is the Gaussian decay rate of the j-th dimension.

[0056] As a further method, the method for continuously optimizing the feedback strategy includes: analyzing the interaction features in the abnormal fluctuation segment, marking the state variables of the interaction features related to the anomaly, setting a penalty coefficient to reduce the value of the state variables, updating the state value function and the action value function, using a deep Q network to update the agent's policy network according to the updated value function, and optimizing the feedback strategy.

[0057] A second aspect of the present invention provides a multi-sensory interaction system based on virtual reality, comprising:

[0058] A virtual scene and intelligent agent creation module, used to create virtual scenes and intelligent agents based on interaction requirements and build a perception model for the intelligent agents;

[0059] An instruction processing module is used to generate a decision network according to a preset knowledge graph, generate an instruction chain according to the priority of the instruction, and the intelligent agent obtains a feedback strategy based on the decision network and the instruction chain;

[0060] A decision generation module, used to generate a decision network according to a preset knowledge graph, generate an instruction chain according to the priority of the instruction, and the intelligent agent obtains a feedback strategy based on the instruction chain and the decision network;

[0061] A feedback and scene update module, used to assign dynamic weights to the feedback strategy, fuse the dynamic weights with the perception model through an attention mechanism, update the virtual scene, and combine it with the feedback strategy to obtain an interaction mode;

[0062] The user emotion optimization module is used to record the user's physical and mental parameters in real time, identify abnormal fluctuation segments, learn the user's emotional tendencies based on the interactions corresponding to the abnormal fluctuation segments, and continuously optimize the feedback strategy.

[0063] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects:

[0064] (1) The present invention generates a decision network by utilizing a preset knowledge graph, enabling the intelligent agent to make decisions based on a rich knowledge reserve. Whether facing different types of virtual scenes or diverse user behavior patterns, the intelligent agent can generate reasonable feedback strategies.

[0065] (2) The present invention generates an instruction chain through instruction priority, and the intelligent agent can execute the feedback strategy in an orderly manner, avoiding disorderly processing of a large number of instructions, making the interaction more efficient, and improving the efficiency and quality of the interaction.

[0066] (3) The present invention adjusts the virtual scene and interaction according to the user's emotional inclination, so that the atmosphere and interaction mode of the virtual scene match the user's emotional state, thereby enhancing the user's perception of the authenticity of the virtual scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 The present invention is a flowchart of a multi-sensory interaction method based on virtual reality in an embodiment of the present invention.

[0068] Figure 2 The figure is a logical schematic diagram of a multi-sensory interaction system based on virtual reality in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0070] Reference Figure 1 As shown, the present invention provides a multi-sensory interaction method based on virtual reality, comprising:

[0071] S100 creates a virtual scene and an intelligent agent based on interaction requirements, and builds a perception model for the intelligent agent;

[0072] In a specific embodiment based on the interactive requirements of surgical training, the experimental environment is: virtual simulation software Unity3D and Kinerct2.0 development, virtual invention or system development configuration requirements are: CPU: support 64-bit, graphics card: support DX11, operating system: Windows8 version or above, memory: 4G or above, VS version: 2012 version and above;

[0073] In the actual evaluation, a virtual scene is created, specifically: an operating table, various surgical instruments (scalpels, tweezers, hemostats, etc.), human model objects and scene models are created to form a virtual operating room scene. The scene contains environmental elements such as lighting and background sounds in the operating room. The elements have the functions and characteristics of a real operating room. The surgical lights can adjust the brightness and angle, and the instruments have different physical properties and operation methods. The intelligent agent is determined to be a surgical assistant, whose functions include assisting the surgeon in passing instruments and providing necessary prompts according to the progress of the operation. The internal architecture of the intelligent agent includes a knowledge storage module that stores knowledge such as surgical procedures and instrument usage methods. The decision module is used to make decisions based on perceived information. The parameters in its state transfer function are set as follows: the current state s t Indicates that the agent is waiting for the scalpel instruction to be delivered, input event e t The doctor reaches out to indicate that he needs a scalpel. The environmental factors e include the noise level in the operating room (normal noise value is 30 decibels) and the lighting conditions of the operating table (good). The time variable t is the 10th minute of the current operation. The decision state s taken by the agent is calculated through the state transfer function. t+1 for picking up the scalpel and passing it to the doctor;

[0074] In the actual evaluation, a multi-sensory self-attention model is introduced to construct a perception model. Specifically, for visual data, the image frame is downsampled from the original resolution of 1920×1080 to 224×224, and a 200×200 area is randomly cropped during training, which can reduce the amount of data while retaining key visual information, including the shape of surgical instruments, the doctor's gestures, etc.; for audio data, the sound signal in the operating room is resampled at a preset frequency of 44.1kHz, and the short-time Fourier transform is calculated to generate a mel spectrum map; for tactile data, a tactile image is generated based on optical tracking and deformation detection. When the intelligent agent picks up the surgical instrument, it can perceive the weight and grip of the instrument through the tactile data; a multi-head attention mechanism is implemented in the perception model, and the number of attention heads is set to 8 to process data of different modalities.

[0075] S200 performs visual, auditory and tactile environmental perception through the agent, extracts time series data of perception features, divides instructions based on the time series data, and calculates instruction priorities;

[0076] It needs to be explained that through visual, auditory and tactile environmental perception, the intelligent agent can obtain rich and comprehensive environmental information. Visual perception can capture the shape, color, motion trajectory, etc. of an object, such as accurately identifying the shape and position of parts in a virtual assembly scene. Auditory perception can receive voice commands, environmental sound effects, etc., such as hearing the instructor's words clearly in a training scene. Tactile perception provides information such as the texture, hardness, and surface texture of an object, such as sensing the contact between instruments and tissues in simulated surgery. This multi-sensory fusion makes the intelligent agent's understanding of the environment more accurate and in-depth;

[0077] It needs to be explained that the division of instructions and priority calculation based on time series data can enable the agent to arrange actions more reasonably. Accurate division of instructions can avoid confusion of actions, such as distinguishing between attack and defense instructions in complex game scenes. Calculating priorities ensures that important instructions are executed first. In emergency drill scenarios, emergency rescue instructions are executed first, and then other auxiliary tasks are processed, which improves the efficiency of actions and the ability to deal with complex situations, makes the behavior of the agent more in line with actual needs, and improves the performance of the entire interactive system;

[0078] In the actual evaluation, the intelligent agent perceives the doctor's gestures, the position and status of surgical instruments through vision, perceives the doctor's voice commands and the sounds of the surgical environment through hearing, and perceives the grip and operation feedback of the instruments through touch. According to the preset interaction rules, the threshold is set. When the doctor's hand-reaching action in the visual information lasts for more than 1 second, it is judged as interaction and the instruction to transfer the instrument is obtained. From these time series data, according to the interaction rules, different instructions are divided into 5 instructions to transfer the instrument and 2 instructions to adjust the brightness of the surgical light.

[0079] In the actual evaluation, the priority of each instruction is calculated, where the current time is 10 minutes, the total duration of the entire surgical task is expected to be 60 minutes, the associated time of the instrument transfer instruction is each time the doctor reaches out, the number of occurrences is 5, the number of operation steps involved (picking up and passing) is 2, and the number of parameters involved (instrument type) is 1. The priority of the instrument transfer instruction can be obtained according to the priority calculation formula. Similarly, the priority of other instructions such as the instruction to adjust the brightness of the surgical light can be calculated.

[0080] S300 generates a decision network according to a preset knowledge graph, generates an instruction chain according to the priority of the instruction, and the intelligent agent obtains a feedback strategy based on the decision network and the instruction chain;

[0081] In the actual evaluation, a decision network is constructed based on a decision tree, and the vector representations of entities (surgical instruments, doctors, surgical steps, etc.) and relationships (usage relationships, sequential relationships, etc.) in the preset knowledge graph are spliced ​​with the output vector of the perception model as the input of the decision network. The vector representation of the scalpel in the knowledge graph is [0.2, 0.3, 0.5] (indicating its function and usage scenario attributes), which is spliced ​​with the current scalpel visual feature vector [0.1, 0.2, 0.7] output by the perception model and then input into the decision network; the surgical rules in the preset knowledge graph (hemostasis before suturing) are used as the branching rules of the decision tree in the decision network, and the decision network is trained. By minimizing the loss function, the total number of leaf nodes in the decision tree is 10, the total number of node categories is 5, the number of samples belonging to a certain category falling on a certain leaf node is 2, the total number of samples falling on the leaf node is 5, and the regularization parameter is 0.1. The decision network is optimized according to the loss function calculation formula;

[0082] In the actual evaluation, the instructions are sorted according to their priority, with the instructions with higher priority being placed in front. The priority of the instruction to transfer the instrument is higher than the instruction to adjust the brightness of the surgical light, forming an instruction chain, and the intelligent agent executes the instruction to transfer the instrument first.

[0083] S400 allocates a dynamic weight to the feedback strategy, fuses the dynamic weight with the perception model through an attention mechanism, updates the virtual scene, and combines it with the feedback strategy to obtain an interaction mode;

[0084] It should be explained that the feedback strategy contains the decision information of the agent under different perceptual modalities. Dynamic weights are assigned to it because the importance of different feedback decisions changes dynamically throughout the interaction process. For example, in a virtual reality architectural design review scenario, when discussing the appearance of the building, the weight of visual-related feedback decisions may be higher; and when it comes to spatial sound experience, the weight of auditory-related feedback decisions will increase. This dynamic weight allocation can better adapt to changes in the interaction scenario;

[0085] It is important to explain that the dynamic weights are fused with the perception model through the attention mechanism. The attention mechanism allows the agent to focus on more important perceptual information in the current scene. Taking the virtual concert scene as an example, when the singer starts to sing solo, by assigning high weights to the auditory perception channel, the agent can focus more on the details of the sound, while combining the visual perception of the singer's performance posture, so that the fused information can better reflect the current focus of interaction and avoid interference from irrelevant information;

[0086] In the actual evaluation, the instruction chain is passed as input to the decision network, and the decision path with the shortest time to minimize the task is searched as the feedback strategy. There are three decision paths. The probabilities of different decision paths are calculated according to the probability distribution formula of the decision path, and the path with the highest probability is selected. The probability of each path is calculated, and the path 2 with the highest probability is selected as the feedback strategy; in the allocation of dynamic weights, for the feedback decision of the delivery device, the number of instructions involved in the visual feedback decision accounts for 0.6 of the total number of instructions, so the visual feedback decision is assigned a weight value of 0.6, and the feature representation of different perception channels is extracted from the perception model. The assigned weight value is weightedly fused with the features of each perception channel, and the attention weight of each channel is calculated. The visual channel feature is represented as [0.2, 0.3, 0.5]. After multiplying by the weight 0.6, the new visual channel feature is obtained, and then the feature maps of different perception channels are weightedly fused channel by channel according to the attention weight;

[0087] In the actual evaluation, the nodes of the scene graph include physical nodes (surgical instruments, operating tables), role nodes (doctors, agents) and environmental nodes (operating room lighting, sound environment), and the edges include spatial relationship edges (the position relationship of surgical instruments on the operating table), interactive relationship edges (the usage relationship between doctors and surgical instruments) and causal relationship edges (the causal relationship between surgical steps). The information in the fused feature graph is associated with the nodes and edges in the scene graph, and the attributes of the nodes and the relationships between the edges in the scene graph are updated. When the agent passes the scalpel, the position attributes of the scalpel in the scene graph and the interactive relationship with the doctor are updated, thereby updating the virtual scene.

[0088] S500 records the user's physical and mental parameters in real time, identifies abnormal fluctuation segments, learns the user's emotional tendencies based on the interactions corresponding to the abnormal fluctuation segments, and continuously optimizes the feedback strategy.

[0089] In the actual evaluation, during the surgical simulation, it is assumed that an abnormal fluctuation segment occurs, such as a doctor's operating error that prolongs the operation time. According to the interaction corresponding to the abnormal fluctuation segment, the state value function and the action value function are obtained, the interaction features in the abnormal fluctuation segment are analyzed, and the state variables of the interaction features related to the abnormality are marked, such as the state variables corresponding to the doctor's operating error. The penalty coefficient is set to reduce its value, the state value function and the action value function are updated, and the deep Q network is used to update the agent's policy network according to the updated value function, and the feedback strategy is optimized so that the agent can better assist the doctor in avoiding problems in subsequent similar situations.

[0090] In this embodiment, the method of creating a virtual scene and an intelligent agent based on interaction requirements and constructing a perception model for the intelligent agent includes:

[0091] Create object and scene models based on the elements, functions and characteristics required for interaction to form a virtual scene;

[0092] Based on the purpose of interaction requirements, the role and function of the intelligent agent are determined, and the internal architecture is constructed, including: knowledge storage and decision-making modules, where the state transfer function of the intelligent agent is expressed as:

[0093]

[0094] Among them, s is the current state of the agent, e is the input event, z is the decision state taken by the agent, f is the environmental factor, t is the time variable, s* is the next state that the agent transfers to in a given state, P(s * |s, e, z, f, t) is the transition to state s in a given state * The probability of, μ(s, e, z, f, t) is the expected value of the next state based on a given state, and σ is the time step;

[0095] A multi-sensory self-attention model is introduced to add multimodal fusion function to the intelligent agent and obtain a perception model. Specifically, for visual data, the image frame is downsampled to a preset size and the area is randomly cropped during training; for audio data, the signal is resampled at a preset frequency and the short-time Fourier transform is calculated to generate a mel spectrum; for tactile data, a tactile image is generated based on optical tracking and deformation detection;

[0096] A multi-head attention mechanism is implemented in the perception model to process data of different modalities. The expression of the multi-head attention mechanism is:

[0097]

[0098] Among them, Q, K and V are query matrix, key matrix and value matrix respectively, W 0 is the linear transformation matrix of the output layer, Concat represents the concatenation operation, n is the number of attention heads, and W i is the weight matrix of the ith attention head, T represents the transpose operation, d i is the size of the i-th attention head dimension, A i is the bias term of the i-th attention head.

[0099] In this embodiment, the method of dividing instructions based on the time series data and calculating instruction priorities includes:

[0100] The time series data is divided into different instructions according to the preset interaction rules, which specifically includes: obtaining a threshold from the preset interaction rules, and if the time series data exceeds the threshold, it is judged as interaction and an instruction is obtained;

[0101] Duplicate instructions and combined instructions are identified based on the similarity metric and information metric of the instructions, where the calculation formulas of the similarity metric and the information metric are as follows:

[0102]

[0103]

[0104] Where W(P, Q) is the Wasserstein distance between the probability distributions P and Q of two instructions, inf is the infimum, which means finding the value with the smallest expected value among all possible γ, γ is the joint distribution, Π(P, Q) is the set of transitions from the probability distribution of instruction P to instruction Q, It is expressed as the expected distance between random variables X and Y under the joint distribution γ, ||XY|| is the Euclidean distance between X and Y, I(P, Q) is the mutual information between the probability distributions P and Q of the two instructions, p(x, y) is the joint probability distribution of X and Y, p(x) and p(y) are the marginal probability distributions of X and Y respectively;

[0105] After normalizing the similarity metric and information metric from 0 to 1, the instructions with similarity metric greater than 0.7 and information metric greater than 0.8 are combined instructions, and the instructions with similarity metric greater than 0.7 and information metric less than or equal to 0.8 are repeated instructions. For repeated instructions, the instruction with the largest joint probability distribution is selected as the final instruction;

[0106] Calculate the priority of each instruction, the expression is:

[0107]

[0108] Among them, P i is the priority of the ith instruction, t c is the current time, t i is the association time of the ith instruction, t t is the total duration of the entire task, η i is the number of times the i-th instruction appears, η t is the total number of instructions, n i，s is the number of operation steps involved in executing the i-th instruction, n i，p is the number of parameters involved in the ith instruction, t i，r The estimated response time required to complete the i-th instruction, d i is the degree of dependence of the ith instruction on other instructions, μ i is the urgency of the ith instruction.

[0109] In this embodiment, the method of generating a decision network according to a preset knowledge graph and generating an instruction chain according to the priority of the instructions includes:

[0110] Build a decision network based on the decision tree, and concatenate the vector representations of entities and relationships in the preset knowledge graph with the output vector of the perception model as the input of the decision network;

[0111] The rules in the preset knowledge graph are used as the branch rules of the decision tree in the decision network. The decision network is trained to learn the rules in the knowledge graph by minimizing the loss function. The expression of the loss function is:

[0112]

[0113] Among them, |T| is the total number of leaf nodes in the decision tree T, K is the total number of categories of nodes in the decision tree, and N tk is the number of samples belonging to category k that fall on leaf node t, N t is the total number of samples falling on leaf node t, α is the regularization parameter;

[0114] Instructions are sorted according to their priority, with instructions with higher priority placed in front, forming an instruction chain.

[0115] In this embodiment, the method for the agent to obtain a feedback strategy based on a decision network and a command chain includes:

[0116] The instruction chain is passed as input to the decision network, including the feature vector and priority of the instruction. The decision path with the shortest time required to complete the task is searched in the decision network as a feedback strategy. The probability distribution expression of different decision paths is:

[0117]

[0118] Where δ is the number of decision paths, x i is the feature vector of the i-th decision path, m is x i , ∑ is the covariance matrix of the decision path distribution, μ is the mean vector of the decision path distribution, and T is the transpose operation.

[0119] In this embodiment, the method of assigning dynamic weights to the feedback strategy and fusing the dynamic weights with the perception model through an attention mechanism includes:

[0120] Obtain visual, auditory, and tactile feedback decisions from the feedback strategy, and assign a weight value of the corresponding number of instructions in the proportion of all instructions to each feedback decision;

[0121] The feature representations of different perceptual channels are extracted from the perceptual model, the assigned weight values ​​are weightedly fused with the features of each perceptual channel, the attention weight of each channel is calculated, and according to the attention weights, the feature maps of different perceptual channels are weightedly fused by channel-by-channel multiplication.

[0122] In this embodiment, the method for updating the virtual scene includes:

[0123] Constructing a scene graph of a virtual scene, wherein nodes include physical nodes, role nodes and environment nodes, and edges include spatial relationship edges, interaction relationship edges and causal relationship edges;

[0124] The information in the fused feature graph is associated with the nodes and edges in the scene graph, the attributes of the nodes and the relationship between the edges in the scene graph are updated, and then the virtual scene is updated.

[0125] In this embodiment, the method for interactively learning the user's emotional tendency based on the abnormal fluctuation segment includes:

[0126] According to the interaction corresponding to the abnormal fluctuation segment, the state value function and action value function of the abnormal fluctuation segment are obtained:

[0127] The expression of the state value function is:

[0128]

[0129] Where h is the number of characteristic functions, ω i is the weight of the i-th feature function, β i is the linear coefficient of the i-th characteristic function, α i is the nonlinear coefficient of the i-th characteristic function, s is the current state of the agent, and f i (s) is the output of the i-th feature function extracted from state s, Φ is the state space, p(θ|S) is the probability of transitioning from state s to state θ, V(θ) is the value of the next state θ, ∈ is the adjustment term, and ζ is the threshold parameter used to control the center point of the tanh function;

[0130] The expression of the action value function is:

[0131]

[0132] Where s is the current state of the agent, r(s, a) is the immediate reward for performing action a in state s, γ is the discount factor, Φ is the state space, P(θ|s, a) is the probability of transferring to the next state θ after taking action a in state s, V(θ) is the value of the next state θ, m is the number of dimensions of the action, and v j is the weight of the jth dimension, δ jis the linear coefficient of the jth dimension, g j (θ,a) is the additional value calculated based on state θ, action a and the jth dimension, ξ j is the attenuation coefficient of the jth dimension, λ j is the Gaussian decay rate of the j-th dimension.

[0133] In this implementation example, wearable devices and sensor technologies are used to monitor the user's physiological parameters in real time, such as heart rate, skin conductance, body temperature, etc. These devices are integrated into daily necessities such as glasses, watches, bracelets, etc.; machine learning algorithms, especially deep learning models, are used to analyze these continuous physiological parameter data to identify abnormal fluctuations. For example, a model is trained to identify sudden changes in heart rate or skin conductance, which may be related to emotional fluctuations, and the user's emotional tendencies are learned by analyzing the user's interaction data (such as voice, text, behavior, etc.) during abnormal fluctuations. Sentiment analysis technology, especially deep learning-based models such as BERT, is used to extract emotional information from text data.

[0134] In this embodiment, the method for continuously optimizing the feedback strategy includes: analyzing the interaction features in the abnormal fluctuation segment, marking the state variables of the interaction features related to the anomaly, setting a penalty coefficient to reduce the value of the state variables, updating the state value function and the action value function, using a deep Q network to update the agent's policy network according to the updated value function, and optimizing the feedback strategy.

[0135] A second aspect of the present invention provides a multi-sensory interaction system based on virtual reality, comprising:

[0136] A virtual scene and intelligent agent creation module, used to create virtual scenes and intelligent agents based on interaction requirements and build a perception model for the intelligent agents;

[0137] An instruction processing module is used to generate a decision network according to a preset knowledge graph, generate an instruction chain according to the priority of the instruction, and the intelligent agent obtains a feedback strategy based on the decision network and the instruction chain;

[0138] A decision generation module, used to generate a decision network according to a preset knowledge graph, generate an instruction chain according to the priority of the instruction, and the intelligent agent obtains a feedback strategy based on the instruction chain and the decision network;

[0139] A feedback and scene update module, used to assign dynamic weights to the feedback strategy, fuse the dynamic weights with the perception model through an attention mechanism, update the virtual scene, and combine it with the feedback strategy to obtain an interaction mode;

[0140] The user emotion optimization module is used to record the user's physical and mental parameters in real time, identify abnormal fluctuation segments, learn the user's emotional tendencies based on the interactions corresponding to the abnormal fluctuation segments, and continuously optimize the feedback strategy.

[0141] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A multi-sensory interaction method based on virtual reality, characterized in that: The following steps are involved: Creating virtual scenes and agents based on interaction requirements, and building a perception model for the agents; The intelligent agent performs visual, auditory and tactile environmental perception, extracts time series data of perception features, divides instructions based on the time series data, and calculates instruction priorities; A decision network is generated according to a preset knowledge graph, and an instruction chain is generated according to the priority of the instructions. The intelligent agent obtains a feedback strategy based on the decision network and the instruction chain; Assigning dynamic weights to the feedback strategy, fusing the dynamic weights with the perception model through an attention mechanism, updating the virtual scene, and combining with the feedback strategy to obtain an interaction mode; The user's physical and mental parameters are recorded in real time, abnormal fluctuation segments are identified, and the user's emotional tendencies are learned based on the interactions corresponding to the abnormal fluctuation segments, so as to continuously optimize the feedback strategy.

2. The multi-sensory interaction method based on virtual reality according to claim 1, characterized in that: The method of creating a virtual scene and an intelligent agent based on interaction requirements and constructing a perception model for the intelligent agent comprises: Create object and scene models based on the elements, functions and characteristics required for interaction to form a virtual scene; Based on the purpose of interaction requirements, the role and function of the intelligent agent are determined, and the internal architecture is constructed, including: knowledge storage and decision-making modules, where the state transfer function of the intelligent agent is expressed as: Among them, s is the current state of the agent, e is the input event, z is the decision state taken by the agent, f is the environmental factor, t is the time variable, and s * is the next state that the agent transfers to in a given state, P(s * |s,e,z,f,t) is to transfer to state s in a given state * The probability of, μ(s,e,z,f,t) is the expected value of the next state based on a given state, and σ is the time step; A multi-sensory self-attention model is introduced to add multimodal fusion function to the intelligent agent and obtain a perception model. Specifically, for visual data, the image frame is downsampled to a preset size and the area is randomly cropped during training; for audio data, the signal is resampled at a preset frequency and the short-time Fourier transform is calculated to generate a mel spectrum; for tactile data, a tactile image is generated based on optical tracking and deformation detection; A multi-head attention mechanism is implemented in the perception model to process data of different modalities. The expression of the multi-head attention mechanism is: Among them, Q, K and V are query matrix, key matrix and value matrix respectively, W 0 is the linear transformation matrix of the output layer, Concat is the concatenation operation, n is the number of attention heads, and W i is the weight matrix of the i-th attention head, T is the transpose operation, d i is the size of the i-th attention head dimension, A i is the bias term of the i-th attention head.

3. The multi-sensory interaction method based on virtual reality according to claim 1, characterized in that: The method for dividing instructions based on the time series data and calculating instruction priorities includes: The time series data is divided into different instructions according to the preset interaction rules, which specifically includes: obtaining a threshold from the preset interaction rules, and if the time series data exceeds the threshold, it is judged as interaction and an instruction is obtained; Duplicate instructions and combined instructions are identified based on the similarity metric and information metric of the instructions, where the calculation formulas of the similarity metric and the information metric are as follows: Where W(P,Q) is the Wasserstein distance between the probability distributions P and Q of two instructions, inf is the infimum, which means finding the value with the smallest expected value among all possible γ, γ is the joint distribution, Π(P,Q) is the set of transitions from the probability distribution of instruction P to instruction Q, It is expressed as the expected distance between random variables X and Y under the joint distribution γ, ||XY|| is the Euclidean distance between X and Y, I(P,Q) is the mutual information between the probability distributions P and Q of the two instructions, p(x,y) is the joint probability distribution of X and Y, p(x) and p(y) are the marginal probability distributions of X and Y respectively; After normalizing the similarity metric and information metric from 0 to 1, the instructions with similarity metric greater than 0.7 and information metric greater than 0.8 are combined instructions, and the instructions with similarity metric greater than 0.7 and information metric less than or equal to 0.8 are repeated instructions. For repeated instructions, the instruction with the largest joint probability distribution is selected as the final instruction; Calculate the priority of each instruction, the expression is: Among them, P i is the priority of the ith instruction, t c is the current time, t i is the association time of the ith instruction, t t is the total duration of the entire task, η i is the number of times the i-th instruction appears, η t is the total number of instructions, n i,s is the number of operation steps involved in executing the i-th instruction, n i,p is the number of parameters involved in the ith instruction, t i,r The estimated response time required to complete the i-th instruction, d i is the degree of dependence of the ith instruction on other instructions, μ i is the urgency of the ith instruction.

4. The multi-sensory interaction method based on virtual reality according to claim 1, characterized in that: The method of generating a decision network according to a preset knowledge graph and generating an instruction chain according to the priority of the instructions includes: Build a decision network based on the decision tree, and concatenate the vector representations of entities and relationships in the preset knowledge graph with the output vector of the perception model as the input of the decision network; The rules in the preset knowledge graph are used as the branch rules of the decision tree in the decision network. The decision network is trained to learn the rules in the knowledge graph by minimizing the loss function. The expression of the loss function is: Among them, |T| is the total number of leaf nodes in the decision tree T, K is the total number of categories of nodes in the decision tree, and N tk is the number of samples belonging to category k that fall on leaf node t, N t is the total number of samples falling on leaf node t, α is the regularization parameter; Instructions are sorted according to their priority, with instructions with higher priority placed in front, forming an instruction chain.

5. The multi-sensory interaction method based on virtual reality according to claim 1, characterized in that: The method for the intelligent agent to obtain a feedback strategy based on a decision network and a command chain includes: The instruction chain is passed as input to the decision network, including the feature vector and priority of the instruction. The decision path with the shortest time required to complete the task is searched in the decision network as a feedback strategy. The probability distribution expression of different decision paths is: Where δ is the number of decision paths, x i is the feature vector of the i-th decision path, m is x i , ∑ is the covariance matrix of the decision path distribution, μ is the mean vector of the decision path distribution, and T is the transpose operation.

6. The multi-sensory interaction method based on virtual reality according to claim 1, characterized in that: The method of assigning dynamic weights to the feedback strategy and fusing the dynamic weights with the perception model through an attention mechanism includes: Obtain visual, auditory, and tactile feedback decisions from the feedback strategy, and assign a weight value of the corresponding number of instructions in the proportion of all instructions to each feedback decision; The feature representations of different perceptual channels are extracted from the perceptual model, the assigned weight values ​​are weightedly fused with the features of each perceptual channel, the attention weight of each channel is calculated, and according to the attention weights, the feature maps of different perceptual channels are weightedly fused by channel-by-channel multiplication.

7. The multi-sensory interaction method based on virtual reality according to claim 1, characterized in that: The method for updating the virtual scene comprises: Constructing a scene graph of a virtual scene, wherein nodes include physical nodes, role nodes and environment nodes, and edges include spatial relationship edges, interaction relationship edges and causal relationship edges; The information in the fused feature graph is associated with the nodes and edges in the scene graph, the attributes of the nodes and the relationship between the edges in the scene graph are updated, and then the virtual scene is updated.

8. The multi-sensory interaction method based on virtual reality according to claim 1, characterized in that: The method for interactively learning the user's emotional tendency based on the abnormal fluctuation segment corresponding to the abnormal fluctuation segment includes: According to the interaction corresponding to the abnormal fluctuation segment, the state value function and action value function of the abnormal fluctuation segment are obtained: The expression of the state value function is: Where h is the number of characteristic functions, ω i is the weight of the i-th feature function, β i is the linear coefficient of the i-th characteristic function, α i is the nonlinear coefficient of the i-th characteristic function, s is the current state of the agent, and f i (s) is the output of the i-th feature function extracted from state s, Φ is the state space, p(θ|S) is the probability of transitioning from state s to state θ, V(θ) is the value of the next state θ, ∈ is the adjustment term, and ζ is the threshold parameter used to control the center point of the tanh function; The expression of the action value function is: Where s is the current state of the agent, r(s,a) is the immediate reward for performing action a in state s, γ is the discount factor, Φ is the state space, P(θ|s,a) is the probability of transferring to the next state θ after taking action a in state s, V(θ) is the value of the next state θ, m is the number of dimensions of the action, and v j is the weight of the jth dimension, δ j is the linear coefficient of the jth dimension, g j (θ,a) is the additional value calculated based on state θ, action a and the jth dimension, ξ j is the attenuation coefficient of the jth dimension, λ j is the Gaussian decay rate of the j-th dimension.

9. The multi-sensory interaction method based on virtual reality according to claim 1, characterized in that: The method for continuously optimizing the feedback strategy includes: analyzing the interaction features in the abnormal fluctuation segment, marking the state variables of the interaction features related to the abnormality, setting a penalty coefficient to reduce the value of the state variable, updating the state value function and the action value function, using a deep Q network to update the agent's strategy network according to the updated value function, and optimizing the feedback strategy.

10. A multi-sensory interaction system based on virtual reality, used to execute a multi-sensory interaction method based on virtual reality according to any one of claims 1 to 9, characterized in that: The system comprises: A virtual scene and intelligent agent creation module, used to create virtual scenes and intelligent agents based on interaction requirements and build a perception model for the intelligent agents; An instruction processing module is used to generate a decision network according to a preset knowledge graph, generate an instruction chain according to the priority of the instruction, and the intelligent agent obtains a feedback strategy based on the decision network and the instruction chain; A decision generation module, used to generate a decision network according to a preset knowledge graph, generate an instruction chain according to the priority of the instruction, and the intelligent agent obtains a feedback strategy based on the instruction chain and the decision network; A feedback and scene update module, used to assign dynamic weights to the feedback strategy, fuse the dynamic weights with the perception model through an attention mechanism, update the virtual scene, and combine it with the feedback strategy to obtain an interaction mode; The user emotion optimization module is used to record the user's physical and mental parameters in real time, identify abnormal fluctuation segments, learn the user's emotional tendencies based on the interactions corresponding to the abnormal fluctuation segments, and continuously optimize the feedback strategy.

Images