User experience interaction method based on VR engine

Abstract

The invention discloses a user experience interaction method based on a VR (virtual reality) engine, and relates to the technical field of interaction experiences, features of voice, gestures, touch and sight signals are extracted through a deep learning model, signal fusion is performed in combination with a cross-modal attention mechanism, user behavior understanding ability is enhanced, and meanwhile, user experience is improved. A synchronous localization and mapping (SLAM) technology is utilized to capture the position and operation context of a user in a real environment in real time, so that interaction is more natural and smoother; secondly, an AR engine is combined with light and shadow consistency calculation, a physical simulation technology and dynamic environment adaptation, deep fusion of a virtual object and a real environment is achieved, the problem of virtual-real separation in a traditional method is solved, and the immersion of interaction is greatly improved; besides, user intentions including interaction types, target objects and task targets are predicted through dynamic Bayesian network modeling, a prediction model is optimized in combination with real-time feedback and a self-learning mechanism, and personalized requirements of users are gradually met.

Description

Technical Field

[0001] The present invention relates to the field of interactive experience technology, and in particular to a user experience interaction method based on a VR engine. Background Art

[0002] The existing interactive experience is mainly based on a graphical user interface. Users complete operations through clicking, sliding, voice input, etc. This interactive method is simple and direct. However, with the complexity of user needs, in situations involving multimodality and virtual scenes, traditional interactive methods cannot meet user experience needs. Mobile devices can be controlled through voice commands, but desktop devices still rely on mouse or keyboard operations. The interactive experience is too fragmented.

[0003] To address these problems, some interaction solutions have introduced voice assistants to process natural language commands and adopted gesture recognition technology to enhance the naturalness of operations, which have optimized the user experience to a certain extent and made the interaction between people and devices more natural. In addition, some solutions have introduced augmented reality (AR) and virtual reality (VR) to optimize the effect of virtual-reality interaction through virtual object rendering and ambient light adaptation, thus bringing users a more immersive experience. However, such improvement solutions have still failed to completely solve the problem of insufficient interaction fusion. Users still need to frequently switch between different input modes, and have failed to achieve truly seamless cross-modal interaction. In addition, the current interaction solutions still lack adaptability and dynamic optimization capabilities, and are unable to dynamically adjust the interaction method and display content according to the user's real-time behavior. Therefore, a higher-level fusion interaction and dynamic adaptation user experience interaction method based on a VR engine is urgently needed to solve such problems. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] The present invention provides a user experience interaction method based on a VR engine to solve the problem of insufficient interaction fusion. Users still need to switch between different input modes, rather than truly achieving seamless cross-mode interaction.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] The embodiment of the present invention provides a user experience interaction method based on a VR engine, which includes:

[0008] Step S1, deploying a signal interface in the AR engine to capture user input signals, including voice commands, gestures, touch operations, and line of sight changes, and using a deep learning model to extract features from the input signals to generate signal features;

[0009] Step S2, integrating signal features, using a probabilistic graph model to generate a user intention model, and predicting the user's operation purpose. The prediction content includes: interaction type, target object, and task goal;

[0010] Step S3, based on the prediction result of the user intention model, the AR engine drives the virtual scene to make corresponding adjustments, including positioning, scaling or rotation of virtual objects, and adaptive optimization of displayed content;

[0011] Step S4, the adjusted virtual scene is deeply integrated with the real environment through the light and shadow consistency calculation and physical property simulation technology of the AR engine, and dynamically adapts to the changes in ambient light to obtain the final interactive effect;

[0012] In step S5, the final interaction effect is output and fed back to the user, and the user's further operations are recaptured and updated to a new round of step S1 input signals using a feedback mechanism.

[0013] As a preferred solution of the user experience interaction method based on VR engine described in the present invention, the AR engine provides a synchronous multimodal data acquisition framework, captures user input signals in real time, and locates the user's position in the real environment through synchronous positioning and mapping SLAM technology.

[0014] As a preferred solution of the user experience interaction method based on VR engine described in the present invention, the step of extracting features of input signals using a deep learning model to generate signal features is as follows:

[0015] The Transformer model is used to encode the speech waveform and extract semantic information. The encoding formula is:

[0016] z speech =f speech (x speech θ speech ),

[0017] Among them, z speech is the extracted semantic feature vector, x speech is the input speech waveform signal, θ speech is the training parameter of the speech model, f speech is a deep learning function for speech feature extraction;

[0018] Capture the user's continuous gesture action frames, combine the 3D convolutional network CNN and the long short-term memory network LSTM for modeling, and the capture formula is:

[0019] z gesture =f gesture (x gesture θ gesture ),

[0020] Among them, z gesture is the high-dimensional feature representation of the gesture signal, x gesture is the input gesture video sequence, θ gesture is the training parameter of the gesture model, f gesture Extract model function for gesture features;

[0021] Extract touch signal features, including click position and sliding trajectory, and encode them using multi-layer perceptron MLP. The encoding formula is:

[0022] z touch =f touch (x touch θ touch ),

[0023] Among them, z touch is the characteristic vector of the touch signal, x touch is the touch input data, including the click position and sliding trajectory, θ touch is the touch model parameter, f touch Extract function for touch feature;

[0024] Extract the sight signal features, use ResNet to process the sight trajectory data, and extract the features of the user's gaze area. The extraction formula is:

[0025] z gaze =f gaze (x gaze θ gaze ),

[0026] Among them, z gaze is the high-dimensional feature of the line of sight signal, x gaze Input data for the sight trajectory, θ gaze is the training parameter of the line of sight model, f gaze Extract model function for sight feature;

[0027] The feature fusion formula is:

[0028] z fused =f fusion (z speech ,z gesture ,z touch ,z gaze θ fusion ),

[0029] Among them, z fused is the fused multimodal feature representation, f fusion is the multimodal fusion function, θ fusion is the multimodal fusion parameter.

[0030] As a preferred solution of the user experience interaction method based on VR engine described in the present invention, the method of fusing signal features also includes using AR engine to analyze the depth information of the real scene and the three-dimensional structure of the virtual scene to obtain environmental features, and fusing the environmental features with the signal features input by the user; the depth information includes the distance of the object and the surface characteristics of the object.

[0031] As a preferred solution of the user experience interaction method based on VR engine described in the present invention, the steps of fusing signal features, using a probabilistic graph model to generate a user intention model, and predicting the user's operation purpose are as follows:

[0032] Taking the multimodal signal feature z in step S1 fused As input features, the AR engine is used to extract environmental features e, including the depth information e of the real scene depth and the three-dimensional structure of the virtual scene 3D ,The depth information includes the distance and surface characteristics of the object, and the ,three-dimensional structure is the shape and spatial distribution of the object;

[0033] Perform feature fusion, the fusion formula is:

[0034] z combined =f attention (z fused ,e;θ fusion ),

[0035] Among them, z combined is the fused feature, f attention is the cross-modal attention mechanism, θ fusion is the parameter of feature fusion;

[0036] The user intention model is modeled through the dynamic Bayesian network DBN, and the posterior probability of the intention is calculated. The calculation formula is:

[0037]

[0038] Among them, P(I|z combined ) is the given fusion feature z combined Under this condition, the posterior probability of user intention I, P(z combined |I) is the conditional probability of the given intention, P(I) is the prior probability of the intention, ∑ i (·) indicates all possible intentions I i Normalize.

[0039] As a preferred solution of the user experience interaction method based on VR engine described in the present invention, the step of driving the virtual scene to make corresponding adjustments based on the prediction result of the user intention model by the AR engine is as follows:

[0040] Adjust the positioning of virtual objects, dynamically update the three-dimensional spatial position of virtual objects based on the target objects and operation area predictions in the user's intention,

[0041] The update formula is:

[0042] p new =p old +Δp,

[0043] Among them, p new is the new position vector of the virtual object, p old is the original position vector of the virtual object, Δp is the position change vector;

[0044] Perform scaling adjustments on virtual objects and adjust the proportions of virtual objects based on task target predictions.

[0045] The scaling formula is:

[0046] S new =S old k,

[0047] Among them, S new is the updated scaling ratio, S old is the original scaling ratio, k is the scaling factor;

[0048] Perform rotation adjustment of the virtual object, and adjust the direction of the virtual object through the user's rotation operation.

[0049] The rotation formula is:

[0050] R new =R old R(θ,u),

[0051] Among them, R new is the updated rotation matrix, R old is the initial rotation matrix, R(θ,u) is the rotation matrix, which consists of the rotation axis u and the rotation angle θ.

[0052] The display content layout in the virtual scene is optimized according to the user intention model. The optimization formula is:

[0053]

[0054] Among them, L opt is the optimized display layout, L is the candidate display layout, and w i is the weight in the user intention model, n is the total number of contents, D i (L) is the distance between the user's focus target and the displayed content.

[0055] As a preferred solution of the user experience interaction method based on VR engine described in the present invention, the step of making corresponding adjustments also includes:

[0056] Record and self-learn the user's operating habits, dynamically adjust the initial state of the virtual scene, and the self-learning formula is: initial =E[p user ],

[0057] Among them, is the initial position, p user is the position distribution in the user's operation history, E is the expected value calculation, and the adjusted virtual scene is transmitted to the user through visual and tactile feedback.

[0058] As a preferred solution of the user experience interaction method based on VR engine described in the present invention, the step of deeply integrating the adjusted virtual scene with the real environment through the light and shadow consistency calculation and physical property simulation technology of AR engine is as follows:

[0059] The AR engine captures the lighting conditions in the real environment in real time, including the direction, intensity and color temperature of the light source, and renders shadows and highlights through a global illumination model, so that the virtual objects are consistent with the light and shadow changes in the real environment;

[0060] The AR engine simulates the physical interaction characteristics between virtual objects and the real environment;

[0061] The AR engine dynamically adjusts the presentation of virtual objects based on the environmental data collected in real time.

[0062] As a preferred solution of the user experience interaction method based on VR engine described in the present invention, the feedback mechanism includes:

[0063] Adjustment of signal capture and feature extraction parameters,

[0064] and adjusting the user intent model parameters based on user repetitive behaviors.

[0065] As a preferred solution of the user experience interaction method based on VR engine described in the present invention, wherein: the step of updating the input signal of step S1 to a new round by using feedback mechanism is:

[0066] Capture voice and gesture signals and adjust the corresponding parameters. The adjustment formula is:

[0067] θ speech,new =θ speech,old +Δθ speech ,

[0068] θ gesture,new =θ gesture,old +Δθ gesture ,

[0069] Among them, θ speech,new and θ gesture,new are the updated speech and gesture capture parameters, θ speech,old and θ gesture,old are the original speech and gesture capture parameters, Δθ speech and Δθ gesture is the incremental parameter adjusted by user feedback,

[0070] The optimization formula for the feature extraction model is:

[0071] z′=z+Δz feedback ,

[0072] Among them, z' is the optimized eigenvector, z is the original eigenvector, Δz feedback Optimization increments introduced for feedback mechanisms;

[0073] Update the prior probability and conditional probability in the user intention model based on the user's repeated behavior and historical operation patterns.

[0074] To model historical behavior, the model formula is:

[0075] P(I) new =αP history (I)+(1-α)P prior (I)

[0076] Among them, P(I) new is the updated prior probability of user intention, α is the historical weight factor, P history (I) represents the prior probability of user intention I based on the user's historical behavior statistics, P prior (I) represents the initial prior probability of the preset user intention I,

[0077] Update the conditional probability, the update formula is:

[0078] P(z′|I) new =P(z′|I)+ΔP feedback (z′|I),

[0079] in,

[0080] P(z'|I) new is the conditional probability after optimization, P(Z'}I) is the conditional probability before optimization, ΔP feedback (z'}I) is the conditional probability increment based on user operation feedback,

[0081] The feedback mechanism not only adjusts the model parameters, but also the adjustment formula is:

[0082] θ′=θ+Δθfeedback ,

[0083] Among them, θ' is the global parameter set after optimization, θ is the global parameter set before optimization, Δθ feedback Parameter increments calculated for the feedback mechanism.

[0084] The beneficial effects of the present invention are as follows: the present invention uses a deep learning model to extract features of voice, gesture, touch and sight signals, and combines a cross-modal attention mechanism to perform signal fusion, thereby enhancing the ability to understand user behavior. At the same time, the synchronous positioning and mapping SLAM technology is used to capture the user's position and operation context in the real environment in real time, making the interaction more natural and smooth; secondly, the AR engine is used in combination with light and shadow consistency calculation, physical simulation technology and dynamic environment adaptation to achieve deep fusion of virtual objects and the real environment, eliminate the problem of virtual-real separation in traditional methods, and greatly enhance the immersiveness of the interaction; in addition, the user's intention is predicted through dynamic Bayesian network modeling, including interaction type, target object and task goal, and the prediction model is optimized in combination with real-time feedback and self-learning mechanism to gradually adapt to the user's personalized needs. In terms of virtual scene adjustment, the AR engine dynamically adjusts the position, scaling and rotation angle of the virtual object according to the user's intention, while optimizing the display content layout, and combines the self-learning mechanism to record the user's operation habits, further reducing redundant operations and improving interaction efficiency;

[0085] The present invention dynamically optimizes signal capture parameters, feature extraction models and intent prediction models through a feedback mechanism, constructs an adaptive closed-loop optimization process, and enhances the adaptability to dynamic changes and the support capability for complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0087] Figure 1 It is a flowchart of the user experience interaction method based on VR engine of the present invention. DETAILED DESCRIPTION

[0088] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0089] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0090] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0091] Example 1, reference Figure 1 This embodiment provides a user experience interaction method based on a VR engine, including:

[0092] Step S1, deploying a signal interface in the AR engine to capture user input signals, including voice commands, gestures, touch operations, and line of sight changes, and using a deep learning model to extract features from the input signals to generate signal features;

[0093] The AR engine provides a synchronous multimodal data collection framework, captures user input signals in real time, and locates the user's position in the real environment through synchronous positioning and mapping SLAM technology;

[0094] The deep learning model is used to extract the features of the input signal. The steps to generate signal features are:

[0095] The Transformer model is used to encode the speech waveform and extract semantic information. The encoding formula is:

[0096] z speech =f speech (x speech θ speech ),

[0097] Among them, z speech is the extracted semantic feature vector, x speech is the input speech waveform signal, θ speech is the training parameter of the speech model, f speech is a deep learning function for speech feature extraction;

[0098] Capture the user's continuous gesture action frames, combine the 3D convolutional network CNN and the long short-term memory network LSTM for modeling, and the capture formula is:

[0099] z gesture =f gesture (x gesture θgesture ),

[0100] Among them, z gesture is the high-dimensional feature representation of the gesture signal, x gesture is the input gesture video sequence, θ gesture is the training parameter of the gesture model, f gesture Extract model function for gesture features;

[0101] Extract touch signal features, including click position and sliding trajectory, and encode them using multi-layer perceptron MLP. The encoding formula is:

[0102] z touch =f touch (x touch θ touch ),

[0103] Among them, z touch is the characteristic vector of the touch signal, x touch is the touch input data, including the click position and sliding trajectory, θ touch is the touch model parameter, f touch Extract function for touch feature;

[0104] Extract the sight signal features, use ResNet to process the sight trajectory data, and extract the features of the user's gaze area. The extraction formula is:

[0105] z gaze =f gaze (x gaze θ gaze ),

[0106] Among them, z gaze is the high-dimensional feature of the line of sight signal, x gaze Input data for the sight trajectory, θ gaze is the training parameter of the line of sight model, f gaze Extract model function for sight feature;

[0107] The feature fusion formula is:

[0108] z fused =f fusion (z speech ,z gesture ,z touch ,z gaze θ fusion ),

[0109] Among them, z fused is the fused multimodal feature representation, f fusion is the multimodal fusion function, θ fusion is the multimodal fusion parameter;

[0110] Specifically, the features of speech, gesture, touch and line of sight are extracted through a deep learning model, and the multimodal features are fused using a cross-modal attention mechanism to generate a unified high-dimensional feature representation, ensuring the integrity and consistency of the signal features.

[0111] Step S2, integrating signal features, using a probabilistic graph model to generate a user intention model, and predicting the user's operation purpose. The prediction content includes: interaction type, target object, and task goal;

[0112] The method of fusing signal features also includes using an AR engine to analyze the depth information of the real scene and the three-dimensional structure of the virtual scene to obtain environmental features, and fusing the environmental features with the signal features input by the user; the depth information includes the distance of the object and the surface characteristics of the object;

[0113] Fusion signal features, using the probabilistic graph model to generate a user intent model, the steps for predicting the user's operation purpose are as follows:

[0114] Taking the multimodal signal feature z in step S1 fused As input features, the AR engine is used to extract environmental features e, including the depth information e of the real scene depth and the three-dimensional structure of the virtual scene 3D ,The depth information includes the distance and surface characteristics of the object, and the ,three-dimensional structure is the shape and spatial distribution of the object;

[0115] Perform feature fusion, the fusion formula is:

[0116] z combined =f attention (z fused ,e;θ fusion ),

[0117] Among them, z combined is the fused feature, f attention is the cross-modal attention mechanism, θ fusion is the parameter of feature fusion;

[0118] The user intention model is modeled through the dynamic Bayesian network DBN, and the posterior probability of the intention is calculated. The calculation formula is:

[0119]

[0120] Among them, P(I|z combined ) is the given fusion feature z combined Under this condition, the posterior probability of user intention I, P(z combined |I) is the conditional probability of the given intent, P(I) is the prior probability of intent, Σ i(·) indicates all possible intentions I i Perform normalization;

[0121] Specifically, by fusing multimodal signal features and environmental features and using dynamic Bayesian network modeling, the user's interaction type, target object, and task goal can be predicted in real time.

[0122] Step S3, based on the prediction result of the user intention model, the AR engine drives the virtual scene to make corresponding adjustments, including positioning, scaling or rotation of virtual objects, and adaptive optimization of displayed content;

[0123] According to the prediction results of the user intention model, the steps of driving the virtual scene to make corresponding adjustments by the AR engine are as follows:

[0124] Adjust the positioning of virtual objects, dynamically update the three-dimensional spatial position of virtual objects based on the target objects and operation area predictions in the user's intention,

[0125] The update formula is:

[0126] p new =p old +Δp,

[0127] Among them, p new is the new position vector of the virtual object, p old is the original position vector of the virtual object, Δp is the position change vector;

[0128] Perform scaling adjustments on virtual objects and adjust the proportions of virtual objects based on task target predictions.

[0129] The scaling formula is:

[0130] S new =S old k,

[0131] Among them, S new is the updated scaling ratio, S old is the original scaling ratio, k is the scaling factor;

[0132] Perform rotation adjustment of the virtual object, and adjust the direction of the virtual object through the user's rotation operation.

[0133] The rotation formula is:

[0134] R new =R old R(θ,u),

[0135] Among them, R new is the updated rotation matrix, R oldis the initial rotation matrix, R(θ,u) is the rotation matrix, which consists of the rotation axis u and the rotation angle θ.

[0136] The display content layout in the virtual scene is optimized according to the user intention model. The optimization formula is:

[0137]

[0138] Among them, L opt is the optimized display layout, L is the candidate display layout, and w i is the weight in the user intention model, n is the total number of contents, D i (L) is the distance between the user's focus target and the displayed content,

[0139] The steps for making corresponding adjustments also include:

[0140] Record and self-learn the user's operating habits, dynamically adjust the initial state of the virtual scene, and the self-learning formula is: initial =E[p user ],

[0141] Among them, is the initial position, p user is the position distribution in the user's operation history, E is the expected value calculation, and the adjusted virtual scene is transmitted to the user through visual and tactile feedback;

[0142] Specifically, through positioning, scaling, rotation and display content optimization, the AR engine can dynamically adjust the virtual scene, accurately respond to the user's intentions, and introduce a self-learning mechanism that can optimize the initial state and response logic after multiple user interactions.

[0143] Step S4, the adjusted virtual scene is deeply integrated with the real environment through the light and shadow consistency calculation and physical property simulation technology of the AR engine, and dynamically adapts to the changes in ambient light to obtain the final interactive effect;

[0144] The adjusted virtual scene is deeply integrated with the real environment through the AR engine's light and shadow consistency calculation and physical property simulation technology.

[0145] The AR engine captures the lighting conditions in the real environment in real time, including the direction, intensity, and color temperature of the light source, and renders shadows and highlights through the global illumination model, so that the virtual objects are consistent with the light and shadow changes in the real environment;

[0146] The AR engine simulates the physical interaction characteristics between virtual objects and the real environment;

[0147] The AR engine dynamically adjusts the presentation of virtual objects based on real-time collected environmental data;

[0148] Specifically, through light and shadow consistency calculation, physical property simulation and dynamic environment adaptation, the deep integration of virtual scenes and real environments is achieved, enhancing the user's sense of immersion and the naturalness of interaction.

[0149] Step S5, the final interaction effect is fed back to the user, and the user's further operations are recaptured and updated to the next round of step S1 input signal using the feedback mechanism;

[0150] Feedback mechanisms include:

[0151] Adjustment of signal capture and feature extraction parameters,

[0152] and adjust the user intent model parameters based on user repetitive behavior,

[0153] The steps in the input signal of step S1 updated to a new round using the feedback mechanism are:

[0154] Capture voice and gesture signals and adjust the corresponding parameters. The adjustment formula is:

[0155] θ speech,new =θ speech,old +Δθ speech ,

[0156] θ gesture,new =θ gesture,old +Δθ gesture ,

[0157] Among them, θ speech,new and θ gesture,new are the updated speech and gesture capture parameters, θ speech,old and θ gesture,old are the original speech and gesture capture parameters, Δθ speech and Δθ gesture is the incremental parameter adjusted by user feedback,

[0158] The optimization formula for the feature extraction model is:

[0159] z′=z+Δz feedback ,

[0160] Among them, z' is the optimized eigenvector, z is the original eigenvector, Δz feedback Optimization increments introduced for feedback mechanisms;

[0161] Update the prior probability and conditional probability in the user intention model based on the user's repeated behavior and historical operation patterns.

[0162] To model historical behavior, the model formula is:

[0163] P(I) new =αPhistory (I)+(1-α)P prior (I)

[0164] Among them, P(I) new is the updated prior probability of user intention, α is the historical weight factor, P history (I) represents the prior probability of user intention I based on the user's historical behavior statistics, P prior (I) represents the initial prior probability of the preset user intention I,

[0165] Update the conditional probability, the update formula is:

[0166] P(z′|I) new =P(z′|I)+ΔP feedback (z′|I),

[0167] in,

[0168] P(z'|I) new is the conditional probability after optimization, P(Z'}I) is the conditional probability before optimization, ΔP feedback (z'}I) is the conditional probability increment based on user operation feedback,

[0169] The feedback mechanism not only adjusts the model parameters, but also the adjustment formula is:

[0170] θ′=θ+Δθ feedback ,

[0171] Among them, θ' is the global parameter set after optimization, θ is the global parameter set before optimization, Δθ feedback parameter increments calculated for the feedback mechanism;

[0172] Specifically, through the feedback mechanism, adaptive optimization is achieved in the signal capture, feature extraction and intent modeling links, gradually improving the ability to adapt to dynamic changes.

[0173] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A user experience interaction method based on a VR engine, characterized in that: include, Step S1, deploying a signal interface in the AR engine to capture user input signals, including voice commands, gestures, touch operations, and line of sight changes, and using a deep learning model to extract features from the input signals to generate signal features; Step S2, integrating signal features, using a probabilistic graph model to generate a user intention model, and predicting the user's operation purpose. The prediction content includes: interaction type, target object, and task goal; Step S3, based on the prediction result of the user intention model, the AR engine drives the virtual scene to make corresponding adjustments, including positioning, scaling or rotation of virtual objects, and adaptive optimization of displayed content; Step S4, the adjusted virtual scene is deeply integrated with the real environment through the light and shadow consistency calculation and physical property simulation technology of the AR engine, and dynamically adapts to the changes in ambient light to obtain the final interactive effect; In step S5, the final interaction effect is output and fed back to the user, and the user's further operations are recaptured and updated to a new round of step S1 input signals using a feedback mechanism.

2. A user experience interaction method based on a VR engine as claimed in claim 1, characterized in that: The AR engine provides a synchronous multimodal data collection framework, captures user input signals in real time, and locates the user's position in the real environment through synchronous positioning and mapping SLAM technology.

3. A user experience interaction method based on a VR engine as claimed in claim 2, characterized in that: The step of extracting features from the input signal using a deep learning model to generate signal features is as follows: The Transformer model is used to encode the speech waveform and extract semantic information. The encoding formula is: z speech =f speech (x speech ;θ speech ), Among them, z speech is the extracted semantic feature vector, x speech is the input speech waveform signal, θ speech is the training parameter of the speech model, f speech is a deep learning function for speech feature extraction; Capture the user's continuous gesture action frames, combine the 3D convolutional network CNN and the long short-term memory network LSTM for modeling, and the capture formula is: z gesture =f gesture (x gesture ;θ gesture ), Among them, z gesture is the high-dimensional feature representation of the gesture signal, x gesture is the input gesture video sequence, θ gesture is the training parameter of the gesture model, f gesture Extract model function for gesture features; Extract touch signal features, including click position and sliding trajectory, and encode them using multi-layer perceptron MLP. The encoding formula is: z touch =f touch (x touch ;θ touch ), Among them, z touch is the characteristic vector of the touch signal, x touch is the touch input data, including the click position and sliding trajectory, θ touch is the touch model parameter, f touch Extract function for touch feature; Extract the sight signal features, use ResNet to process the sight trajectory data, and extract the features of the user's gaze area. The extraction formula is: z gaze =f gaze (x gaze ;θ gaze ), Among them, z gaze is the high-dimensional feature of the line of sight signal, x gaze Input data for the sight trajectory, θ gaze is the training parameter of the line of sight model, f gaze Extract model function for sight line feature; The feature fusion formula is: With fused =f fusion (With speech ,With gesture ,With touch ,With gaze ;θ fusion ), Among them, z fused is the fused multimodal feature representation, f fusion is the multimodal fusion function, θ fusion is the multimodal fusion parameter.

4. A user experience interaction method based on a VR engine as claimed in claim 3, characterized in that: The method of fusing the signal features also includes using an AR engine to analyze the depth information of the real scene and the three-dimensional structure of the virtual scene to obtain environmental features, and fusing the environmental features with the signal features input by the user; Depth information includes the distance to the object and the surface characteristics of the object.

5. A user experience interaction method based on a VR engine as claimed in claim 4, characterized in that: The steps of fusing signal features, using a probabilistic graph model to generate a user intention model, and predicting the user's operation purpose are as follows: Taking the multimodal signal feature z in step S1 fused As input features, the AR engine is used to extract environmental features e, including the depth information e of the real scene depth and the three-dimensional structure of the virtual scene 3D ,The depth information includes the distance and surface characteristics of the object, and the ,three-dimensional structure is the shape and spatial distribution of the object; Perform feature fusion, the fusion formula is: With combined =f attention (With fused ,e;θ fusion ), Among them, z combined is the fused feature, f attention is the cross-modal attention mechanism, θ fusion is the parameter of feature fusion; The user intention model is modeled through the dynamic Bayesian network DBN, and the posterior probability of the intention is calculated. The calculation formula is: Among them, P(I|z combined ) is the given fusion feature z combined Under this condition, the posterior probability of user intention I, P(z combined |I) is the conditional probability of the given intention, P(I) is the prior probability of the intention, ∑ i (·) indicates all possible intentions I i Normalize.

6. A user experience interaction method based on a VR engine as claimed in claim 5, characterized in that: The step of driving the virtual scene to make corresponding adjustments based on the prediction result of the user intention model is as follows: Adjust the positioning of virtual objects, dynamically update the three-dimensional spatial position of virtual objects based on the target objects and operation area predictions in the user's intention, The update formula is: p new =p old +Δp, Among them, p new is the new position vector of the virtual object, p old is the original position vector of the virtual object, Δp is the position change vector; Perform scaling adjustments on virtual objects and adjust the proportions of virtual objects based on task target predictions. The scaling formula is: S new =S old ·k, Among them, S new is the updated scaling ratio, S old is the original scaling ratio, k is the scaling factor; Perform rotation adjustment of the virtual object, and adjust the direction of the virtual object through the user's rotation operation. The rotation formula is: R new =R old ·R(θ,u), Among them, R new is the updated rotation matrix, R old is the initial rotation matrix, R(θ,u) is the rotation matrix, which consists of the rotation axis u and the rotation angle θ. The display content layout in the virtual scene is optimized according to the user intention model. The optimization formula is: Among them, L opt is the optimized display layout, L is the candidate display layout, and w i is the weight in the user intention model, n is the total number of contents, D i (L) is the distance between the user's focus target and the displayed content.

7. A user experience interaction method based on a VR engine as claimed in claim 6, characterized in that: The step of making corresponding adjustments also includes: Record and self-learn the user's operating habits, dynamically adjust the initial state of the virtual scene, and the self-learning formula is: initial =E[p user ], Among them, is the initial position, p user is the position distribution in the user's operation history, E is the expected value calculation, and the adjusted virtual scene is transmitted to the user through visual and tactile feedback.

8. A user experience interaction method based on a VR engine as claimed in claim 7, characterized in that: The adjusted virtual scene is deeply integrated with the real environment through the light and shadow consistency calculation and physical property simulation technology of the AR engine. The AR engine captures the lighting conditions in the real environment in real time, including the direction, intensity and color temperature of the light source, and renders shadows and highlights through a global illumination model, so that the virtual objects are consistent with the light and shadow changes in the real environment; The AR engine simulates the physical interaction characteristics between virtual objects and the real environment; The AR engine dynamically adjusts the presentation of virtual objects based on the environmental data collected in real time.

9. A user experience interaction method based on a VR engine as claimed in claim 8, characterized in that: The feedback mechanism includes: Adjustment of signal capture and feature extraction parameters, and adjusting the user intent model parameters based on user repetitive behaviors.

10. A user experience interaction method based on a VR engine as claimed in claim 9, characterized in that: The step of using the feedback mechanism to update the input signal of the new round of step S1 is: Capture voice and gesture signals and adjust the corresponding parameters. The adjustment formula is: i speech,new =θ speech,old +Δθ speech , i gesture,new =θ gesture,old +Δθ gesture , Among them, θ speech,new and θ gesture,new are the updated speech and gesture capture parameters, θ speech,old and θ gesture,old are the original speech and gesture capture parameters, Δθ speech and Δθ gesture is the incremental parameter adjusted by user feedback, The optimization formula for the feature extraction model is: z′=z+Δz feedback , Among them, z' is the optimized eigenvector, z is the original eigenvector, Δz feedback Optimization increments introduced for feedback mechanisms; Update the prior probability and conditional probability in the user intention model based on the user's repeated behavior and historical operation patterns. To model historical behavior, the model formula is: P(I) new =αP history (I)+(1-α)P prior (I), Among them, P(I) new is the updated prior probability of user intention, α is the historical weight factor, P history (I) represents the prior probability of user intention I based on the user's historical behavior statistics, P prior (I) represents the initial prior probability of the preset user intention I, Update the conditional probability, the update formula is: P(z′|I) new =P(z′|I)+ΔP feedback (z′|I), in, P(z'|I) new is the conditional probability after optimization, P(z'|I) is the conditional probability before optimization, ΔP feedback (z'|I is the conditional probability increment based on user operation feedback, The feedback mechanism not only adjusts the model parameters, but also the adjustment formula is: θ′=θ+Δθ feedback , Among them, θ' is the global parameter set after optimization, θ is the global parameter set before optimization, Δθ feedback Parameter increments calculated for the feedback mechanism.

Images