VR motion sickness detection method based on Mamb-Transform fusion
The VR motion sickness detection method based on Mamba-Transformer fusion addresses the shortcomings of existing technologies in modal fusion depth, multi-scale modeling, and individual adaptability, achieving high-precision, low-latency motion sickness prediction, and is suitable for health monitoring in virtual reality environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NORTH CHINA UNIVERSITY OF TECHNOLOGY
- Filing Date
- 2026-01-27
- Publication Date
- 2026-05-12
AI Technical Summary
Existing VR motion sickness detection technologies have shortcomings in modal fusion depth, multi-scale modeling, and individual adaptability. They are unable to simultaneously capture the "long-term accumulation" and "instantaneous mutation" triggering mechanisms of motion sickness, and the model's generalization ability is weak due to cross-user individual differences.
A VR motion sickness detection method based on Mamba-Transformer fusion is adopted. The method aligns eye movement and head movement features through a multi-view embedding module, captures global and local features using a dual-path temporal modeling network, and dynamically adjusts feature weights by combining a fusion routing network to achieve adaptive prediction.
It significantly improves prediction accuracy and robustness across users and scenarios, reduces deployment costs, can keenly capture visual-vestibular conflict, adapts to individual differences, and improves the model's prediction accuracy and computational efficiency.
Smart Images
Figure CN122004759A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of virtual reality and human-computer interaction technology, and in particular to a VR motion sickness detection method based on Mamba-Transformer fusion. Background Technology
[0002] Motion sickness is a major side effect of prolonged exposure to virtual environments (VEs), with symptoms including nausea, disorientation, and eye strain. As VR technology becomes increasingly prevalent in entertainment, healthcare, and training, effectively predicting and mitigating motion sickness is becoming crucial.
[0003] Existing technologies have shifted from early subjective questionnaire assessments (such as the SSQ proposed by Kennedy et al.) to objective, automated predictions based on physiological signals. Although some studies have incorporated eye-tracking and head-movement data, there are still significant shortcomings in terms of technical means and modeling depth.
[0004] First, the depth of modal interaction is insufficient and the differences in physiological attributes are ignored. Although existing research has begun to integrate multimodal data, the fusion mechanisms are too simplistic. For example, the MS-STTN model proposed by Jeong and Han, while using Transformer to extract the spatiotemporal features of eye movements and head movements, still faces problems such as reliance on synchronous data acquisition and excessively large label intervals. In addition, existing solutions often simply stitch modal data together in the early stages, ignoring the physical attribute differences in the temporal characteristics between eye movement signals (high frequency, fine detail, reflecting instantaneous disturbances) and head movement signals (low frequency, spatial orientation, reflecting long-term intentions), making it difficult for the model to capture the core trigger of motion sickness, "visual-vestibular conflict."
[0005] Second, the lack of multi-scale temporal modeling capabilities makes it difficult to balance global and local features. Motion sickness involves both "long-term accumulated discomfort" and "local mutations" caused by sudden visual disturbances. Existing CNN-LSTM architectures (such as PhysioDNN or HMDPrediction) or basic Transformer architectures (such as Informer) struggle to capture features at both scales simultaneously. For example, models like DeepLSTM tend to lose long-range dependencies when processing long sequences due to the limitations of their recursive structures; while traditional global attention mechanisms suffer from high computational complexity and insufficient sensitivity when dealing with subtle local fluctuations.
[0006] Third, there is a lack of adaptive adjustment mechanisms to address individual differences. The perception of motion sickness exhibits strong individual variability. While some solutions, such as MAC, introduce a dual-attention mechanism for fusion, their architectures are mostly static and cannot dynamically adjust the prediction focus based on the user's real-time physiological feedback. Although the research by Keshavarz et al. used machine learning to quantify motion sickness, the lack of long-term dynamic modeling resulted in weak generalization ability and predictive robustness across users and scenarios.
[0007] Given the limitations of existing solutions in terms of modal fusion depth, multi-scale modeling, and individual adaptability, designing a predictive model that can adapt to specific physical properties of eye / head movements and adaptively balance global trends and local disturbances is a long-standing technical challenge in the field of VR health monitoring. Summary of the Invention
[0008] To address the aforementioned shortcomings in existing technologies, this invention provides a VR motion sickness detection method based on Mamba-Transformer fusion. This method solves the limitations of existing motion sickness detection technologies, such as the limitations of single models in processing multi-scale physiological signals, insufficient modal fusion depth, difficulty in taking into account both the "long-term accumulation" and "transient mutation" dual induction mechanisms of motion sickness, and weak model generalization ability due to cross-user individual differences.
[0009] To achieve the aforementioned objectives, the present invention employs the following technical solution: a VR motion sickness detection method based on Mamba-Transformer fusion, comprising: Multimodal temporal data in a virtual reality environment is acquired and preprocessed to obtain raw eye movement features and raw head movement features; the multimodal temporal data includes eye movement data reflecting high-frequency visual attention and head movement data reflecting low-frequency spatial posture. The original eye movement features and original head movement features are input into a pre-trained VR motion sickness detection network to detect motion sickness and obtain motion sickness detection results.
[0010] Furthermore, the eye movement data includes the eyelid opening and closing angle of the left and right eyes, the three-dimensional coordinates of the gaze direction, the three-dimensional coordinates of the gaze origin, and the pupil diameter; the head movement data includes the three-dimensional coordinates of the head position and the head rotation angle.
[0011] Furthermore, the preprocessing specifically involves: resampling the multimodal time series data according to a unified set frequency to unify the time granularity; normalizing the data after unifying the time granularity and mapping it to the [0,1] interval to obtain the preprocessed multimodal time series data.
[0012] Furthermore, the VR motion sickness detection network includes: A multi-view embedding module is used to align the original eye-tracking features and the original head-tracking features based on a cross-attention mechanism to obtain an embedding representation containing modality conflict information. A dual-path temporal modeling network is used to capture preliminary global and preliminary local features based on embedded representations containing modal conflict information. The fusion routing network includes a modal fusion subnetwork and a path fusion subnetwork. The modal fusion subnetwork is used to obtain global and local features based on preliminary global and local features. The path fusion subnetwork is used to dynamically calculate the weights of global and local features, and then perform weighted fusion of global and local features based on the obtained weights to obtain weighted fused features. The motion sickness prediction module is used to predict motion sickness scores based on weighted fusion features.
[0013] Furthermore, based on the cross-attention mechanism, the original eye-tracking features and original head-tracking features are aligned to obtain an embedding representation containing modality conflict information, specifically: Map either the original eye-tracking feature or the original head-tracking feature to a query vector, and map the other modality to a key vector and a value vector; The cross-attention weight vector is calculated using the following expression:
[0014] In the formula, Represents the cross-attention weight vector. express Activation function Represents the query vector. The superscript T denotes the key vector; the superscript T denotes the transpose of the matrix. Indicates the dimension of the key vector; The fused embedding representation is calculated based on the cross-attention weight vector, and its expression is:
[0015] In the formula, To integrate embedded representations, It is a value vector; Extract token embeddings from raw eye-tracking and raw head-tracking features; The embedded representation containing modal conflict information is computed based on the fusion embedded representation, and its expression is:
[0016] In the formula, This represents an embedding representation that contains modal conflict information. Token embeddings representing raw eye movement features or raw head movement features.
[0017] Furthermore, the dual-path temporal modeling network includes a global path for capturing preliminary global features and a local path for capturing preliminary local features; The specific method for capturing preliminary global features is as follows: The embedded representation containing modal conflict information is input into the Mamba module, and the state space model SSM in the Mamba module is used to perform recursive state updates to obtain preliminary global temporal features. The preliminary global temporal features are input into the KAN hybrid expert module to obtain the preliminary global features, the expression of which is:
[0018] In the formula, Indicates preliminary global features. This represents the expert network index in the KAN hybrid expert module. Indicates the number of expert networks. Indicates the first The activation weights of an expert network. Indicates the first An expert network performs a nonlinear transformation on the input features using spline functions. It represents global temporal characteristics.
[0019] Furthermore, specific methods for capturing preliminary local features include: Position encoding is performed on the original eye movement features and the original head movement features respectively; The location-encoded features are separated into low-frequency trend terms. and high-frequency residuals ; Slicing operations are performed on the low-frequency trend term and the high-frequency residual term respectively to obtain overlapping time slices; A local window attention mechanism is applied to the overlapping time slices to obtain the trend branch output and the residual branch output; The trend branch output and the residual branch output are added together, and then average pooling is performed along the channel dimension to obtain preliminary local features.
[0020] Furthermore, preliminary global features include preliminary eye-movement global features and preliminary head-movement global features; preliminary local features include preliminary eye-movement local features and preliminary head-movement local features. For the initial global features, the following steps are used to generate global features: Linear transformations were performed on the preliminary global eye movement features and the preliminary global head movement features to obtain the eye movement modality saliency score and the head movement modality saliency score, respectively. The eye-movement modality saliency score and the head-movement modality saliency score are concatenated and then subjected to softmax normalization to generate an attention weight vector that satisfies the probability distribution. The original eye-tracking features and original head-tracking features are dynamically scaled using element-wise multiplication with attention weight vectors that satisfy the probability distribution; The dynamically scaled features are summed element by element to obtain the global features; For the initial local features, generate local features using the same method as generating global features.
[0021] Furthermore, the weights of global features are dynamically calculated through the path fusion sub-network, specifically including: The original eye movement features and original head movement features are added element by element to obtain the input sequence; The input sequence is mapped to a unified latent space representation through a linear projection layer, resulting in dimension-aligned features. Dimensionally aligned features are compressed using adaptive average pooling; Based on the compressed features, normalized weights for global features and local features are generated through a fully connected layer and a softmax activation function.
[0022] Furthermore, the loss function used when training the VR motion sickness detection network is:
[0023] In the formula, Represents the total loss function. This represents the mean squared error loss function. For balance coefficient, This is the regularization loss term.
[0024] The beneficial effects of this invention are as follows: 1. By heterogeneously fusing Mamba and Transformer, this invention overcomes the trade-off between single models when dealing with long-term cumulative effects (such as dizziness caused by prolonged exposure) and local details (such as discomfort caused by momentary visual lag). This invention captures the cumulative effects of prolonged exposure through a global path, captures the momentary discomfort caused by sudden visual disturbances through a local path, and dynamically balances the two through an adaptive fusion mechanism. This significantly improves prediction accuracy and robustness across users and scenarios without requiring complex EEG devices.
[0025] 2. Relying on the sensor data built into the HMD (Head-Mounted Display), no additional EEG / ECG equipment is required, reducing deployment costs. By distinguishing between high-frequency eye movements (reflecting transient visual stimulus responses) and low-frequency head movements (reflecting vestibular spatial movements), it provides high-quality input data with clear physical meaning for subsequent capture of "visual-vestibular conflict".
[0026] 3. Early strong interaction between modalities was achieved through a cross-attention mechanism. Since motion sickness is often caused by visual-vestibular conflict, this design allows the model to perceive the inconsistency between eye movements (visual) and head movements (vestibular) trajectories in the early stages of encoding, which is more sensitive to capturing the physiological causes of motion sickness than simple feature splicing.
[0027] 4. KAN is introduced to replace the traditional MLP as the expert network. By utilizing the smoothing properties of spline functions, it can better fit the nonlinear trajectory of physiological signals and can dynamically select experts according to different inputs, effectively solving the problem of large differences in the sensitivity of different users to motion sickness (individual variability).
[0028] 5. The decomposition strategy separates steady-state behavior from sudden fluctuations, enhancing the model's sensitivity to subtle signal changes. The local window attention mechanism restricts the focus to a short time window, accurately capturing transient perturbations that cause motion sickness (such as rapid line-of-sight jumps), while reducing computational complexity to linear. It not only accurately captures transient perturbations that cause acute motion sickness and supplements details ignored by the global path, but also improves computational efficiency. Attached Figure Description
[0029] Figure 1 A flowchart of a VR motion sickness detection method based on Mamba-Transformer fusion is provided for an embodiment; Figure 2 This is a schematic diagram illustrating the specific process of motion sickness detection using the VR motion sickness detection network provided in this embodiment. Detailed Implementation
[0030] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0031] like Figure 1 As shown, in one embodiment of the present invention, a VR motion sickness detection method based on Mamba-Transformer fusion includes the following steps: S1. Acquire multimodal temporal data in a virtual reality environment and preprocess it to obtain raw eye movement features and raw head movement features.
[0032] To address the physiological differences in motion sickness triggers, modal features of different frequency bands were extracted from VR headsets. In this embodiment, during the S1 high-fidelity data acquisition and preprocessing, the input data came from virtual reality headsets with eye-tracking capabilities, such as the HTC VIVE Pro Eye. The multimodal temporal data included eye-tracking data reflecting high-frequency visual attention and head-tracking data reflecting low-frequency spatial posture. The eye-tracking data included eyelid opening (Eyeopenness), gaze direction (x, y, z) three-dimensional coordinates, gaze origin (x, y, z) three-dimensional coordinates, and pupil diameter. The head-tracking data included head position (x, y, z) three-dimensional coordinates and head rotation angle (yaw, pitch, roll).
[0033] Data acquisition relies solely on the HMD's built-in sensors, eliminating the need for additional EEG / ECG equipment and reducing deployment costs. By distinguishing between high-frequency eye movements (reflecting transient visual stimulus responses) and low-frequency head movements (reflecting vestibular spatial motion), it provides high-quality input data with clear physical meaning for subsequent capture of "visual-vestibular conflict."
[0034] Considering that the original data typically has a high sampling rate (90Hz) and contains noise, all data are downsampled to 25Hz to resample the multimodal time series data and unify the time granularity. This frequency can retain the time resolution required to capture motion sickness features while significantly reducing the computational load. The data with unified time granularity is then Min-Max normalized and mapped to the [0,1] interval to eliminate the dimensional differences between eye movement and head movement data, resulting in preprocessed multimodal time series data.
[0035] S2. Input the original eye movement features and original head movement features into the pre-trained VR motion sickness detection network to perform motion sickness detection and obtain motion sickness detection results.
[0036] The specific testing process is as follows: Figure 2 As shown.
[0037] The VR motion sickness detection network includes: A. Multi-View Embedding module: This module aligns original eye-tracking and head-tracking features based on a cross-attention mechanism to obtain an embedded representation containing modality conflict information. Specifically: Map either the original eye-tracking features or the original head-tracking features to a query vector, and map the other modality to a key vector and a value vector. Specifically, when calculating the eye-tracking embedding: , ,
[0038] in, These are primitive eye movement features. These are primitive head movement characteristics. , , All are projection weights, but the opposite applies when calculating head-motion embedding.
[0039] The cross-attention weight vector is calculated using the following expression:
[0040] In the formula, Represents the cross-attention weight vector. express Activation function Represents the query vector. The superscript T denotes the key vector; the superscript T denotes the transpose of the matrix. Indicates the dimension of the key vector; The fused embedding representation is calculated based on the cross-attention weight vector, and its expression is:
[0041] In the formula, To integrate embedded representations, It is a value vector; Extract token embeddings from raw eye-tracking and raw head-tracking features; The embedded representation containing modal conflict information is computed based on the fusion embedded representation, and its expression is:
[0042] In the formula, This represents an embedding representation that contains modal conflict information. Token embeddings representing raw eye movement features or raw head movement features.
[0043] Feature alignment in this manner not only includes the temporal information of the modality itself in the output feature vector, but also explicitly encodes the "degree of conflict" or "consistency" between eye movements and head movements at that moment, laying the foundation for subsequent capture of motion sickness triggers. Early strong interaction between modalities is achieved through a cross-attention mechanism. Since motion sickness is often caused by visual-vestibular conflict, this design allows the model to perceive the trajectory inconsistency between eye movements (visual) and head movements (vestibular) in the early stages of encoding, enabling it to more sensitively capture the physiological triggers of motion sickness than simple feature splicing.
[0044] B. A dual-path temporal modeling network, comprising a global path for capturing initial global features and a local path for capturing initial local features. The complementary design of the dual-path heterogeneous modeling, with the global and local paths respectively addressing different temporal modeling challenges.
[0045] The specific method for capturing global features is as follows: The embedded representation containing modal conflict information is input into the Mamba module, and the state-space model (SSM) in the Mamba module is used for recursive state updates to obtain global temporal features; the state-space model update rule is as follows:
[0046]
[0047] In the formula, The hidden state is represented by A, B, and C, which are all trainable matrices.
[0048] The global temporal features are input into the KAN hybrid expert module to obtain preliminary global features, the expression of which is:
[0049] In the formula, Indicates preliminary global features. This represents the expert network index in the KAN hybrid expert module. Indicates the number of expert networks. Indicates the first The activation weight of each expert Indicates the first An expert network performs a nonlinear transformation on the input features using spline functions. This represents global temporal features. Preliminary global features include preliminary eye-tracking global features and preliminary head-tracking global features.
[0050] The Mamba architecture, with its linear time complexity, enables efficient parallel processing of long-sequence data, making it suitable for capturing the cumulative features of motion sickness over extended periods. Compared to traditional MLPs, KANs can more smoothly fit the complex fluctuations in physiological signals. Introducing KANs to replace traditional MLPs as the expert network, leveraging the smoothing properties of spline functions, allows for better fitting of the nonlinear trajectories of physiological signals. Combined with the MoE mechanism, it can dynamically select experts based on different inputs, effectively addressing the issue of individual variability in the sensitivity to motion sickness among different users.
[0051] Specific methods for capturing preliminary local features include: Position encoding is performed on the original eye movement features and the original head movement features respectively; The location-encoded features are separated into low-frequency trend terms that reflect steady changes. and the high-frequency residual term reflecting high-frequency jitter Slicing operations are performed on the low-frequency trend term and the high-frequency residual term respectively to obtain overlapping time slices; Applying a local window attention mechanism to overlapping time slices yields the trend branch output and the residual branch output; this mechanism restricts the model to compute only within a fixed time window. The elements within the context are correlated, thus ignoring irrelevant long-range background and focusing on capturing microscopic features that may induce acute discomfort, such as abrupt changes in gaze or momentary head tremors. Its expression is:
[0052] In the formula, Display window Attention function, Display window The query vector, Display window The key vector, Display window The value vector.
[0053] The trend branch output and the residual branch output are summed and then averaged along the channel dimension to obtain preliminary local features. These preliminary local features include preliminary eye-tracking local features and preliminary local-global features.
[0054] The decomposition strategy separates steady-state behavior from sudden fluctuations, enhancing the model's sensitivity to subtle signal changes. The local window attention mechanism restricts focus to a short time window, accurately capturing transient perturbations that cause motion sickness (such as rapid line-of-sight jumps) while reducing computational complexity to linear. This not only accurately captures transient perturbations causing acute motion sickness and supplements details ignored by the global path, but also improves computational efficiency.
[0055] By heterogeneously fusing Mamba and Transformer, this invention overcomes the trade-off challenge of single models when dealing with long-term cumulative effects (such as dizziness caused by prolonged exposure) and local details (such as discomfort caused by momentary visual lag). This invention captures the cumulative effects of prolonged exposure through a global path, captures the momentary discomfort caused by sudden visual disturbances through a local path, and dynamically balances the two through an adaptive fusion mechanism. This significantly improves prediction accuracy and robustness across users and scenarios without requiring complex EEG devices.
[0056] C. Fusion Router, comprising a modal fusion subnetwork and a path fusion subnetwork. The modal fusion subnetwork is used to obtain global and local features based on preliminary global and local features. The path fusion subnetwork is used to dynamically calculate the weights of the global and local features, and then performs a weighted fusion of the global and local features based on the obtained weights to obtain the weighted fused features, the expression of which is:
[0057] in, For weighted fusion features, The weights are the weights corresponding to the global features. These are the weights corresponding to local features.
[0058] Preliminary global features include preliminary eye movement global features and preliminary head movement global features; preliminary local features include preliminary eye movement local features and preliminary head movement local features.
[0059] For the initial global features, the following steps are used to generate global features: Linear transformations were performed on the preliminary global eye movement features and the preliminary global head movement features to obtain the eye movement modality saliency score and the head movement modality saliency score, respectively. The eye-movement modality saliency score and the head-movement modality saliency score are concatenated and then subjected to softmax normalization to generate an attention weight vector that satisfies the probability distribution. The original eye-tracking features and original head-tracking features are dynamically scaled using element-wise multiplication with attention weight vectors that satisfy the probability distribution; The dynamically scaled features are summed element by element to obtain the global features; For the initial local features, generate local features using the same method as generating global features.
[0060] The weights of global features are dynamically calculated through the path fusion subnetwork, specifically including: The original eye movement features and original head movement features are added element by element to obtain the input sequence; The input sequence is mapped to a unified latent space representation through a linear projection layer, resulting in dimension-aligned features. Dimensionally aligned features are compressed using adaptive average pooling; Based on the compressed features, normalized weights for global features and local features are generated through a fully connected layer and a softmax activation function.
[0061] A model is adaptively designed to address the pathological mechanisms and temporal characteristics of visual-vestibular conflict in motion sickness. By heterogeneously fusing Mamba and Transformer, the challenge of balancing long-term cumulative effects (such as vertigo caused by prolonged exposure) with local details (such as discomfort caused by momentary visual interruptions) in single models is overcome. This invention captures the cumulative effects of prolonged exposure through global pathways, captures the momentary discomfort caused by sudden visual disturbances through local pathways, and dynamically balances the two through an adaptive fusion mechanism. This significantly improves prediction accuracy and robustness across users and scenarios without requiring complex EEG devices.
[0062] D. Motion Sickness Prediction Module: This module predicts motion sickness scores based on weighted fusion features using a linear regression layer. Its expression is as follows:
[0063] in, This is a predictive value for motion sickness. Let b be the weight matrix and b be the bias term.
[0064] The loss function used when training the VR motion sickness detection network is:
[0065] In the formula, Represents the total loss function. This represents the mean squared error loss function. For balance coefficient, This is the regularization loss term.
[0066] The Adam optimizer was used, with an initial learning rate set to 1e-4. The training process consisted of 128 epochs, during which the learning rate was dynamically adjusted using a cosine annealing strategy. After calculating the loss for each batch, gradient backpropagation was performed. For mini-batches smaller than a preset size, a gradient accumulation strategy was used to maintain stable updates until the validation set error was minimized.
[0067] The adaptive fusion mechanism can dynamically adjust the contributions of global and local features based on sample characteristics (whether long-term accumulation or sudden stimulation is dominant). For example, it increases global weights in calm scenarios and local weights in violent motion scenarios, significantly enhancing the model's scenario robustness. The addition of a KAN regularization loss term effectively controls the complexity of the spline function, preventing overfitting on the training data and ensuring the continuity and smoothness of the motion sickness score prediction results.
[0068] To verify the beneficial effects of this implementation scheme, comparative and ablation experiments were conducted on two public datasets (MSCVR and VRCybersickness).
[0069] 1. Comparative experimental data: Table 1 shows that the present invention (CyberGATE) outperforms existing technologies in all core metrics for measuring prediction accuracy.
[0070] Table 1
[0071] On the MSCVR dataset, our proposed solution reduced the mean absolute error (MAE) to 0.08 and the root mean square error (RMSE) to 0.19. In comparison, the MAE of the pure Transformer-based MS-STTN model is 0.12, and the MAE of the CNN-LSTM-based AutoDetectionCS model is as high as 0.56. This indicates that our proposed solution can more accurately regress the user's true level of motion sickness.
[0072] On the VR Cybersickness dataset, our approach also achieved the best overall performance (MAE 0.52), significantly outperforming the standard Mamba model (MAE 0.70) and the Informer model (MAE 0.95). This demonstrates that relying solely on long sequence modeling (Mamba / Informer) is insufficient to capture all features of motion sickness, and must be combined with the local paths designed in our approach.
[0073] 2. Ablation experiment data: Table 2 shows that the present invention (FullCyberGATE) is not a simple superposition of modules, but rather the organic combination of each part produces a synergistic effect.
[0074] Table 2
[0075] First, the necessity of the global path was verified. After removing the Mamba backbone or KAN experts, the RMSE error increased to 0.29 and 0.84 (MSCVR), respectively, proving that long-range cumulative features are crucial for predicting the baseline value.
[0076] Secondly, the necessity of local paths was verified. Removing local paths caused the MAE to deteriorate from 0.08 to 0.37, proving that ignoring instantaneous fluctuations would severely reduce the sensitivity of the model.
[0077] Furthermore, the effectiveness of adaptive fusion was verified. After removing the adaptive fusion mechanism (i.e., using fixed weights), the MAE error increased to 1.05, indicating that dynamically adjusting the strategy for different samples is the key to solving the problem of individual differences.
[0078] In summary, this implementation method, through physiological mechanism-driven network design, successfully solves the problems of feature alignment, multi-scale modeling, and individual adaptability in existing technologies, and achieves high-precision, low-latency, and highly generalizable prediction of VR motion sickness.
[0079] This invention provides a VR motion sickness detection method based on Mamba-Transformer fusion, suitable for high-precision health monitoring mechanisms in virtual reality environments. Based on two different motion sickness induction mechanisms—long-term cumulative effects and instantaneous visual perturbations—the method comprehensively considers the temporal complementarity of eye-tracking and head-tracking signals in the modeling process to improve overall prediction performance. Specifically, the global path utilizes a hybrid Mamba and KAN expert network to capture long-range dependency features, while the local path uses a decomposed perceptual PatchTransformer to capture short-term fluctuation features. The weights of both are dynamically adjusted through adaptive fusion routing. Performance comparisons and ablation experiments were conducted on two public datasets, MSCVR and VRCybersickness. The results demonstrate that this method significantly outperforms existing methods in terms of root mean square error (RMSE) and mean absolute error (MAE), with each key module contributing significantly to the performance improvement. This method helps to enhance cross-user robustness while improving the accuracy of motion sickness prediction.
Claims
1. A VR motion sickness detection method based on Mamba-Transformer fusion, characterized in that, include: Multimodal temporal data in a virtual reality environment is acquired and preprocessed to obtain raw eye-tracking features and raw head-tracking features; The multimodal temporal data includes eye-tracking data reflecting high-frequency visual attention and head-tracking data reflecting low-frequency spatial posture. The original eye movement features and original head movement features are input into a pre-trained VR motion sickness detection network to detect motion sickness and obtain motion sickness detection results.
2. The method according to claim 1, characterized in that, Eye movement data includes the opening and closing of the eyelids of the left and right eyes, the three-dimensional coordinates of the gaze direction, the three-dimensional coordinates of the gaze origin, and the pupil diameter; head movement data includes the three-dimensional coordinates of the head position and the head rotation angle.
3. The method according to claim 1, characterized in that, The preprocessing process involves resampling the multimodal time series data at a uniform set frequency to unify the time granularity; then normalizing the data after unifying the time granularity to map it to the [0,1] interval to obtain the preprocessed multimodal time series data.
4. The method according to claim 1, characterized in that, The VR motion sickness detection network includes: A multi-view embedding module is used to align the original eye-tracking features and the original head-tracking features based on a cross-attention mechanism to obtain an embedding representation containing modality conflict information. A dual-path temporal modeling network is used to capture preliminary global and preliminary local features based on embedded representations containing modal conflict information. The fusion routing network includes a modal fusion subnetwork and a path fusion subnetwork. The modal fusion subnetwork is used to obtain global and local features based on preliminary global and local features. The path fusion subnetwork is used to dynamically calculate the weights of global and local features, and then perform weighted fusion of global and local features based on the obtained weights to obtain weighted fused features. The motion sickness prediction module is used to predict motion sickness scores based on weighted fusion features.
5. The method according to claim 4, characterized in that, Based on the cross-attention mechanism, the original eye-tracking features and original head-tracking features are aligned to obtain an embedding representation containing modality conflict information, specifically: Map either the original eye-tracking feature or the original head-tracking feature to a query vector, and map the other modality to a key vector and a value vector; The cross-attention weight vector is calculated using the following expression: In the formula, Represents the cross-attention weight vector. express Activation function Represents the query vector. The superscript T denotes the key vector; the superscript T denotes the transpose of the matrix. Indicates the dimension of the key vector; The fused embedding representation is calculated based on the cross-attention weight vector, and its expression is: In the formula, To integrate embedded representations, It is a value vector; Extract token embeddings from raw eye-tracking and raw head-tracking features; The embedded representation containing modal conflict information is computed based on the fusion embedded representation, and its expression is: In the formula, This represents an embedding representation that contains modal conflict information. Token embeddings representing raw eye movement features or raw head movement features.
6. The method according to claim 4, characterized in that, The dual-path temporal modeling network includes a global path for capturing initial global features and a local path for capturing initial local features; The specific method for capturing preliminary global features is as follows: The embedded representation containing modal conflict information is input into the Mamba module, and the state space model SSM in the Mamba module is used to perform recursive state updates to obtain global temporal features. The global temporal features are input into the KAN hybrid expert module to obtain preliminary global features, the expression of which is: In the formula, Indicates preliminary global features. This represents the expert network index in the KAN hybrid expert module. Indicates the number of expert networks. Indicates the first The activation weights of an expert network. Indicates the first An expert network performs a nonlinear transformation on the input features using spline functions. It represents global temporal characteristics.
7. The method according to claim 6, characterized in that, Specific methods for capturing preliminary local features include: Position encoding is performed on the original eye movement features and the original head movement features respectively; The location-encoded features are separated into low-frequency trend terms and high-frequency residual terms; Slicing operations are performed on the low-frequency trend term and the high-frequency residual term respectively to obtain overlapping time slices; A local window attention mechanism is applied to the overlapping time slices to obtain the trend branch output and the residual branch output; The trend branch output and the residual branch output are added together, and then average pooling is performed along the channel dimension to obtain preliminary local features.
8. The method according to claim 4, characterized in that, Preliminary global features include preliminary eye-movement global features and preliminary head-movement global features; preliminary local features include preliminary eye-movement local features and preliminary head-movement local features. For the initial global features, the following steps are used to generate global features: Linear transformations were performed on the preliminary global eye movement features and the preliminary global head movement features to obtain the eye movement modality saliency score and the head movement modality saliency score, respectively. The eye-movement modality saliency score and the head-movement modality saliency score are concatenated and then subjected to softmax normalization to generate an attention weight vector that satisfies the probability distribution. The original eye-tracking features and original head-tracking features are dynamically scaled using element-wise multiplication with attention weight vectors that satisfy the probability distribution; The dynamically scaled features are summed element by element to obtain the global features; For the initial local features, generate local features using the same method as generating global features.
9. The method according to claim 4, characterized in that, The weights of global features are dynamically calculated through the path fusion subnetwork, specifically including: The original eye-tracking features and original head-tracking features are added element-wise to obtain the input sequence; The input sequence is mapped to a unified latent space representation through a linear projection layer, resulting in dimension-aligned features. Dimensionally aligned features are compressed using adaptive average pooling; Based on the compressed features, normalized weights for global features and local features are generated through a fully connected layer and a softmax activation function.
10. The method according to claim 1, characterized in that, The loss function used when training the VR motion sickness detection network is: In the formula, Represents the total loss function. This represents the mean squared error loss function. For balance coefficient, This is the regularization loss term.