Motion sickness detection device and system and storage medium

Abstract

The invention relates to the technical field of physiological signal processing, and discloses a motion sickness detection device and system and a storage medium. Based on a signal preprocessing module, a feature extraction module, a feature weighting module, a global feature capture module and a classification detection module which are sequentially connected, the device executes the following steps: performing preprocessing and data enhancement on an electroencephalogram signal to obtain a preprocessed electroencephalogram signal; a convolution module is used for sampling up and down in a time frequency dimension to obtain local time frequency characteristics of the preprocessed electroencephalogram signals; performing feature weighting on the local time-frequency features to obtain a feature weighted graph; inputting the feature weighted graph into a Transform module to capture global features to obtain the global features; according to the method, global features are used as data to be classified to be input into a full connection layer and finally pass through a Softmax layer to calculate prediction probabilities of multiple categories, so that efficient feature extraction and accurate classification of the electroencephalogram signals are realized, and motion sickness symptoms of a subject are effectively monitored and evaluated.

Description

Technical Field

[0001] The present application relates to the technical field of physiological signal processing, and in particular to a motion sickness detection device and system and a storage medium. Background Art

[0002] Motion sickness is a common physiological reaction, mainly due to the mismatch between the body's movement and visual perception, and its specific manifestations include nausea, dizziness, sweating, vomiting and other symptoms. With the popularization of virtual reality technology, the impact of virtual reality motion sickness on VR user experience has become more and more obvious. Effective detection and relief of virtual reality motion sickness is a problem that needs to be solved urgently. How to effectively capture the symptoms of virtual reality motion sickness in real time and determine the severity of motion sickness of users in the virtual reality experience in a timely manner is a top priority. Deep learning methods have been proven to be superior to traditional machine learning methods in many fields, such as computer vision, natural language processing, and biomedical signal processing. Mouad Riyad, Abdellah Adib and others developed a ConvNet based on the Inception and Xception architectures, which uses convolutional layers to extract temporal and spatial features, and uses separable convolutions and deep convolutions to achieve faster and more efficient ConvNets. However, although CNN can capture local features in different regions of EEG well, local features between different layers are difficult to associate. The Transformer structure can use a multi-head attention mechanism to learn the relationship between different feature layers, and has good versatility and strong comprehensive feature extraction capabilities.

[0003] In the prior art, there is currently a patented technical document entitled VR Vertigo Objective Assessment Method and Assessment System Fusion of Multi-dimensional EEG Features, and the patent application number is 202310886710.8. The method includes: collecting EEG signal data and removing artifacts and electrooculogram noise to obtain denoised signals; extracting time domain, frequency domain, time-frequency domain and spatial domain features after segmenting the data, and screening out features with higher accuracy; fusing the selected features to generate a feature matrix, and training the VR Vertigo prediction model through machine learning (SVM), thereby improving the accuracy of VR Vertigo assessment in multiple dimensions.

[0004] In the objective evaluation method and system for VR dizziness that integrates multi-dimensional EEG features, although the amount of information is increased by integrating multi-dimensional EEG features, there may be high similarity or overlap between some features, that is, redundant information is introduced, which makes it difficult for the model to effectively distinguish different feature dimensions, affecting the accuracy of the evaluation. The use of support vector machine (SVM) to extract and classify EEG signal features depends largely on the quality of the input features. If the feature extraction is insufficient or inaccurate, the accuracy of SVM will be significantly reduced. At the same time, SVM is mainly used for binary classification tasks, and its performance is poor when monitoring different degrees of motion sickness symptoms and achieving multi-classification goals.

[0005] In addition to the above patents, there is also a patented technical solution called a method and device for evaluating virtual reality motion sickness based on EEG data, with patent application number 202211045684.8. The method includes: obtaining the EEG data of the person to be evaluated in a preset state through an EEG acquisition device; analyzing the data and calculating the characteristic values ​​of a preset frequency band; using a tree feature selection method to extract the characteristic values ​​of a specific channel; inputting the extracted characteristic values ​​into a preset machine learning model to generate a virtual reality motion sickness evaluation result.

[0006] In a virtual reality motion sickness evaluation method and device based on EEG data, motion sickness is evaluated only based on static eigenvalues, while motion sickness reaction has a time accumulation effect. If dynamic feature extraction and time series models (such as long short-term memory network LSTM or Transformer) can be introduced, the dynamic changes of motion sickness in VR experience can be better captured. Summary of the invention

[0007] In order to solve the above problems, the present application provides a motion sickness detection device, system and storage medium, aiming to achieve efficient feature extraction and accurate classification of EEG signals, so as to effectively monitor and evaluate the motion sickness symptoms of the subject. The present application first inputs the EEG signals processed by short-time Fourier transform and continuous wavelet transform into the convolutional neural network (CNN) module, extracts local time-frequency features through convolution operation, and then uses the ECA (Efficient Channel Attention) module to perform weighted processing on the features extracted by CNN, dynamically adjusts the weights of each channel through the attention mechanism, and enhances the model's attention to key EEG features. Furthermore, the Transformer module is used to capture the global features of the EEG signal, and the bottleneck structure is used to reduce the amount of calculation, improve the computational efficiency and generalization ability of the model, and finally the processed features are input into the classifier, and the features are classified into four categories through the Softmax function, so as to evaluate the degree of motion sickness of the subject.

[0008] In order to achieve the above purpose, the technical solution adopted in this application is as follows: In a first aspect, the present application provides a motion sickness detection device, the device comprising a signal preprocessing module, a feature extraction module, a feature weighting module, a global feature capture module and a classification detection module connected in sequence, wherein: The signal preprocessing module is configured to preprocess and data enhance the EEG signal to obtain a preprocessed EEG signal; The feature extraction module is configured to obtain local time-frequency features of the preprocessed EEG signal by upsampling in the time-frequency dimension using the convolution module; The feature weighting module is configured to perform feature weighting on the local time-frequency features to obtain a feature weighted graph; The global feature capture module is configured to input the feature weighted map into the Transformer module to capture the global feature and obtain the global feature; The classification detection module is configured to input the global features as data to be classified into a fully connected layer based on at least two set motion sickness categories, and finally pass through a Softmax layer to calculate the prediction probabilities of multiple categories.

[0009] Furthermore, the signal preprocessing module is further configured to: The EEG signal was filtered using a 0.5 Hz high-pass filter to eliminate low-frequency drift, and then a 50.0 Hz low-pass filter was used to remove high-frequency noise components from the data. The filtered data was subjected to independent principal component analysis to remove artifacts and obtain a continuous signal; wherein the artifacts include eye movements, blinking, electromyography and electrocardiography.

[0010] Furthermore, the preprocessed EEG signal includes a first preprocessed EEG signal and a second preprocessed EEG signal, and the signal preprocessing module is further configured as follows: The EEG segments were extracted from the continuous signal using overlapping sliding windows of set length, with the overlapping of sliding windows set to 50% during the motion sickness episode and 75% before the episode. Then, short-time Fourier transform was used to extract the time-frequency features as the first preprocessed EEG signal. The formula for the short-time Fourier transform is: ; In the formula, represents the original signal, represents the moving window function, is the kernel function of Fourier transform, is an imaginary unit, is the angular frequency, represents the result of short-time Fourier transform, represents a certain time point of the signal, represents the current time position in the signal, Represents the signal frequency component; Based on the continuous signal, continuous wavelet transform is used to perform data processing to decompose the continuous signal into wavelet coefficients at different scales, so as to obtain detailed features at different frequencies and time points as the second preprocessed EEG signal; The formula of the continuous wavelet transform is: ; In the formula, represents the result of continuous wavelet transform, represents the continuous signal to be analyzed, represents the mother wavelet function, represents the time offset, Represents the scale parameter.

[0011] Furthermore, the feature extraction module is configured to be further configured to: The convolution operation is performed on the preprocessed EEG signal, and the calculation formula is as follows: ; In the formula, Indicates l The convolutional layer j Output feature maps, represents the convolution operation, Indicates l The convolutional layer j Deviations, Represents the input of the current convolution calculation, i represents the convolutional layer i Input feature maps, M j Indicates j The set of input feature map indices associated with the output feature maps, Indicates l In the convolutional layer, i Input feature map to j The convolution kernel of the output feature map; The output feature map after convolution is processed by the activation function and the output is: ; In the formula, It represents the value of the output feature map of the convolutional layer after being processed by the activation function. Represents the activation function pair The processing result, Re LU represents the activation function, and max represents the maximum value function; Output Then, batch normalization is performed through the BN layer, that is, the mean and variance are calculated and then standardized and linearly transformed to obtain the local time-frequency characteristics of the preprocessed EEG signal. The calculation process is expressed as: ; In the formula, m represents the current batch data size, represents the mean of all samples in the batch, represents the variance of the batch samples, ε represents a preset positive number to prevent division by zero, , represents the learnable parameters of the batch normalization process, Represents the local time-frequency feature results of the output, Represents the normalized eigenvalue.

[0012] Furthermore, the feature weighting module is further configured as follows: The local time-frequency features are globally averaged pooled on dimensions W and H to obtain the global features of each channel. The calculation formula is: ; In the formula, Y represents the feature output after global average pooling, Represents the local feature value of the lth layer at a specific channel c, position (i, j). i Represents the horizontal index of the feature map (width direction), j Represents the index of the feature map in the vertical direction (height direction); Fast convolution with ECA kernel size k is used to obtain local cross-channel interactions, where k is adaptively determined by the channel dimension C of the input feature map: ; In the formula, C represents the number of input channels, and b are hyperparameters that control the size of the convolution. odd Indicates that the calculation result is rounded down to the nearest odd number. Represents an adaptive function that dynamically adjusts the convolution kernel size based on the number of input channels; Convolution is performed on the channels after global average pooling to calculate the attention weights of different channels : ; In the formula, represents the Sigmoid activation function, which is used to map cross-channel interactions to channel-wise attention weights. conv1D represents a one-dimensional convolution operation, y Represents the channel vector after global average pooling; The local time-frequency features are used as input feature maps, and a weighted feature map is obtained based on the input feature maps and attention weights of different channels.

[0013] Furthermore, the Transformer module includes a self-attention module, a multi-head attention module, a normalization module and a feedforward neural network module, and there is a normalization layer before the self-attention module and after the multi-head self-attention module; The calculation formula of the self-attention module is as follows: ; In the formula, Attention ( Q , K , V ) represents the output of the attention mechanism, Q , K and V Respectively represent the query vector, key vector, and value vector obtained through different linear transformations, is a normalization factor, represents the dimension of the key vector, and T represents the number of time steps; On the basis of the self-attention module, the multi-head attention module is used to capture the features of different representation subspaces. The calculation formula of the multi-head attention module is: ; In the formula, MultiHead ( Q , K , V ) represents the output result of the multi-head attention mechanism, Concat represents stitching together multiple attention representations, W o is the output weight matrix for linear transformation, Head 1. Head 2 and headn Represents the attention output of each head in the multi-head attention mechanism; The output of the multi-head attention module is layer normalized to maintain numerical stability. The formula for layer normalization is: ; In the formula, Y LN Represents the output features after layer normalization. Y MHA represents the output features obtained by the multi-head attention module, Represents the features input to the transformer module, LayerNorm Representation layer normalization operation; The calculation formula of the feedforward neural network module is: ; In the formula, RFFN represents the output features of the feedforward neural network module, Conv represents the convolution operation, DWConv Represents depthwise convolution.

[0014] Furthermore, the calculation formula of the Softmax layer is: ; In the formula, Soft max( z i ) represents the input value z i The normalized result after Softmax transformation, z i represents the i-th component of the input, j Indicates that the jth component is currently being processed. K represents the total length of the input vector, Represents the input component z i The index value of Represents the exponential value of the j-th component in the input vector.

[0015] In a second aspect, the present application provides a motion sickness detection system, the system comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the following method: Preprocessing and data enhancement are performed on the EEG signal to obtain a preprocessed EEG signal; The convolution module is used to upsample and downsample in the time-frequency dimension to obtain the local time-frequency features of the preprocessed EEG signal; Performing feature weighting on the local time-frequency features to obtain a feature weighted graph; Inputting the feature weighted graph into a Transformer module to capture global features to obtain global features; Based on at least two set categories of motion sickness, the global features are input into a fully connected layer as data to be classified, and finally pass through a Softmax layer to calculate the prediction probabilities of multiple categories.

[0016] Furthermore, the EEG signal is preprocessed, including: The EEG signal was filtered using a 0.5 Hz high-pass filter to eliminate low-frequency drift, and then a 50.0 Hz low-pass filter was used to remove high-frequency noise components from the data. The filtered data was subjected to independent principal component analysis to remove artifacts and obtain a continuous signal; wherein the artifacts include eye movements, blinking, electromyography and electrocardiography.

[0017] In a third aspect, the present application provides a non-transitory computer-readable storage medium storing instructions. When the instructions are executed by a processor, the following steps are performed: Preprocessing and data enhancement are performed on the EEG signal to obtain a preprocessed EEG signal; The convolution module is used to upsample and downsample in the time-frequency dimension to obtain the local time-frequency features of the preprocessed EEG signal; Performing feature weighting on the local time-frequency features to obtain a feature weighted graph; Inputting the feature weighted graph into a Transformer module to capture global features to obtain global features; Based on at least two set categories of motion sickness, the global features are input into a fully connected layer as data to be classified, and finally pass through a Softmax layer to calculate the prediction probabilities of multiple categories.

[0018] This application has at least the following beneficial effects: 1. Use sliding windows with different overlaps for segment extraction. Set it to 75% before the onset of motion sickness to capture subtle trends and potential warning signals and avoid missing key features. Especially in the period close to the onset of the attack, a higher overlap rate can ensure that more sample segments are covered and identify abnormal features before the onset of the attack. The overlap setting is reduced to 50% during the onset of the attack, which helps to reduce computational redundancy and improve processing efficiency.

[0019] 2. Use dual channels to process EEG signals. Among them, short-time Fourier transform (STFT) uses a sliding window to extract the time-frequency components of the signal. However, STFT has a limitation on the window size and is suitable for signals whose frequency does not change over time. Continuous wavelet transform (CWT) decomposes the signal into a set of frequency components and presents their distribution in the time domain and spectrum domain by compressing, scaling, and shifting the signal. Since all frequencies become apparent in CWT, it has certain advantages. Dual-channel processing combines the two methods to make the results more accurate.

[0020] 3. The Efficient Channel Attention (ECA) module is introduced to weight feature channels. Different weights are assigned to different feature channels through lightweight convolution operations. The importance of feature channels is automatically screened and emphasized, so that the model can more accurately focus on feature channels that have a greater impact on motion sickness detection, ensuring that the extraction process is more targeted, reducing the interference of redundant features on detection results, and improving detection accuracy.

[0021] 4. A bottleneck structure is introduced into the Transformer module for computational optimization. By reducing the computational dimension to improve efficiency while maintaining sufficient feature expression capabilities, the model can capture the temporal changes and global dependencies of EEG features in a more efficient way while keeping the computation lightweight. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A structural diagram of a motion sickness detection device is proposed for an embodiment of the present application; in the figure, 100 is a signal preprocessing module, 200 is a feature extraction module, 300 is a feature weighting module, 400 is a global feature capture module, and 500 is a classification detection module.

[0023] Figure 2 This is a data processing flow chart of the motion sickness detection device provided in an embodiment of the present application; in the figure, CWT represents continuous wavelet transform and STFT represents short-time Fourier transform.

[0024] Figure 3 A structural diagram of the CNN-ECA-Transformer model provided in an embodiment of the present application; in the figure, 3×3ConV represents 3×3 convolution, 1×1ConV represents 1×1 convolution, ReLU BN represents using ReLU first and then BN, ReLU is an activation function, BN is a normalization operation on the data, SA represents a self-attention layer, RFFN represents a feedforward neural network module, and MHSA represents a multi-head self-attention mechanism.

[0025] Figure 4 This is a structural diagram of the ECA module provided in an embodiment of the present application. In the figure, H and W represent dimensions, C represents the number of channels, GAP represents global average pooling, and Sigmoid represents the Sigmoid function, which is used to map variables to between 0 and 1.

[0026] Figure 5 This is a flow chart of a motion sickness detection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0028] The specific implementation of the present application is further described in detail below in conjunction with the drawings and examples.

[0029] The present application embodiment provides a motion sickness detection device. Figure 1 , is a structural diagram of a motion sickness detection device provided in an embodiment of the present application. The device includes the following components connected in sequence: a signal preprocessing module 100, a feature extraction module 200, a feature weighting module 300, a global feature capture module 400 and a classification detection module 500.

[0030] The functions of each component and its specific definition are described in detail below.

[0031] The signal preprocessing module 100 is configured to perform preprocessing and data enhancement on the EEG signal to obtain a preprocessed EEG signal.

[0032] In some embodiments, the preprocessed EEG signal includes a first preprocessed EEG signal and a second preprocessed EEG signal. Figure 2 As shown, it is a data processing flow chart of the motion sickness detection device provided by the embodiment of the present application. The collected EEG signal (i.e., electroencephalogram signal) is processed by CWT (continuous wavelet transform) and STFT (short-time Fourier transform) to obtain a first pre-processed electroencephalogram signal and a second pre-processed electroencephalogram signal, which are sent to the CNN-ECA-Transformer model for processing, wherein the CNN-ECA-Transformer model includes a CNN module, an ECA module, and a Transformer module, and the data processing flow of the CNN-ECA-Transformer model is a feature extraction module 200, a feature weighting module 300, and a global feature capture module 400. The global features obtained by the processing corresponding to the two pre-processed electroencephalogram signals are processed in the fully connected layer and then classified by the Softmax layer to obtain four categories of motion sickness, and the four categories of motion sickness correspond to four symptom intensities, namely, "none", "mild", "moderate" and "severe".

[0033] Specifically, the collected EEG signals were preprocessed and data enhanced. First, the EEG signals were filtered and artifacts were removed. The recorded EEG data were filtered using the EEGLAB tool in MATLAB. A 0.5Hz high-pass filter was used to eliminate low-frequency drift, and a 50.0Hz low-pass filter was used to remove high-frequency noise components from the data. The filtered data was then subjected to independent principal component analysis (ICA) to remove artifacts such as eye movements, blinking, electromyography, and electrocardiography.

[0034] On this basis, an overlapping sliding window with a length of 10 seconds is used to extract EEG segments from the continuous signal, and the overlapping degree of the sliding window is set to 50% during the period of motion sickness and 75% before the onset. Then, short-time Fourier transform (STFT) is used to extract time-frequency features as the input of the model, and the original EEG signal data volume is expanded to facilitate later training.

[0035] The short-time Fourier transform formula is: ; In the formula, represents the original signal, represents the moving window function, is the kernel function of Fourier transform, is an imaginary unit, is the angular frequency, represents the result of short-time Fourier transform, represents a certain time point of the signal, represents the current time position in the signal, Represents the signal frequency component.

[0036] At the same time, the EEG signal after ICA processing is processed using continuous wavelet transform (CWT). CWT provides detailed features at different frequencies and time points by decomposing the signal into wavelet coefficients at different scales.

[0037] The formula for continuous wavelet transform is: ; In the formula, represents the result of continuous wavelet transform, represents the continuous signal to be analyzed, represents the mother wavelet function, represents the time offset, Represents the scale parameter.

[0038] The mother wavelet is the Morlet wavelet, which can extract features with equal variance in frequency and time and is widely used in the field of neural signal analysis. The Morlet wavelet is composed of a sine wave modulated by a Gaussian function and can capture local oscillation components in the time domain. The formula is: .

[0039] The feature extraction module 200 is configured to obtain local time-frequency features of the preprocessed EEG signal by upsampling and downsampling in the time-frequency dimension using a convolution module.

[0040] In some embodiments, the feature extraction module 200 uses a convolutional neural network to obtain local time-frequency features of the preprocessed EEG signal.

[0041] In deep learning, Convolutional Neural Networks (CNN) is one of the most widely used typical classifiers in the field of images and sequences. The typical convolutional neural network architecture mainly includes the following important components: convolution layer, activation function layer, pooling layer, fully connected layer and output layer. The main function of the convolution layer is to extract features from the original input. Usually, multiple convolution layers are used to continuously extract features. The feature level is getting higher and higher, and it is relatively more reliable.

[0042] like Figure 3 As shown, it is a structural diagram of the CNN-ECA-Transformer model provided in the embodiment of the present application. The feature extraction module 200 uses the CNN module in the CNN-ECA-Transformer model to upsample and downsample in the time-frequency dimension to capture the local time-frequency features of the electroencephalogram (EEG) signal, and further shortens the length of the time series. The CNN module includes multiple deep convolution layers to extract feature information of different resolutions through layer-by-layer convolution. The specific steps are: The processed EEG signal is convolved for a total of 3 convolutional layers. The kernel size of each layer is 3×3, the stride is 1, and the padding is 1. Each convolutional layer is followed by a ReLU activation layer and a Batch Normalization (BN) layer.

[0043] The overall calculation formula is as follows: ; In the formula, Indicates l The convolutional layer j Output feature maps, represents the convolution operation, Indicates l The convolutional layer j Deviations, Represents the input of the current convolution calculation, i represents the convolutional layer i Input feature maps, M j Indicates j The set of input feature map indices associated with the output feature maps, Indicates l In the convolutional layer, i Input feature map to j The convolution kernel of the output feature map.

[0044] That is,l Layer, for the j Output feature map , using the convolution kernel from the i-th input feature map of the previous layer Perform convolution and add bias .

[0045] Output feature map after convolution The output obtained after ReLU activation function is: ; In the formula, It represents the value of the output feature map of the convolutional layer after being processed by the activation function. Represents the activation function pair The processing result, Re LU represents the activation function and max represents the maximum value function.

[0046] Output Then the batch normalization is performed through the BN layer, that is, the mean and variance are calculated and then standardized and linearly transformed, and we have: ; In the formula, m represents the current batch data size, represents the mean of all samples in the batch, represents the variance of the batch samples, ε represents a preset positive number to prevent division by zero, , represents the learnable parameters of the batch normalization process, Represents the local time-frequency feature results of the output, Represents the normalized eigenvalue.

[0047] The feature weighting module 300 is configured to perform feature weighting on the local time-frequency features to obtain a feature weighting graph.

[0048] In order to enhance the representation ability of the network and make the data features more distinguishable, the feature weighting module 300 uses the ECA module to perform feature weighting on the local time-frequency features to obtain a feature weighting graph. The structure of the ECA module is as follows: Figure 4 As shown, the ECA module generates channel attention through fast 1D convolution, and its kernel size can be adaptively determined by the nonlinear mapping of the channel dimension.

[0049] Specifically, the feature weighting module 300 is configured to perform feature weighting on the local time-frequency features through the following steps to obtain a feature weighting graph: Step 1: First, perform global average pooling on the dimensions W and H on the output after convolution of the CNN module to obtain the global features of each channel. The specific formula is: ; In the formula, Y represents the feature output after global average pooling, Represents the local feature value of the lth layer at a specific channel c, position (i, j). i Represents the horizontal index of the feature map (width direction), j Indicates the index of the feature map in the vertical direction (height direction).

[0050] Step 2: Use fast convolution with ECA kernel size k to obtain local cross-channel interactions, where k is adaptively determined by the channel dimension C of the input feature map: ; Where C represents the number of input channels, and b are hyperparameters that control the size of the convolution, usually =2, b=1, odd Indicates that the calculation result is rounded down to the nearest odd number. Represents an adaptive function that dynamically adjusts the convolution kernel size based on the number of input channels C.

[0051] Step 3: Convolve the channels after global average pooling to calculate the attention weights of different channels: ; In the formula, represents the Sigmoid activation function, which maps cross-channel interactions to the attention weights in the channel direction. conv1D represents a one-dimensional convolution operation, y Represents the channel vector after global average pooling. The weighted feature map is then obtained by calculating between the input feature map and the channel weight.

[0052] The global feature capture module 400 is configured to input the feature weighted map into a Transformer module to capture global features and obtain global features.

[0053] The structure of the Transformer module is as follows Figure 3 As shown, the global feature capture module 400 further inputs the output result after ECA processing into the Transformer module to capture global features, so as to make up for the shortcomings of local features extracted by CNN. The Transformer module includes four parts: self-attention module, multi-head attention module, normalization module and feedforward neural network module. There are normalization layers before the self-attention module and after the multi-head self-attention module.

[0054] The formula of the self-attention module is as follows: ; In the formula, Attention ( Q , K , V ) represents the output of the attention mechanism, Q, K, and V represent the query vector (Query), key vector (Key), and value vector (Value) obtained through different linear transformations, respectively. is a normalization factor, where represents the dimension of the key vector and T represents the number of time steps.

[0055] On the basis of the self-attention module, a multi-head attention module is used to capture the features of different representation subspaces. The calculation formula of the multi-head attention mechanism is: ; In the formula, MultiHead ( Q , K , V ) represents the output of the multi-head attention mechanism, and Concat represents stitching multiple attention representations together; the calculation of each Head is similar to the ordinary self-attention mechanism, but the parameters are independent; W o is the output weight matrix for linear transformation, Head 1. Head 2 and headn Represents the attention output of each head in the multi-head attention mechanism.

[0056] The output of the multi-head attention module is then layer normalized to maintain numerical stability. The formula for layer normalization is: ; In the formula, Y LN Represents the output features after layer normalization, Y MHA represents the output features obtained by the multi-head attention module, is the feature input to the transformer module. The normalization layer here contains a residual connection, which retains the original feature information of the input and reduces information loss.

[0057] The feedforward neural network module uses a bottleneck structure and residual connection to further express features. The structure consists of three convolutional layers. The convolution kernels of the first and third convolutional layers are 1×1, the stride is 1, and the padding is zero. The convolution kernel of the second convolutional layer is 3×3, the stride is 1, and the padding is 1. Each convolutional layer includes a BN layer, and the activation function ReLU is only applied to the first two layers.

[0058] The specific formula is: ; In the formula, RFFN represents the output features of the feedforward neural network module, Conv represents the convolution operation, DWConv Represents depthwise convolution.

[0059] The classification detection module 500 is configured to input the global features as data to be classified into a fully connected layer based on at least two set motion sickness categories, and finally pass through a Softmax layer to calculate the prediction probabilities of multiple categories.

[0060] In this embodiment, the classification detection module 500 is used to integrate the processed feature data and perform classification processing. The classification is divided into four levels of motion sickness detection: none, mild, moderate and severe based on the SSQ score level. The feature data to be classified is input into the fully connected layer and finally directly input into the Softmax layer to calculate the prediction probabilities of the four categories. The calculation formula of Softmax is: ; In the formula, Soft max( z i ) represents the input value z i The normalized result after Softmax transformation, z i represents the i-th component of the input, j Indicates that the jth component is currently being processed. K represents the total length of the input vector, Represents the input component z i The index value of Represents the exponential value of the j-th component in the input vector.

[0061] The present application also provides a motion sickness detection system, the system comprising: Memory for storing computer programs; A processor is used to execute the computer program to implement the following method: Preprocessing and data enhancement are performed on the EEG signal to obtain a preprocessed EEG signal; The convolution module is used to upsample and downsample in the time-frequency dimension to obtain the local time-frequency features of the preprocessed EEG signal; Performing feature weighting on the local time-frequency features to obtain a feature weighted graph; Inputting the feature weighted graph into a Transformer module to capture global features to obtain global features; Based on at least two set categories of motion sickness, the global features are input into a fully connected layer as data to be classified, and finally pass through a Softmax layer to calculate the prediction probabilities of multiple categories.

[0062] In some embodiments, the EEG signal is pre-processed, including: The EEG signal was filtered using a 0.5 Hz high-pass filter to eliminate low-frequency drift, and then a 50.0 Hz low-pass filter was used to remove high-frequency noise components from the data. The filtered data was subjected to independent principal component analysis to remove artifacts and obtain a continuous signal; wherein the artifacts include eye movements, blinking, electromyography and electrocardiography.

[0063] In some embodiments, the EEG signal is preprocessed and data enhanced to obtain the preprocessed EEG signal, including: The EEG segments were extracted from the continuous signal using overlapping sliding windows of set length, with the overlapping of sliding windows set to 50% during the motion sickness episode and 75% before the episode. Then, short-time Fourier transform was used to extract the time-frequency features as the first preprocessed EEG signal. The formula for the short-time Fourier transform is: ; In the formula, represents the original signal, represents the moving window function, is the kernel function of Fourier transform, is an imaginary unit, is the angular frequency, represents the result of short-time Fourier transform, represents a certain time point of the signal, represents the current time position in the signal, Represents the signal frequency component; Based on the continuous signal, continuous wavelet transform is used to perform data processing to decompose the continuous signal into wavelet coefficients at different scales, so as to obtain detailed features at different frequencies and time points as the second preprocessed EEG signal; The formula of the continuous wavelet transform is: ; In the formula, represents the result of continuous wavelet transform, represents the continuous signal to be analyzed, represents the mother wavelet function, represents the time offset, Represents the scale parameter.

[0064] In some embodiments, the convolution module is used to upsample and downsample in the time-frequency dimension to obtain the local time-frequency features of the preprocessed EEG signal, including: The convolution operation is performed on the preprocessed EEG signal, and the calculation formula is as follows: ; In the formula, Indicates l The convolutional layer j Output feature maps, represents the convolution operation, Indicates l The convolutional layer j Deviations, Represents the input of the current convolution calculation, i represents the convolutional layer i Input feature maps, M j Indicates j The set of input feature map indices associated with the output feature maps, Represents the convolution kernel from the i-th input feature map to the j-th output feature map in the l-th convolution layer; The output feature map after convolution is processed by the activation function and the output is: ; In the formula, It represents the value of the output feature map of the convolutional layer after being processed by the activation function. Represents the activation function pair The processing result, Re LU represents the activation function, and max represents the maximum value function;

[0065] Output Then, batch normalization is performed through the BN layer, that is, the mean and variance are calculated and then standardized and linearly transformed to obtain the local time-frequency characteristics of the preprocessed EEG signal. The calculation process is expressed as: ; In the formula, m represents the current batch data size, represents the mean of all samples in the batch, represents the variance of the batch samples, ε represents a preset positive number to prevent division by zero, , represents the learnable parameters of the batch normalization process, Represents the local time-frequency feature results of the output, Represents the normalized eigenvalue.

[0066] In some embodiments, the feature weighted graph is input into a Transformer module to capture global features to obtain global features, including: The local time-frequency features are globally averaged pooled on dimensions W and H to obtain the global features of each channel. The calculation formula is: ; In the formula, Y represents the feature output after global average pooling, Represents the local feature value of the lth layer at a specific channel c, position (i, j). i Represents the index of the feature map in the horizontal direction, j Represents the index of the feature map in the vertical direction; Fast convolution with ECA kernel size k is used to obtain local cross-channel interactions, where k is adaptively determined by the channel dimension C of the input feature map: ; Where C represents the number of input channels, and b are hyperparameters that control the size of the convolution. odd Indicates that the calculation result is rounded down to the nearest odd number. Represents an adaptive function that dynamically adjusts the convolution kernel size based on the number of input channels; Convolution is performed on the channels after global average pooling to calculate the attention weights of different channels : ; In the formula, represents the Sigmoid activation function, which is used to map cross-channel interactions to channel-wise attention weights. conv1D represents a one-dimensional convolution operation, y Represents the channel vector after global average pooling; The local time-frequency features are used as input feature maps, and a weighted feature map is obtained based on the input feature maps and attention weights of different channels.

[0067] In some embodiments, the Transformer module includes a self-attention module, a multi-head attention module, a normalization module, and a feed-forward neural network module, and there is a normalization layer before the self-attention module and after the multi-head self-attention module; The calculation formula of the self-attention module is as follows: ; In the formula, Attention ( Q , K , V ) represents the output of the attention mechanism, Q , K andV Respectively represent the query vector, key vector, and value vector obtained through different linear transformations, is a normalization factor, represents the dimension of the key vector, and T represents the number of time steps; On the basis of the self-attention module, the multi-head attention module is used to capture the features of different representation subspaces. The calculation formula of the multi-head attention module is: ; In the formula, MultiHead ( Q , K , V ) represents the output result of the multi-head attention mechanism, Concat represents stitching together multiple attention representations, W o is the output weight matrix for linear transformation, Head 1. Head 2 and headn Represents the attention output of each head in the multi-head attention mechanism; The output of the multi-head attention module is layer normalized to maintain numerical stability. The formula for layer normalization is: ; In the formula, Y LN Represents the output features after layer normalization. Y MHA represents the output features obtained by the multi-head attention module, Represents the features input to the transformer module, LayerNorm Representation layer normalization operation; The calculation formula of the feedforward neural network module is: ; In the formula, RFFN represents the output features of the feedforward neural network module, Conv represents the convolution operation, DWConv Represents depthwise convolution.

[0068] In some embodiments, the calculation formula of the Softmax layer is: ; In the formula, Soft max( z i ) represents the input value z i The normalized result after Softmax transformation, zi represents the i-th component of the input, j Indicates that the jth component is currently being processed. K represents the total length of the input vector, Represents the input component z i The index value of Represents the exponential value of the j-th component in the input vector.

[0069] The present application also provides a non-transitory computer-readable storage medium storing instructions. When the instructions are executed by a processor, Figure 5 As shown, perform the following steps: S1, preprocessing and data enhancement of the EEG signal to obtain a preprocessed EEG signal; S2, using the convolution module to upsample and downsample in the time-frequency dimension to obtain the local time-frequency features of the preprocessed EEG signal; S3, performing feature weighting on the local time-frequency features to obtain a feature weighted graph; S4, inputting the feature weighted graph into the Transformer module to capture the global features and obtain the global features; S5. Based on at least two set categories of motion sickness, the global features are input into a fully connected layer as data to be classified, and finally passed through a Softmax layer to calculate the prediction probabilities of multiple categories.

[0070] It should be noted that the motion sickness detection system and the non-transitory computer-readable storage medium storing instructions proposed in the embodiments of the present application belong to the same technical concept as the previous motion sickness detection device, have the same technical principles and can achieve the same beneficial effects, which will not be repeated here.

[0071] The above implementation modes are only used to illustrate the present application, and are not intended to limit the present application. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present application. Therefore, all equivalent technical solutions also belong to the scope of the present application, and the scope of patent protection of the present application shall be limited by the claims.

Claims

1. A motion sickness detection device, characterized in that: The device comprises a signal preprocessing module, a feature extraction module, a feature weighting module, a global feature capture module and a classification detection module which are connected in sequence, wherein: The signal preprocessing module is configured to preprocess and data enhance the EEG signal to obtain a preprocessed EEG signal; The feature extraction module is configured to obtain local time-frequency features of the preprocessed EEG signal by upsampling in the time-frequency dimension using the convolution module; The feature weighting module is configured to perform feature weighting on the local time-frequency features to obtain a feature weighted graph; The global feature capture module is configured to input the feature weighted map into the Transformer module to capture the global feature and obtain the global feature; The classification detection module is configured to input the global features as data to be classified into a fully connected layer based on at least two set motion sickness categories, and finally pass through a Softmax layer to calculate the prediction probabilities of multiple categories.

2. The motion sickness detection device according to claim 1, wherein: The signal preprocessing module is further configured to: The EEG signal was filtered using a 0.5 Hz high-pass filter to eliminate low-frequency drift, and then a 50.0 Hz low-pass filter was used to remove high-frequency noise components from the data. The filtered data was subjected to independent principal component analysis to remove artifacts and obtain a continuous signal; wherein the artifacts include eye movements, blinking, electromyography and electrocardiography.

3. The motion sickness detection device according to claim 2, characterized in that: The preprocessed EEG signal includes a first preprocessed EEG signal and a second preprocessed EEG signal, and the signal preprocessing module is further configured as follows: The EEG segments were extracted from the continuous signal using overlapping sliding windows of set length, with the overlapping of sliding windows set to 50% during the motion sickness episode and 75% before the episode. Then, short-time Fourier transform was used to extract the time-frequency features as the first preprocessed EEG signal. The formula for the short-time Fourier transform is: ； In the formula, represents the original signal, represents the moving window function, is the kernel function of Fourier transform, is an imaginary unit, is the angular frequency, represents the result of short-time Fourier transform, represents a certain time point of the signal, represents the current time position in the signal, Represents the signal frequency component; Based on the continuous signal, continuous wavelet transform is used to perform data processing to decompose the continuous signal into wavelet coefficients at different scales, so as to obtain detailed features at different frequencies and time points as the second preprocessed EEG signal; The formula of the continuous wavelet transform is: ； In the formula, represents the result of continuous wavelet transform, represents the continuous signal to be analyzed, represents the mother wavelet function, represents the time offset, Represents the scale parameter.

4. The motion sickness detection device according to claim 1, wherein: The feature extraction module is configured to be further configured to: The convolution operation is performed on the preprocessed EEG signal, and the calculation formula is as follows: ； In the formula, Indicates l The convolutional layer j Output feature maps, i.e., eigenvalues ​​after convolution calculation; represents the convolution operation, Indicates l The convolutional layer j Deviations, Represents the input of the current convolution calculation, i represents the convolutional layer i Input feature maps, M j Indicates j The set of input feature map indices associated with the output feature maps, Indicates l In the convolutional layer, i Input feature map to j The convolution kernel of the output feature map; The output feature map after convolution is processed by the activation function and the output is: ； In the formula, It represents the value of the output feature map of the convolutional layer after being processed by the activation function. Represents the activation function pair The processing result, Re LU represents the activation function, and max represents the maximum value function; Output Then, batch normalization is performed through the BN layer, that is, the mean and variance are calculated and then standardized and linearly transformed to obtain the local time-frequency characteristics of the preprocessed EEG signal. The calculation process is expressed as: ； In the formula, m represents the amount of data in the current batch. represents the mean of all samples in the batch, represents the variance of the batch samples, ε represents a preset positive number to prevent division by zero, , represents the learnable parameters of the batch normalization process, Represents the local time-frequency feature results of the output, Represents the normalized eigenvalue.

5. The motion sickness detection device according to claim 1, wherein: The feature weighting module is further configured as follows: The local time-frequency features are globally averaged pooled on dimensions W and H to obtain the global features of each channel. The calculation formula is: ； In the formula, Y represents the feature output after global average pooling, Represents the local eigenvalue of the lth layer at a specific channel c, position (i, j); i Represents the index of the feature map in the horizontal direction, j Represents the index of the feature map in the vertical direction; Fast convolution with ECA kernel size k is used to obtain local cross-channel interactions, where k is adaptively determined by the channel dimension C of the input feature map: ； In the formula, C represents the number of input channels, and b are hyperparameters that control the size of the convolution. odd Indicates that the calculation result is rounded down to the nearest odd number. Represents an adaptive function that dynamically adjusts the convolution kernel size based on the number of input channels; Convolution is performed on the channels after global average pooling to calculate the attention weights of different channels : ； In the formula, represents the Sigmoid activation function, which is used to map cross-channel interactions to channel-wise attention weights. conv1D represents a one-dimensional convolution operation, y Represents the channel vector after global average pooling; The local time-frequency features are used as input feature maps, and a weighted feature map is obtained based on the input feature maps and attention weights of different channels.

6. The motion sickness detection device according to claim 1, wherein: The Transformer module includes a self-attention module, a multi-head attention module, a normalization module and a feedforward neural network module, and there is a normalization layer before the self-attention module and after the multi-head self-attention module; The calculation formula of the self-attention module is as follows: ； In the formula, Attention ( Q , K , V ) represents the output of the attention mechanism, Q , K and V Respectively represent the query vector, key vector, and value vector obtained through different linear transformations, is a normalization factor, represents the dimension of the key vector, and T represents the number of time steps; On the basis of the self-attention module, the multi-head attention module is used to capture the features of different representation subspaces. The calculation formula of the multi-head attention module is: ； In the formula, MultiHead ( Q , K , V ) represents the output result of the multi-head attention mechanism, Concat represents stitching together multiple attention representations, W o is the output weight matrix for linear transformation, Head 1. Head 2 and headn Represents the attention output of each head in the multi-head attention mechanism; The output of the multi-head attention module is layer normalized to maintain numerical stability. The formula for layer normalization is: ； In the formula, Y LN represents the output features after layer normalization, Y MHA represents the output features obtained by the multi-head attention module, Represents the features input to the transformer module, LayerNorm Representation layer normalization operation; The calculation formula of the feedforward neural network module is: ； In the formula, RFFN represents the output features of the feedforward neural network module, Conv represents the convolution operation, DWConv Represents depthwise convolution.

7. The motion sickness detection device according to claim 1, wherein: The calculation formula of the Softmax layer is: ； In the formula, Soft max( z i ) represents the input value z i The normalized result after Softmax transformation, z i represents the i-th component of the input, j Indicates that the jth component is currently being processed. K represents the total length of the input vector, Represents the input component z i The index value of Represents the exponential value of the j-th component in the input vector.

8. A motion sickness detection system, characterized in that: The system comprises: Memory for storing computer programs; A processor is used to execute the computer program to implement the following method: Preprocessing and data enhancement are performed on the EEG signal to obtain a preprocessed EEG signal; The convolution module is used to upsample and downsample in the time-frequency dimension to obtain the local time-frequency features of the preprocessed EEG signal; Performing feature weighting on the local time-frequency features to obtain a feature weighted graph; Inputting the feature weighted graph into a Transformer module to capture global features to obtain global features; Based on at least two set categories of motion sickness, the global features are input into a fully connected layer as data to be classified, and finally pass through a Softmax layer to calculate the prediction probabilities of multiple categories.

9. The motion sickness detection system according to claim 8, wherein: Preprocess the EEG signals, including: The EEG signal was filtered using a 0.5 Hz high-pass filter to eliminate low-frequency drift, and then a 50.0 Hz low-pass filter was used to remove high-frequency noise components from the data. The filtered data was subjected to independent principal component analysis to remove artifacts and obtain a continuous signal; wherein the artifacts include eye movements, blinking, electromyography and electrocardiography.

10. A non-transitory computer-readable storage medium storing instructions, characterized in that: When the instruction is executed by the processor, the following steps are performed: Preprocessing and data enhancement are performed on the EEG signal to obtain a preprocessed EEG signal; The convolution module is used to upsample and downsample in the time-frequency dimension to obtain the local time-frequency features of the preprocessed EEG signal; Performing feature weighting on the local time-frequency features to obtain a feature weighted graph; Inputting the feature weighted graph into a Transformer module to capture global features to obtain global features; Based on at least two set categories of motion sickness, the global features are input into a fully connected layer as data to be classified, and finally pass through a Softmax layer to calculate the prediction probabilities of multiple categories.

Images