Virtual reality driven Stroop mild cognitive impairment detection method based on eye movement signals

Abstract

The invention relates to an eye movement signal-based virtual reality driven Stroop mild cognitive impairment detection method, which comprises the following steps of: (1) extracting eye movement signal data acquired by a virtual reality integrated eye movement sensor in a plurality of tasks, and preprocessing the eye movement signal data; (2) carrying out sample division on the preprocessed data to construct a database; (3) respectively dividing different training sets and verification sets according to strategies of cross validation and blind test experiments of one testee; (4) wavelet scattering features are extracted from the data through wavelet scattering transformation; (5) inputting the features into a CNN network combined with time attention for training and learning, and sending the features into a full-connection layer to predict a result; and (6) integrating prediction results corresponding to different tasks to obtain a final result, and evaluating the final result. The method can effectively detect people with mild cognitive impairment, and reduces the time cost of professional medical staff and the economic cost of detection while increasing the immersion.

Description

Technical Field

[0001] This invention belongs to the biomedical field, specifically relating to a multi-cognitive assessment task based on eye-tracking signals and a research method for feature extraction of signals and detection of patients with mild cognitive impairment using a network based on wavelet scattering transform, attention, CNN and ensemble strategies. Background Technology

[0002] Mild cognitive impairment (MCI) is an intermediate stage between normal cognition and Alzheimer's disease (AD), with a high probability of progressing to AD. With the aging population, the prevalence of MCI among the elderly in China is increasing year by year, and MCI will progress to dementia to varying degrees. Studies predict that by 2030, the number of people with dementia in China will reach 23.3 million, and the total cost of dementia is expected to reach US$114.2 billion, placing a heavy burden on families and society. MCI is a crucial intervention point for AD prevention and control; intervention targeting the MCI stage may be the most effective strategy to delay the onset of AD. Therefore, differentiating patients with MCI helps delay the onset of AD and enables earlier treatment.

[0003] In recent years, studies have used various biomarkers, such as β-amyloid 42 (Aβ42), Tau protein, and phosphorylated Tau (p-tau), to detect patients with mild cognitive impairment. However, the extraction of these biomarkers is an invasive process for patients. There are also methods that use computers to perform cognitive tests and analyze data such as images using machine learning or artificial intelligence models. Although these methods are non-invasive, they lack an immersive environment. Virtual reality devices that integrate eye-tracking sensors can collect eye-tracking signals in a non-invasive manner, providing more immersive tasks and reducing discomfort during the assessment process.

[0004] Eye-tracking signals acquired using eye-tracking sensors in virtual reality devices are complex signals that are non-stationary and non-periodic, possessing high temporal and spatial resolution. However, eye-tracking signals are easily confused. Wavelet scattering can effectively extract discriminative time-frequency features related to mild cognitive impairment. By utilizing attention mechanisms, combining CNN with wavelet scattering transform, and employing an ensemble strategy for multi-task analysis to make the final prediction, it is possible to effectively distinguish between patients with mild cognitive impairment and healthy controls (HC).

[0005] Therefore, using multiple cognitive assessment tasks based on the virtual reality Stroop test, and leveraging networks based on wavelet scattering transform, attention, CNN, and ensemble strategies for feature extraction and classification, can effectively distinguish between people with mild cognitive impairment and healthy controls. Summary of the Invention

[0006] Purpose of the invention: In order to effectively screen people with mild cognitive impairment, this invention provides a method for detecting people with mild cognitive impairment based on physiological signals measured by virtual reality multi-cognitive assessment tasks. This method identifies the similarities and differences between eye movement signals of two different cognitive states based on the characteristics such as non-stationarity and non-periodicity of eye movement signals, thereby distinguishing patients with mild cognitive impairment from healthy controls, and can be used to assist in disease screening.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows: a virtual reality-driven Stroop mild cognitive impairment detection method based on eye-tracking signals, the method comprising the following steps:

[0008] Step (1) Use the four tasks of the virtual reality-based Stroop test to collect eye movement signal data during the four tasks, perform preprocessing, and obtain the eye movement datasets of the four tasks.

[0009] Step (1-1) involves performing confidence-based filtering, linear interpolation to compensate for blink-related signal loss, edge noise trimming, and high-pass filtering (cutoff frequency: 1 Hz) on the eye-tracking dataset for each task to remove low-frequency drift. This method yields a relatively clean signal.

[0010] Step (2) divides the eye-tracking data into samples at fixed time intervals, resulting in each sample having a size of C channels (C=21) and a length of L (L=240), thus increasing the number of samples, as shown in Table 1;

[0011] Step (2-1) To ensure a sufficient number of samples, the ET clean signal of each subject in each task is divided into 2-second intervals using a Hamming window, with a 1-second overlap between each segment.

[0012] Step (3) For the leave-one-subjects cross-validation experiment and the blind test experiment, the training and validation datasets are divided according to different strategies.

[0013] Step (3-1) Leave-one-subject cross-validation: In each round, leave one subject's sample as the validation set and the rest as the training set, repeating this process n-1 times. Leave-one-subject cross-validation can better evaluate the model's generalization ability when there are fewer subjects.

[0014] Step (3-2) involves a blind test where 50% of the participants are randomly selected as the validation set, and the remainder as the training set. Blind tests simulate real-world evaluation and better illustrate the model's performance in practical applications.

[0015] Step (4) Perform wavelet scattering transform on the eye-tracking signal dataset to obtain the wavelet scattering transform feature matrix corresponding to each task;

[0016] Step (4-1) involves performing wavelet scattering transform on the eye-tracking signal of the j-th channel (j=1,…C) of the i-th task (i=A,B,C,D). Perform scattering and calculate its wavelet scattering coefficients from order 0 to 2:

[0017]

[0018] in, They represent The 0th, 1st, and 2nd order scattering coefficients, the 0th order scattering coefficient capture signal The low-frequency component is calculated using the following formula:

[0019]

[0020] in It is a Gaussian low-pass filter with wavelet scaling parameter J (J=3), * denotes convolution operation, and first-order scattering coefficients. Features at a specific frequency scale λ:

[0021]

[0022] Where ψ λ It is a Morlet wavelet with frequency λ, where |·| denotes the modulus operation. Second-order scattering coefficients. Capturing features at two different frequency scales, λ and μ:

[0023]

[0024] Step (4-2) involves concatenating the coefficients of each channel to obtain the wavelet scattering feature matrix. The wavelet scattering feature is the feature obtained after the original eye movement signal has undergone wavelet scattering transformation. It solves the inherent non-stationarity and non-periodicity of the eye movement signal, thus obtaining a more stable feature representation.

[0025] Step (5) inputs the wavelet scattering feature matrix into the CNN network with time attention for training and learning, and then performs feature dimension transformation and feeds it into the fully connected layer to obtain the prediction result for each task.

[0026] Step (5-1) feeds the wavelet scattering feature matrix of the eye movement signal from step (4-2) into the Time Attention (TA) network to further dynamically allocate the weights of the time steps, and obtains the weighted feature matrix.

[0027] Step (5-2) feeds the weighted feature matrix obtained in step (5-1) into the CNN network for training.

[0028] Step (5-3) transforms the feature dimensions of the features from step (5-2) and feeds them into the fully connected layer to obtain the prediction results for each task. Temporal attention can weight the importance of the wavelet scattering transformed features in the time dimension, making the model pay more attention to rapidly changing time steps, which are often accompanied by eye-tracking behavior. 1D CNNs can further extract enhanced local temporal features.

[0029] Step (6) Integrate the prediction results corresponding to different tasks to obtain the final result and evaluate it;

[0030] Step (6-1) integrates and votes the prediction results of the models trained on the datasets corresponding to each task to obtain the final prediction result; this allows for a comprehensive evaluation of the performance of each task, making the overall evaluation effect more accurate.

[0031] Step (6-2) For both leave-one-out crossover and blind tests, the prediction results are evaluated using the average classification accuracy, recall, precision, F1 score, AUC value, and confusion matrix.

[0032] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the aforementioned Stroop mild cognitive impairment detection method based on eye-tracking signals and driven by virtual reality.

[0033] A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the aforementioned Stroop mild cognitive impairment detection method based on eye-tracking signals and driven by virtual reality.

[0034] The beneficial effects of this invention are as follows:

[0035] This invention employs a virtual reality-driven Stroop task to acquire corresponding eye-tracking signals, extracts the wavelet scattering feature matrix of the signals, and then feeds it into a network combining temporal attention and CNN for learning. Finally, an integrated strategy is used to detect mild cognitive impairment. Compared to existing Stroop scales or other cognitive assessment scales, this invention's method offers a stronger sense of immersion and significantly reduces professional training costs and assessment time. Compared to invasive detection methods, this invention's method is non-invasive, reducing subject discomfort and saving economic costs. Compared to existing methods that use eye-tracking features to detect mild cognitive impairment, this invention uses wavelet scattering transform to directly extract features from the preprocessed eye-tracking signals, eliminating the need for manual feature extraction and selection, thus achieving a more automated feature extraction method.

[0036] Table 1. Sample size and total sample size of mild cognitive impairment and healthy controls in the four tasks (A, B, C, D).

[0037] Task Mild cognitive impairment Health comparison total A 996 1151 2147 B 906 1152 2058 C 750 993 1743 D 972 1167 2139 total 3624 4463 8087 Attached Figure Description

[0038] Figure 1 This invention discloses a virtual reality multi-cognitive assessment task.

[0039] Figure 2 This is a flowchart of the method disclosed in this invention;

[0040] Figure 3 This is a partial eye movement signal map of a patient with mild cognitive impairment;

[0041] Figure 4 This is a diagram of the network structure proposed in this invention;

[0042] Figure 5 The confusion matrix results of leave-one-out cross-validation are shown in the figure.

[0043] Figure 6 This is a diagram showing the confusion matrix results of the blind test experiment. Detailed Implementation

[0044] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments.

[0045] Example: To effectively detect individuals with mild cognitive impairment and reduce labor and economic costs, this invention effectively utilizes the virtual reality-driven Stroop task to acquire eye-tracking signals, and analyzes the eye-tracking signals using strategies such as wavelet scattering transform, temporal attention, convolutional neural network models, and ensemble voting.

[0046] like Figure 1As shown, this invention discloses a multi-cognitive assessment task based on the virtual reality-based Stroop test, such as... Figure 2 This is a flowchart for detecting mild cognitive impairment in this invention, specifically including the following steps:

[0047] Step (1) uses four tasks of the virtual reality-based Stroop test to collect eye movement signal data during the four tasks, performs preprocessing, and obtains eye movement datasets for the four tasks. The data source is the Second Affiliated Hospital of Nanjing Medical University; the data comes from 17 patients with mild cognitive impairment and 21 healthy controls. Step (1-1) For the eye movement dataset of each task, confidence-based filtering, linear interpolation to compensate for blink-related signal loss, edge noise trimming, and high-pass filtering (cutoff frequency: 1Hz) are performed to remove low-frequency drift. This method can obtain a relatively clean signal.

[0048] Step (2) divides the eye-tracking data into samples at fixed time intervals, resulting in each sample having a size of C channels (C=21) and a length of L (L=240). The eye-tracking signal data is as follows: Figure 3 As shown in Table 1, the number of samples was increased; in step (2-1), to ensure a sufficient number of samples, the ET clean signal of each subject in each task was divided into 2-second intervals using a Hamming window, with a 1-second overlap between each segment.

[0049] Step (3) For the leave-one-subjects cross-validation experiment and the blind test experiment, the training and validation datasets are divided according to different strategies.

[0050] Step (3-1) Leave-one-subject cross-validation: In each round, leave one subject's sample as the validation set and the rest as the training set, repeating this process n-1 times. Leave-one-subject cross-validation can better evaluate the model's generalization ability when there are fewer subjects.

[0051] Step (3-2) involves a blind test where 50% of the participants are randomly selected as the validation set, and the remainder as the training set. Blind tests simulate real-world evaluation and better illustrate the model's performance in practical applications.

[0052] Step (4) performs wavelet scattering transform on the eye-tracking signal dataset to obtain the wavelet scattering transform feature matrix corresponding to each task, such as... Figure 4 As shown;

[0053] Step (4-1) involves performing wavelet scattering transform on the eye-tracking signal of the j-th channel (j=1,…C) of the i-th task (i=A,B,C,D). Perform scattering and calculate its wavelet scattering coefficients from order 0 to 2:

[0054]

[0055] in, They represent The 0th, 1st, and 2nd order scattering coefficients, the 0th order scattering coefficient capture signal The low-frequency component is calculated using the following formula:

[0056]

[0057] in It is a Gaussian low-pass filter with wavelet scaling parameter J, where * denotes convolution operation and first-order scattering coefficients. Features at a specific frequency scale λ:

[0058]

[0059] Where ψ λ It is a Morlet wavelet with frequency λ, where |·| denotes the modulus operation. Second-order scattering coefficients. Capturing features at two different frequency scales, λ and μ:

[0060]

[0061] Step (4-2) involves concatenating the coefficients of each channel to obtain the wavelet scattering feature matrix;

[0062] Step (5) inputs the wavelet scattering feature matrix into the CNN network with time attention for training and learning, then performs feature dimension transformation and feeds it into the fully connected layer for prediction to obtain the prediction result for each task.

[0063] Step (5-1) feeds the wavelet scattering feature matrix of the eye movement signal from step (4-2) into the Time Attention (TA) network to further dynamically allocate the weights of the time steps, and obtains the weighted feature matrix.

[0064] Step (5-2) feeds the weighted feature matrix obtained in step (5-1) into the CNN network for training.

[0065] Step (5-3) transforms the feature dimension of the features from step (5-2) and feeds them into the fully connected layer for prediction, thus obtaining the prediction results for each task.

[0066] Step (6) Integrate the prediction results corresponding to different tasks to obtain the final result and evaluate it;

[0067] Step (6-1) integrates and votes the prediction results of the models trained on the datasets corresponding to each task to obtain the final prediction results;

[0068] Step (6-2) evaluates the prediction results using average classification accuracy, recall, precision, F1 score, and AUC value. Specific implementation examples:

[0070] This invention aims to use deep learning methods to detect individuals with mild cognitive impairment, and data issues need to be considered. Deep learning often requires a large amount of data for the network model to perform effectively; therefore, data collection and database establishment are crucial.

[0071] Step (1) Eye movement signal data were collected during the four tasks of the Stroop test based on virtual reality. Eye movement data was collected using the eye-tracking sensor built into the virtual reality device. Before collecting eye movement data for each task, the subjects underwent eye-tracking calibration to ensure the accuracy and validity of the eye movement signals. The collected data underwent preprocessing operations such as blink interpolation and drift removal to obtain the eye movement datasets for the four tasks. The data source was the Second Affiliated Hospital of Nanjing Medical University; the data came from 17 patients with mild cognitive impairment and 21 healthy controls.

[0072] In step (2), to increase the amount of data, the preprocessed eye-tracking data is divided into samples according to a fixed time, resulting in each sample having a size of C channels (C=21) and a length of L (L=240), thus increasing the number of samples.

[0073] Step (3) For the leave-one-subject cross-validation experiment and the blind test experiment, the training and validation datasets are divided according to different strategies. In the leave-one-subject cross-validation, one subject's sample is left as the validation set each time, and the rest are used as the training set, and this is repeated n-1 times. In the blind test experiment, 50% of the subjects are randomly selected as the validation set, and the rest are used as the training set.

[0074] Step (4) uses a wavelet scattering transform based on Kymatio with the highest order of 2, Morletwavelet as bandpass filter and Gaussian lowpass filter, wavelet bandwidth of 3, scatters each segment and each channel, each channel obtains 13-dimensional wavelet scattering coefficients, the signal length is downsampled to 30, and the 21 channels are combined to obtain a 273×30 wavelet scattering feature matrix;

[0075] Step (5) involves feeding the extracted wavelet scattering feature matrix of the eye-tracking signal into a temporal attention network to further dynamically allocate the weights of the time steps, resulting in a weighted feature matrix with the dimension remaining unchanged. The weighted feature matrix is ​​then fed into a CNN network for learning. The CNN network can learn local features in different time series patterns and output discriminative features. The fully connected layer then outputs a prediction result of size 2, which represents the number of categories, thus obtaining the prediction vector for each task.

[0076] Step (6): After obtaining the prediction vectors for different tasks, the probability of the corresponding category is calculated through the softmax layer. The probability vectors output by all tasks are integrated and voted to obtain the final prediction result that is robust and reduces task bias.

[0077] In this invention, to address the problem to be solved by the invention, the output channels of the fully connected layer network are set to 2, the number of output channels of the CNN network is set to 256, and the learning rate is set to 0.00001.

[0078] Finally, the learning results of the network are evaluated using accuracy, recall, precision, F1 score, and AUC.

[0079] In this invention, MCI (Mild Cognitive Impairment) represents mild cognitive impairment, HC (Healthy Controls) represents healthy controls, TP (True Positive) represents MCI samples correctly predicted by the model, TN (True Negative) represents HC samples correctly predicted by the model, FP (False Positive) represents HC samples incorrectly predicted by the model, and FN (False Negative) represents mild cognitive impairment samples incorrectly predicted by the model. TP, TN, FP, and FN together form the confusion matrix. The confusion matrices corresponding to the leave-one-out cross-validation experiment and the blind test experiment of this invention are as follows: Figure 5 and Figure 6 As shown.

[0080] Accuracy is defined as the probability that all samples are correctly classified:

[0081]

[0082] Precision rate is the percentage of samples predicted as having mild cognitive impairment that also actually have mild cognitive impairment.

[0083]

[0084] Recall rate is the percentage of samples that actually have mild cognitive impairment but were classified as having mild cognitive impairment.

[0085]

[0086] The F1 score, which considers both precision and recall, is the harmonic mean of precision and recall. It is often used as the final evaluation method for machine learning classification methods; a higher F1 score for each class indicates a better classification result. The F1 score for each class is expressed as follows:

[0087]

[0088] The AUC-ROC curve measures a model's performance across all possible classification thresholds. The ROC curve depicts the relationship between the false positive rate and the true positive rate at different thresholds. A value closer to 1 indicates better model performance.

[0089] Table 2 lists the classification metrics for different tasks. The experimental results in Table 2 show that Task B performs best among all tasks, and the classification performance can be further improved after the integration strategy, which demonstrates the effectiveness of the network proposed in this invention and the use of the integration strategy.

[0090] Table 2 Classification Indicators under Different Tasks and Strategies

[0091]

[0092] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the present invention without departing from its novel spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A virtual reality-driven Stroop method for detecting mild cognitive impairment based on eye-tracking signals, characterized in that, The method includes the following steps: Step (1) Use the four tasks of the virtual reality-based Stroop test to collect eye movement signal data during the four tasks, perform preprocessing, and obtain the eye movement datasets of the four tasks. Step (2) divides the eye-tracking data into samples at fixed time intervals, resulting in each sample having a size of C channels (C=21) and a length of L (L=240), thus increasing the number of samples; Step (3) For leave-one-subjects cross-validation experiments and blind tests, the training and validation datasets are divided according to different strategies; Step (4) Perform wavelet scattering transform on the eye-tracking signal dataset to obtain the wavelet scattering transform feature matrix; Step (5) inputs the wavelet scattering feature matrix into the CNN network with time attention for training and learning, then performs feature dimension transformation and feeds it into the fully connected layer for prediction to obtain the prediction result for each task. Step (6) Integrate the prediction results corresponding to different tasks to obtain the final result and evaluate it.

2. The method for detecting mild cognitive impairment based on eye-tracking signals and driven by virtual reality according to claim 1, characterized in that, Step (1) is as follows: Step (1-1) For the eye-tracking dataset of each task, perform confidence-based filtering, linear interpolation to compensate for blink-related signal loss, edge noise trimming, and high-pass filtering (cutoff frequency: 1 Hz) to remove low-frequency drift.

3. The Stroop mild cognitive impairment detection method based on eye-tracking signals driven by virtual reality according to claim 1, characterized in that, Step (2) is as follows: Step (2-1) To ensure a sufficient number of samples, the ET clean signal of each subject in each task is divided into 2-second intervals using a Hamming window, with a 1-second overlap between each segment.

4. The Stroop mild cognitive impairment detection method based on eye-tracking signals driven by virtual reality according to claim 1, characterized in that, Step (3) is as follows: Step (3-1) Leave one subject out of the n subjects for cross-validation. Each time, leave one subject's sample as the validation set and the rest as the training set. Repeat this process n-1 times. Step (3-2) The blind test experiment involves randomly selecting 50% of the subjects as the validation set and the rest as the training set.

5. The Stroop mild cognitive impairment detection method based on eye-tracking signals driven by virtual reality according to claim 1, characterized in that, Step (4) is as follows: Step (4-1) involves performing wavelet scattering transform on the eye-tracking signal of the j-th channel (j=1,…C) of the i-th task (i=A,B,C,D). Perform scattering and calculate its wavelet scattering coefficients from order 0 to 2: in, They represent The 0th, 1st, and 2nd order scattering coefficients, the 0th order scattering coefficient capture signal The low-frequency component is calculated using the following formula: in It is a Gaussian low-pass filter with wavelet scaling parameter J (J=3), * denotes convolution operation, and first-order scattering coefficients. Features at a specific frequency scale λ: Where ψ λ It is a Morlet wavelet with frequency λ, |·| denotes the modulus operation, and the second-order scattering coefficients. Capturing features at two different frequency scales, λ and μ: Step (4-2) involves concatenating the coefficients of each channel to obtain the wavelet scattering feature matrix corresponding to each task.

6. The Stroop mild cognitive impairment detection method based on eye-tracking signals driven by virtual reality according to claim 2, characterized in that, Step (5) is as follows: Step (5-1) feeds the wavelet scattering feature matrix of the eye-tracking signal from step (4-2) into the Time Attention (TA) network to further dynamically allocate weights for each time step, resulting in a weighted feature matrix. Step (5-2) involves feeding the weighted feature matrix obtained in step (5-1) into the CNN network for training. Step (5-3) transforms the feature dimension of the features from step (5-2) and feeds them into the fully connected layer for prediction, thus obtaining the prediction results for each task.

7. The Stroop mild cognitive impairment detection method based on eye-tracking signals driven by virtual reality according to claim 3, characterized in that, Step (6) is as follows: Step (6-1) integrates and votes the prediction results of the models trained on the datasets corresponding to each task to obtain the final prediction results; Step (6-2) evaluates the prediction results for both leave-one-out crossover and blind tests using average classification accuracy, recall, precision, F1 score, AUC value, and confusion matrix.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the Stroop mild cognitive impairment detection method based on eye-tracking signals driven by virtual reality as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by a processor, the computer instructions implement the Stroop mild cognitive impairment detection method based on eye-tracking signals driven by virtual reality as described in any one of claims 1-7.

Images