Cognitive impairment screening method and device based on vr eye tracker and brain-computer interface

Abstract

The application discloses a cognitive disorder screening device and method based on a VR eye tracker and a brain-computer interface, and the device comprises a virtual reality scene generation module, an eye tracker data acquisition module, a brain-computer interface data acquisition module, a data analysis processing module and a cognitive disorder screening module. The virtual reality scene generation module is used for generating a virtual reality scene containing a cognitive task. The data analysis processing module is used for analyzing and processing the acquired eye movement data information set and electroencephalogram signal information set, and extracting a characteristic parameter sequence. The cognitive disorder screening module is used for performing cognitive disorder screening processing on the characteristic parameter sequence, and obtaining cognitive disorder information of a user. The cognitive disorder information comprises information about whether there is a cognitive disorder and a cognitive disorder degree.

Description

Technical Field

[0001] This invention relates to the fields of virtual reality and brain cognition, and specifically to a method and apparatus for screening cognitive impairment based on a VR eye tracker and a brain-computer interface. Background Technology

[0002] With the accelerating trend of an aging society, an increasing number of people are experiencing cognitive impairment. Cognitive impairment is a common neurological disorder, including dementia and mild cognitive impairment. Early and accurate screening is crucial for the diagnosis and treatment of cognitive impairment. Currently, commonly used screening methods for cognitive impairment include neuropsychological tests and imaging examinations, but these methods have certain limitations, such as high subjectivity, complex operation, and insensitivity to early signs of impairment.

[0003] How to achieve rapid, accurate, and simple early screening for cognitive impairment is an urgent problem that needs to be solved in the context of the current aging society. Summary of the Invention

[0004] This invention primarily addresses the problem of how to achieve rapid, accurate, and simple early screening for cognitive impairment. It discloses a method and device for cognitive impairment screening based on a VR eye tracker and a brain-computer interface. The purpose of this invention is to provide a method and device for cognitive impairment screening based on a virtual reality eye tracker and a brain-computer interface, thereby overcoming the limitations of existing cognitive impairment screening methods and improving the accuracy, objectivity, and efficiency of screening.

[0005] In a first aspect, this application discloses a cognitive impairment screening device based on a VR eye tracker and a brain-computer interface, comprising: a virtual reality scene generation module, an eye tracker data acquisition module, a brain-computer interface data acquisition module, a data analysis and processing module, and a cognitive impairment screening module;

[0006] The virtual reality scene generation module is used to generate virtual reality scenes that include cognitive tasks;

[0007] The eye tracker data acquisition module is used to acquire in real time a set of eye movement data information when the user completes a cognitive task in a virtual reality scene; the set of eye movement data information includes a fixation point information sequence, a fixation time information sequence, and a saccade frequency information sequence; the fixation point information sequence includes fixation point location information; the fixation time information sequence includes fixation time information; the saccade frequency information sequence includes saccade frequency information.

[0008] The brain-computer interface data acquisition module is used to acquire a set of brainwave signal information when the user completes a cognitive task in a virtual reality scene; the set of brainwave signal information includes a brainwave signal sequence.

[0009] The data analysis and processing module is connected to the virtual reality scene generation module, the eye tracker data acquisition module, the brain-computer interface data acquisition module, and the cognitive impairment screening module, respectively. It is used to analyze and process the acquired eye movement data information set and EEG signal information set to extract feature parameter sequences. The feature parameter sequences include an eye movement pattern abnormality parameter sequence and an EEG signal spectrum feature sequence.

[0010] The cognitive impairment screening module is used to perform cognitive impairment screening processing on the feature parameter sequence to obtain the user's cognitive impairment information; the cognitive impairment information includes whether cognitive impairment exists and the degree of cognitive impairment.

[0011] The data analysis and processing module is used to analyze and process the acquired eye movement data set and EEG signal set, and extract feature parameter sequences, including:

[0012] The data analysis and processing module performs preprocessing operations on the collected eye movement data information set and EEG signal information set respectively to obtain preprocessed eye movement data information set and EEG signal information set; performs abnormal feature analysis processing on the preprocessed eye movement data information set to obtain an eye movement pattern abnormal parameter sequence; and performs frequency domain abnormality identification processing on the preprocessed EEG signal information set to obtain an EEG signal spectral feature sequence.

[0013] The abnormal eye movement pattern parameter sequence and the electroencephalogram (EEG) signal spectral feature sequence are fused to obtain the feature parameter sequence.

[0014] The cognitive impairment screening module is used to perform cognitive impairment screening processing on the feature parameter sequence to obtain the user's cognitive impairment information, including:

[0015] The cognitive impairment screening module uses the trained cognitive impairment screening model to process the feature parameter sequence to obtain a cognitive impairment probability value.

[0016] Determine whether the probability value of the cognitive impairment is greater than a preset impairment probability threshold to obtain screening result information; if the screening result information is yes, confirm that the user has a cognitive impairment, confirm the degree of cognitive impairment information as the probability value of the cognitive impairment, and obtain cognitive impairment information; if the screening result information is no, confirm that the user does not have a cognitive impairment.

[0017] A second aspect of this application discloses a cognitive impairment screening method based on a VR eye tracker and a brain-computer interface, implemented using the aforementioned cognitive impairment screening device based on a VR eye tracker and a brain-computer interface, comprising:

[0018] S1, using the virtual reality scene generation module, generate a virtual reality scene containing a cognitive task; display the virtual reality scene containing the cognitive task to the user;

[0019] S2, using the eye tracker data acquisition module, the user's eye movement data information set is acquired in real time when completing cognitive tasks in the virtual reality scene; using the brain-computer interface data acquisition module, the user's electroencephalogram (EEG) signal information set is acquired when completing cognitive tasks in the virtual reality scene.

[0020] S3, using the data analysis and processing module, analyze and process the collected eye movement data information set and EEG signal information set to extract the feature parameter sequence;

[0021] S4. Using the cognitive impairment screening module, cognitive impairment screening processing is performed on the feature parameter sequence to obtain the user's cognitive impairment information.

[0022] The process of analyzing and processing the acquired eye-tracking data set and EEG signal set to extract a sequence of feature parameters includes:

[0023] S31, perform preprocessing operations on the collected eye movement data information set and EEG signal information set respectively to obtain the preprocessed eye movement data information set and EEG signal information set;

[0024] S32, perform abnormal feature analysis on the preprocessed eye movement data information set to obtain an abnormal parameter sequence of eye movement pattern;

[0025] S33, perform frequency domain anomaly recognition processing on the preprocessed set of EEG signal information to obtain the EEG signal spectral feature sequence;

[0026] S34, the abnormal eye movement pattern parameter sequence and the electroencephalogram (EEG) signal spectral feature sequence are fused to obtain the feature parameter sequence.

[0027] The preprocessing operation includes:

[0028] S311, Normalize each information set to obtain the corresponding normalized information set;

[0029] S312, perform data cleaning processing on each normalized information set to obtain the corresponding cleaned data set;

[0030] S313, perform pattern recognition and discrimination processing on each cleaned data set to obtain the corresponding preprocessed information set.

[0031] The pattern recognition and discrimination process includes:

[0032] S3131, for each data attribute in the cleaned data set, using the data collection information of the data as the independent variable and the data value of the data as the dependent variable, perform autoregressive-moving average modeling to obtain the regression model for each data attribute; use the regression model to calculate and process the independent variable to obtain the regression data value; determine whether the absolute value of the difference between the regression data value and the corresponding dependent variable value is greater than a set first regression discrimination threshold; if it is greater than the first regression discrimination threshold, delete the data from the cleaned data set; if it is less than or equal to the first regression discrimination threshold, do not process the data.

[0033] S3132, perform fusion processing on all the data in the cleaned data set after S3131 to obtain a preprocessed information set.

[0034] The preprocessed eye-tracking data set is subjected to abnormal feature analysis to obtain an eye-tracking pattern abnormal parameter sequence, including:

[0035] The gaze point information sequence in the preprocessed eye movement data information set is subjected to deviation evaluation processing to obtain the deviation value of each gaze point information sequence;

[0036] The sequence of gaze point information with deviation values ​​greater than a preset deviation threshold is identified as the gaze anomaly parameter sequence;

[0037] The calculation expression for the deviation evaluation process is:

[0038]

[0039] Where p1 is the calculation parameter, a i Let (x0, y0, z0) be the deviation value of the i-th gaze point information sequence, and (x0, y0, z0) be the position coordinates of the preset target gaze point in the cognitive task. ij ,y ij ,z ij ) represents the coordinates of the j-th gaze point in the i-th gaze point information sequence, and M represents the number of gaze point coordinates contained in a gaze point information sequence;

[0040] The fixation time information sequence and saccade frequency information sequence in the preprocessed eye movement data information set are subjected to joint anomaly assessment processing to obtain the time-frequency deviation value of each fixation time information;

[0041] Fixation time and saccade frequency information that are greater than the preset time-frequency deviation threshold are identified as time-frequency abnormal parameters.

[0042] All gaze abnormality parameter sequences and time-frequency abnormality parameters are merged to obtain the eye movement pattern abnormality parameter sequence;

[0043] The calculation expression for the joint anomaly assessment process is as follows:

[0044] c i =arctan(|t0-t) i | / |t0+t i |+|f0-f i | / |f0+f i |) / pi+(|t0-t i |+|f0-f i |) / (t0+f0),

[0045] Among them, c i Let t0 and f0 be the time-frequency anomaly parameters of the i-th fixation time information in the fixation time information sequence, respectively, and let t0 and f0 be the standard fixation time value and the standard saccade frequency value, respectively. i and f i pi represents the i-th fixation time information in the fixation time information sequence and the i-th saccade frequency information in the saccade frequency information sequence, respectively, where pi is the constant pi.

[0046] The step of performing frequency domain anomaly recognition processing on the preprocessed EEG signal information set to obtain an EEG signal spectral feature sequence includes:

[0047] For each EEG signal sequence in the preprocessed EEG signal information set, high-order feature quantity calculation processing is performed to obtain anomaly discrimination value;

[0048] The spectral sequence of an EEG signal sequence whose abnormality discrimination value is greater than a set abnormality threshold is identified as an abnormal EEG sequence;

[0049] By utilizing all the abnormal EEG sequences, a spectral feature sequence of EEG signals was constructed.

[0050] The expression for calculating the higher-order feature quantities is:

[0051]

[0052] α=ω1R4+ω2R 23 +ω3R′ 56 ,

[0053] Where α is the anomaly detection value of the EEG signal sequence, FFT() represents the Discrete Fourier Transform operation, x(n) represents the nth element of the EEG signal sequence, and y(n) represents the nth element of the frequency domain signal of the EEG signal sequence. R represents the conjugate element of y(n), ω1, ω2, and ω3 represent the preset first, second, and third weighting coefficients, respectively, N represents the length of the EEG signal sequence, R4 represents the fourth-order self-accumulator of the EEG signal sequence, and R23 R5′6 represents the second and third order cross-cumulative quantities of the EEG signal sequence, and R5′6 represents the fifth and sixth order cross-conjugate cumulative quantities of the EEG signal sequence.

[0054] The cognitive impairment screening model includes: an input module, a feature extraction module, a feature fusion module, a feature processing module, and a prediction module;

[0055] The input module is used to receive the obtained feature parameter sequence;

[0056] The feature extraction module is connected to the input module and the feature fusion module respectively, and is used to extract features from the feature parameter sequence to obtain a feature signal set.

[0057] The feature fusion module is used to perform fusion processing on the feature signal set to obtain a fused feature signal set;

[0058] The feature processing module is connected to the feature fusion module and the prediction module respectively, and is used to perform temporal correlation processing on the fused feature signal set to obtain a temporal cross feature signal set.

[0059] The prediction module is used to perform prediction processing on the temporal cross feature signal set to obtain the probability value of cognitive impairment;

[0060] The feature extraction module includes four feature extractors corresponding to the signal modes; each feature extractor is used to perform feature extraction processing on the feature parameter sequence of the corresponding mode to obtain the corresponding feature signal.

[0061] The feature fusion module includes four multi-head attention sub-modules; each multi-head attention sub-module is used to process the feature signal of the corresponding modality; the dimension of each multi-head attention sub-module matches the dimension of the feature signal of the modality it processes; the input of each multi-head attention sub-module is connected to the output of the corresponding feature extractor.

[0062] The feature processing module includes four long short-term memory network sub-modules; each long short-term memory network sub-module is used to process the feature parameter sequence of the corresponding modality; each long short-term memory network sub-module is connected to the output of the corresponding multi-head attention sub-module;

[0063] The prediction module includes a fully connected network and an output terminal; the four input terminals of the fully connected network are respectively connected to the output terminals of the four long short-term memory network sub-modules; the output terminal of the prediction module is used to output the probability value of cognitive impairment.

[0064] The beneficial effects of this invention are as follows:

[0065] This invention introduces virtual reality technology and eye-tracking and brain-computer interface (BCI) technologies into the field of cognitive impairment screening. Virtual reality technology can create realistic virtual environments that simulate various daily life scenarios, increasing the fun and engagement of the test; eye-tracking can monitor the user's eye movement trajectory in real time and obtain information such as visual attention; BCI technology can directly detect the brain's neural activity and reflect the state of cognitive function. By fusing multimodal physiological characteristics and detecting abnormal features, this invention enables early screening of users' cognitive impairments.

[0066] This invention enhances user engagement and the realism of testing through virtual reality technology, enabling better simulation of cognitive scenarios in daily life. By combining eye-tracking and brain-computer interface technologies, it acquires cognitive-related information from multiple dimensions, improving the accuracy and objectivity of screening. Furthermore, through automated data analysis and the establishment of a screening model, this invention reduces manual intervention and improves screening efficiency.

[0067] The cognitive impairment screening model established in this invention utilizes a feature extraction module to extract and process features from input multimodal abnormal signals. Through a feature fusion module, it achieves the fusion of cognitive impairment-related quantities from multiple types of features. Using a feature processing module and a prediction module, it performs anomaly analysis and prediction on the fused quantities, ultimately obtaining the probability value of the user's cognitive impairment. This invention achieves rapid and accurate screening of users' cognitive impairment through the cognitive impairment screening model. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of the device of the present invention;

[0069] Figure 2 This is a flowchart illustrating the implementation of the method of the present invention. Detailed Implementation

[0070] To better understand the content of this invention, an embodiment is provided here.

[0071] Figure 1 This is a schematic diagram of the device of the present invention; Figure 2 This is a flowchart illustrating the implementation of the method of the present invention.

[0072] In a first aspect, this application discloses a cognitive impairment screening device based on a VR eye tracker and a brain-computer interface, comprising: a virtual reality scene generation module, an eye tracker data acquisition module, a brain-computer interface data acquisition module, a data analysis and processing module, and a cognitive impairment screening module;

[0073] The virtual reality scene generation module is used to generate virtual reality scenes that include cognitive tasks; the cognitive tasks include object recognition, spatial navigation, memory testing, etc.

[0074] The eye-tracking data acquisition module is used to acquire a set of eye-tracking data information of the user in a virtual reality scene in real time. The set of eye-tracking data information includes a fixation point information sequence, a fixation time information sequence, and a saccade frequency information sequence. The fixation point information sequence includes fixation point location information; the fixation time information sequence includes fixation time information; and the saccade frequency information sequence includes saccade frequency information.

[0075] The brain-computer interface data acquisition module is used to acquire a set of electroencephalogram (EEG) signal information when the user completes a cognitive task. The set of EEG signal information includes an EEG signal sequence.

[0076] The data analysis and processing module is connected to the virtual reality scene generation module, the eye tracker data acquisition module, the brain-computer interface data acquisition module, and the cognitive impairment screening module, respectively. It is used to analyze and process the acquired eye movement data information set and EEG signal information set to extract feature parameter sequences. The feature parameter sequences include an eye movement pattern abnormality parameter sequence and an EEG signal spectrum feature sequence.

[0077] The cognitive impairment screening module is used to perform cognitive impairment screening processing on the feature parameter sequence to obtain the user's cognitive impairment information; the cognitive impairment information includes whether cognitive impairment exists and the degree of cognitive impairment.

[0078] The data analysis and processing module analyzes and processes the collected eye movement data set and EEG signal set to extract a sequence of feature parameters, including:

[0079] The data analysis and processing module performs preprocessing operations on the collected eye movement data information set and EEG signal information set respectively to obtain a preprocessed eye movement data information set and an EEG signal signal information set; and performs abnormal feature analysis processing on the preprocessed eye movement data information set to obtain an abnormal eye movement pattern parameter sequence.

[0080] The preprocessed EEG signal information set is subjected to frequency domain anomaly recognition processing to obtain the EEG signal spectral feature sequence;

[0081] The abnormal eye movement pattern parameter sequence and the electroencephalogram (EEG) signal spectral feature sequence are fused to obtain the feature parameter sequence.

[0082] The cognitive impairment screening module is used to perform cognitive impairment screening processing on the feature parameter sequence to obtain the user's cognitive impairment information, including:

[0083] The cognitive impairment screening module uses the trained cognitive impairment screening model to process the feature parameter sequence to obtain a cognitive impairment probability value.

[0084] Determine whether the probability value of the cognitive impairment is greater than a preset impairment probability threshold to obtain screening result information; if the screening result information is yes, confirm that the user has a cognitive impairment, confirm the degree of cognitive impairment information as the probability value of the cognitive impairment, and obtain cognitive impairment information; if the screening result information is no, confirm that the user does not have a cognitive impairment.

[0085] The preprocessing operation includes:

[0086] Normalize each set of information to obtain the corresponding normalized set of information;

[0087] Data cleaning is performed on each normalized information set to obtain the corresponding cleaned data set.

[0088] Pattern recognition and discrimination processing is performed on each cleaned dataset to obtain the corresponding preprocessed information set;

[0089] The pattern recognition and discrimination process includes:

[0090] S201, for each data attribute in the cleaned dataset, using the data collection information as the independent variable and the data value as the dependent variable, perform autoregressive-moving average modeling to obtain regression models for each data attribute; use the regression models to calculate and process the independent variables to obtain regression data values; determine whether the absolute value of the difference between the regression data value and the corresponding dependent variable value is greater than a set first regression threshold; if it is greater than the first regression threshold, delete the data from the cleaned dataset; if it is less than or equal to the first regression threshold, do not process the data.

[0091] S202, perform fusion processing on all the data in the cleaned data set after S201 to obtain a preprocessed information set;

[0092] The preprocessed eye-tracking data set is subjected to abnormal feature analysis to obtain an eye-tracking pattern abnormal parameter sequence, including:

[0093] The gaze point information sequence in the preprocessed eye movement data information set is subjected to deviation evaluation processing to obtain the deviation value of each gaze point information sequence;

[0094] The sequence of gaze point information with deviation values ​​greater than a preset deviation threshold is identified as the gaze anomaly parameter sequence;

[0095] The calculation expression for the deviation evaluation process is:

[0096]

[0097] Where p1 is the calculation parameter, a i Let (x0, y0, z0) be the deviation value of the i-th gaze point information sequence, and (x0, y0, z0) be the position coordinates of the preset target gaze point in the cognitive task. ij ,y ij ,z ij ) represents the coordinates of the j-th gaze point in the i-th gaze point information sequence, and M represents the number of gaze point coordinates contained in a gaze point information sequence;

[0098] The fixation time information sequence and saccade frequency information sequence in the preprocessed eye movement data information set are subjected to joint anomaly assessment processing to obtain the time-frequency deviation value of each fixation time information;

[0099] Fixation time and saccade frequency information that are greater than the preset time-frequency deviation threshold are identified as time-frequency abnormal parameters.

[0100] All gaze abnormality parameter sequences and time-frequency abnormality parameters are merged to obtain the eye movement pattern abnormality parameter sequence;

[0101] The calculation expression for the joint anomaly assessment process is as follows:

[0102] c i =arctan(|t0-t) i | / |t0+t i |+|f0-f i | / |f0+f i |) / pi+(|t0-t i |+|f0-f i |) / (t0+f0),

[0103] Among them, c i Let t0 and f0 be the time-frequency anomaly parameters of the i-th fixation time information in the fixation time information sequence, respectively, and let t0 and f0 be the standard fixation time value and the standard saccade frequency value, respectively. i and f i These are the i-th fixation time information in the fixation time information sequence and the i-th saccade frequency information in the saccade frequency information sequence, respectively, where pi is the constant pi.

[0104] The step of performing frequency domain anomaly recognition processing on the preprocessed EEG signal information set to obtain an EEG signal spectral feature sequence includes:

[0105] For each EEG signal sequence in the preprocessed EEG signal information set, high-order feature quantity calculation processing is performed to obtain anomaly discrimination value;

[0106] The spectral sequence of an EEG signal sequence whose abnormality discrimination value is greater than a set abnormality threshold is identified as an abnormal EEG sequence;

[0107] By utilizing all the abnormal EEG sequences, a spectral feature sequence of EEG signals was constructed.

[0108] The expression for calculating the higher-order feature quantities is:

[0109]

[0110] α=ω1R4+ω2R 23 +ω3R′ 56 ,

[0111] Where α is the anomaly detection value of the EEG signal sequence, FFT() represents the Discrete Fourier Transform operation, x(n) represents the nth element of the EEG signal sequence, and y(n) represents the nth element of the frequency domain signal of the EEG signal sequence. R represents the conjugate element of y(n), ω1, ω2, and ω3 represent the preset first, second, and third weighting coefficients, respectively, N represents the length of the EEG signal sequence, R4 represents the fourth-order self-accumulator of the EEG signal sequence, and R 23 R′ represents the second- and third-order cross-cumulants of an EEG signal sequence. 56 This represents the fifth and sixth order cross-conjugate cumulative quantity of the EEG signal sequence.

[0112] The cognitive impairment screening model includes: an input module, a feature extraction module, a feature fusion module, a feature processing module, and a prediction module;

[0113] The input module is used to receive the obtained feature parameter sequence;

[0114] The feature extraction module is connected to the input module and the feature fusion module respectively, and is used to extract features from the feature parameter sequence to obtain a feature signal set.

[0115] The feature fusion module is used to perform fusion processing on the feature signal set to obtain a fused feature signal set;

[0116] The feature processing module is connected to the feature fusion module and the prediction module respectively, and is used to perform temporal correlation processing on the fused feature signal set to obtain a temporal cross feature signal set.

[0117] The prediction module is used to perform prediction processing on the time-series cross-feature signal set to obtain the probability value of cognitive impairment.

[0118] The feature extraction module includes four feature extractors corresponding to the signal modes; each feature extractor is used to perform feature extraction processing on the feature parameter sequence of the corresponding mode to obtain the corresponding feature signal.

[0119] The signals of the four signal modes of the feature parameter sequence include the gaze abnormality parameter sequence, the gaze time information of the time-frequency abnormality parameter, the saccade frequency information of the time-frequency abnormality parameter, and the abnormal EEG sequence.

[0120] The feature extraction module includes four sub-modules for extracting feature parameter sequences of corresponding modalities. For the gaze abnormality parameter sequence, the corresponding feature extraction sub-module uses a Transformer network. For the gaze time information of the time-frequency abnormality parameter, the corresponding feature extraction sub-module uses a pre-trained VGG16net model. For the eye saccade frequency information of the time-frequency abnormality parameter, the corresponding feature extraction sub-module uses a CNN network. For the abnormal EEG sequence, the corresponding feature extraction sub-module uses an RNN network.

[0121] The feature fusion module includes four multi-head attention sub-modules; each multi-head attention sub-module is used to process the feature signal of the corresponding modality; the dimension of each multi-head attention sub-module matches the dimension of the feature signal of the modality it processes; the input of each multi-head attention sub-module is connected to the output of the corresponding feature extractor.

[0122] The feature processing module includes four long short-term memory network sub-modules; each long short-term memory network sub-module is used to process the feature parameter sequence of the corresponding modality; each long short-term memory network sub-module is connected to the output of the corresponding multi-head attention sub-module;

[0123] The prediction module includes a fully connected network and an output terminal; the four input terminals of the fully connected network are respectively connected to the output terminals of the four long short-term memory network sub-modules; the output terminal of the prediction module is used to output the probability value of cognitive impairment.

[0124] Fully connected networks are used to implement the Softmax function.

[0125] The multi-head attention submodule is used to perform modal cross-feature calculation processing on its corresponding feature signal to obtain modal cross-features; the output of a multi-head attention submodule is 4 modal cross-feature vectors; each multi-head attention submodule includes n attention heads;

[0126] The modal crossover feature calculation process is expressed as follows:

[0127]

[0128]

[0129] Among them, F l-uLet h be the modal cross-feature vector between the feature signal of the current multi-head attention submodule and the nth feature signal, where l is the index of the current multi-head attention submodule, l = 1, 2, ..., Nu = 1, 2, ..., N, h i This is the output of the i-th attention head in the current multi-head attention submodule. These are the key, value, and weighted query vector of the i-th attention head in the current multi-head attention submodule, where i = 1, 2, ..., n. q, k, and v are the key, value, and query vector input to the current multi-head attention submodule, respectively. v and k are the l-th feature signal, and q is the u-th feature signal. f is the first calculation function, whose expression is:

[0130]

[0131]

[0132] Where d is a preset calculation constant, and s(q2, q1) is an intermediate calculation quantity.

[0133] The training process of the cognitive impairment screening model includes:

[0134] A training dataset is constructed using historical feature parameter sequences and corresponding cognitive impairment probability values; the training data in the training dataset consists of historical feature parameter sequences, and the cognitive impairment probability values ​​corresponding to the historical feature parameter sequences are the label information corresponding to the training data.

[0135] Initialize the number of training iterations;

[0136] The training data in the training dataset is used as input data and input into the cognitive impairment screening model.

[0137] The input data is processed using the cognitive impairment screening model to obtain predicted values;

[0138] The difference between the predicted value and the label information corresponding to the input data is calculated to obtain the difference value.

[0139] Determine whether the difference value satisfies the convergence condition to obtain the first determination result;

[0140] When the first judgment result is negative, it is determined whether the training iteration count value is equal to the training count threshold to obtain the second judgment result;

[0141] When the second judgment result is negative, the model training state is determined to be that the termination training condition is not met.

[0142] When the second judgment result is yes, the model training state is determined to meet the termination training condition;

[0143] When the first judgment result is yes, it is determined that the model training state meets the termination training condition;

[0144] When the training state of the model does not meet the termination training condition, the parameters of the cognitive impairment screening model are updated using the parameter update model, the training iteration count is increased by 1, and the training data in the training dataset is used as input data to the cognitive impairment screening model.

[0145] When the model training state meets the termination training condition, the training process of the cognitive impairment screening model is completed, and the trained cognitive impairment screening model is obtained.

[0146] The parameter update model is as follows:

[0147]

[0148] θ←θ+v;

[0149] In the formula, Let v be the difference value calculated for the i-th training data in the training dataset, θ be the parameter update value, η be the parameters of the module to be updated, η be the initial parameter learning rate, and α be the momentum angle parameter, where 0 ≤ α ≤ π / 4. This indicates taking the partial derivative with respect to the variable θ;

[0150] The difference value satisfies the convergence condition when it is less than a preset convergence threshold; the difference value does not satisfy the convergence condition when it is not less than a preset convergence threshold.

[0151] The difference calculation process can be implemented using a loss function.

[0152] The loss function can be the cross-entropy loss function.

[0153] When using the aforementioned cognitive impairment screening device based on VR eye tracker and brain-computer interface to screen for cognitive impairment, the steps include:

[0154] (1) Preparation stage

[0155] Select appropriate virtual reality devices, eye trackers, and brain-computer interface devices, and calibrate and adjust them. Invite a group of healthy volunteers and known users of cognitive impairment to participate in the testing.

[0156] (2) Virtual Reality Scene Generation

[0157] Design virtual reality scenarios that incorporate different cognitive tasks, such as a virtual supermarket where test subjects need to find specific products.

[0158] (3) Data Acquisition

[0159] The test subject wears the device to enter the virtual reality scene, and at the same time the eye tracker and brain-computer interface device begin to collect data.

[0160] (4) Data Analysis

[0161] The collected data is preprocessed to remove noise and outliers. Eye movement features (such as fixation point distribution and saccade velocity) and electroencephalogram (EEG) features (such as power spectral density in specific frequency bands) are extracted.

[0162] (5) Establish a screening model

[0163] A cognitive impairment screening model was established based on the extracted features.

[0164] (6) Model Validation

[0165] The model was validated using a reserved test set to evaluate its accuracy, sensitivity, and specificity; the cognitive impairment screening model was used to screen users.

[0166] The virtual reality scene generation module can be implemented using virtual reality glasses; the eye tracker data acquisition module can be implemented using an eye tracker or a VR eye tracker developed by Xi'an Blue Brain Technology Co., Ltd.; and the brain-computer interface data acquisition module can be implemented using a BCI acquisition device.

[0167] A second aspect of this application discloses a cognitive impairment screening method based on a VR eye tracker and a brain-computer interface, implemented using the aforementioned cognitive impairment screening device based on a VR eye tracker and a brain-computer interface, comprising:

[0168] S1, using the virtual reality scene generation module, generate a virtual reality scene containing a cognitive task; display the virtual reality scene containing the cognitive task to the user;

[0169] S2, using the eye tracker data acquisition module, the user's eye movement data information set in the virtual reality scene is acquired in real time; using the brain-computer interface data acquisition module, the user's electroencephalogram (EEG) signal information set when completing cognitive tasks is acquired.

[0170] S3, using the data analysis and processing module, analyze and process the collected eye movement data information set and EEG signal information set to extract the feature parameter sequence;

[0171] S4, using the cognitive impairment screening module, perform cognitive impairment screening processing on the feature parameter sequence to obtain the user's cognitive impairment information.

[0172] The preprocessing operation includes:

[0173] Normalize each set of information to obtain the corresponding normalized set of information;

[0174] Data cleaning is performed on each normalized information set to obtain the corresponding cleaned data set.

[0175] Pattern recognition and discrimination processing is performed on each cleaned dataset to obtain the corresponding preprocessed information set;

[0176] The pattern recognition and discrimination process includes:

[0177] S201, for each data attribute in the cleaned dataset, using the data collection information as the independent variable and the data value as the dependent variable, perform autoregressive-moving average modeling to obtain regression models for each data attribute; use the regression models to calculate and process the independent variables to obtain regression data values; determine whether the absolute value of the difference between the regression data value and the corresponding dependent variable value is greater than a set first regression threshold; if it is greater than the first regression threshold, delete the data from the cleaned dataset; if it is less than or equal to the first regression threshold, do not process the data.

[0178] S202, perform fusion processing on all the data in the cleaned data set after S201 to obtain a preprocessed information set;

[0179] The data acquisition information refers to the data acquisition time information;

[0180] The preprocessed eye-tracking data set is subjected to abnormal feature analysis to obtain an eye-tracking pattern abnormal parameter sequence, including:

[0181] The gaze point information sequence in the preprocessed eye movement data information set is subjected to deviation evaluation processing to obtain the deviation value of each gaze point information sequence;

[0182] The sequence of gaze point information with deviation values ​​greater than a preset deviation threshold is identified as the gaze anomaly parameter sequence;

[0183] The calculation expression for the deviation evaluation process is:

[0184]

[0185] Where p1 is the calculation parameter, a iLet (x0, y0, z0) be the deviation value of the i-th gaze point information sequence, and (x0, y0, z0) be the position coordinates of the preset target gaze point in the cognitive task. ij ,y ij ,z ij ) represents the coordinates of the j-th gaze point in the i-th gaze point information sequence, and M represents the number of gaze point coordinates contained in a gaze point information sequence;

[0186] A joint anomaly assessment process is performed on the fixation time information sequence and saccade frequency information sequence in the preprocessed eye movement data information set to obtain the time-frequency deviation value of each fixation time information;

[0187] Fixation time and saccade frequency information that are greater than the preset time-frequency deviation threshold are identified as time-frequency abnormal parameters.

[0188] All gaze abnormality parameter sequences and time-frequency abnormality parameters are merged to obtain the eye movement pattern abnormality parameter sequence;

[0189] The calculation expression for the joint anomaly assessment process is:

[0190] c i =arctan(|t0-t) i | / |t0+t i |+|f0-f i | / |f0+f i |) / pi+(|t0-t i |+|f0-f i |) / (t0+f0),

[0191] Among them, c i Let t0 be the time-frequency anomaly parameter of the i-th fixation time information in the fixation time information sequence, and f0 be the standard fixation time value and the standard saccade frequency value, respectively. i and f i These are the i-th fixation time information in the fixation time information sequence and the i-th saccade frequency information in the saccade frequency information sequence, respectively, where pi is the constant pi.

[0192] The step of performing frequency domain anomaly recognition processing on the preprocessed EEG signal information set to obtain an EEG signal spectral feature sequence includes:

[0193] For each EEG signal sequence in the preprocessed EEG signal information set, high-order feature quantity calculation processing is performed to obtain anomaly discrimination value;

[0194] The spectral sequence of an EEG signal sequence whose abnormality discrimination value is greater than a set abnormality threshold is identified as an abnormal EEG sequence;

[0195] By utilizing all the abnormal EEG sequences, a spectral feature sequence of EEG signals was constructed.

[0196] The expression for calculating the higher-order feature quantities is:

[0197]

[0198] α=ω1R4+ω2R 23 +ω3R′ 56 ,

[0199] Where α is the anomaly detection value of the EEG signal sequence, FFT() represents the Discrete Fourier Transform operation, x(n) represents the nth element of the EEG signal sequence, and y(n) represents the nth element of the frequency domain signal of the EEG signal sequence. R represents the conjugate element of y(n), ω1, ω2, and ω3 represent the preset first, second, and third weighting coefficients, respectively, N represents the length of the EEG signal sequence, R4 represents the fourth-order self-accumulator of the EEG signal sequence, and R 23 R′ represents the second- and third-order cross-cumulants of an EEG signal sequence. 56 This represents the fifth and sixth order cross-conjugate cumulative quantity of the EEG signal sequence.

[0200] The cognitive impairment screening module performs cognitive impairment screening processing on the feature parameter sequence to obtain the user's cognitive impairment information, including:

[0201] The cognitive impairment screening module uses the trained cognitive impairment screening model to process the feature parameter sequence to obtain a cognitive impairment probability value.

[0202] Determine whether the probability value of the cognitive impairment is greater than a preset impairment probability threshold to obtain screening result information; if the screening result information is yes, confirm that the user has a cognitive impairment and confirm that the degree of cognitive impairment information is the probability value of the cognitive impairment; if the screening result information is no, confirm that the user does not have a cognitive impairment.

[0203] The cognitive impairment screening model includes: an input module, a feature extraction module, a feature fusion module, a feature processing module, and a prediction module;

[0204] The input module is used to receive the obtained feature parameter sequence;

[0205] The feature extraction module is connected to the input module and the feature fusion module respectively, and is used to extract features from the feature parameter sequence to obtain a feature signal set.

[0206] The feature fusion module is used to perform fusion processing on the feature signal set to obtain a fused feature signal set;

[0207] The feature processing module is connected to the feature fusion module and the prediction module respectively, and is used to perform temporal correlation processing on the fused feature signal set to obtain a temporal cross feature signal set.

[0208] The prediction module is used to perform prediction processing on the time-series cross-feature signal set to obtain the probability value of cognitive impairment.

[0209] The feature extraction module includes four feature extractors corresponding to the signal modes; each feature extractor is used to perform feature extraction processing on the feature parameter sequence of the corresponding mode to obtain the corresponding feature signal.

[0210] The signals of the four signal modes of the feature parameter sequence include the gaze abnormality parameter sequence, the gaze time information of the time-frequency abnormality parameter, the saccade frequency information of the time-frequency abnormality parameter, and the abnormal EEG sequence.

[0211] The feature extraction module includes four sub-modules for extracting feature parameter sequences of corresponding modalities. For the gaze abnormality parameter sequence, the corresponding feature extraction sub-module uses a Transformer network. For the gaze time information of the time-frequency abnormality parameter, the corresponding feature extraction sub-module uses a pre-trained VGG16net model. For the eye saccade frequency information of the time-frequency abnormality parameter, the corresponding feature extraction sub-module uses a CNN network. For the abnormal EEG sequence, the corresponding feature extraction sub-module uses an RNN network.

[0212] The feature fusion module includes four multi-head attention sub-modules; each multi-head attention sub-module is used to process the feature signal of the corresponding modality; the dimension of each multi-head attention sub-module matches the dimension of the feature signal of the modality it processes; the input of each multi-head attention sub-module is connected to the output of the corresponding feature extractor.

[0213] The feature processing module includes four long short-term memory network sub-modules; each long short-term memory network sub-module is used to process the feature parameter sequence of the corresponding modality; each long short-term memory network sub-module is connected to the output of the corresponding multi-head attention sub-module;

[0214] The prediction module includes a fully connected network and an output terminal; the four input terminals of the fully connected network are respectively connected to the output terminals of the four long short-term memory network sub-modules; the output terminal of the prediction module is used to output the probability value of cognitive impairment.

[0215] Fully connected networks are used to implement the Softmax function.

[0216] The multi-head attention submodule is used to perform modal cross-feature calculation processing on its corresponding feature signal to obtain modal cross-features; the output of a multi-head attention submodule is 4 modal cross-feature vectors; each multi-head attention submodule includes n attention heads;

[0217] The modal crossover feature calculation process is expressed as follows:

[0218]

[0219]

[0220] Among them, F l-u Let be the modal cross-feature vector between the feature signal of the current multi-head attention submodule and the nth feature signal, where l is the index of the current multi-head attention submodule, l = 1, 2, ..., N, u = 1, 2, ..., N, h i This is the output of the i-th attention head in the current multi-head attention submodule. These are the key, value, and weighted query vector of the i-th attention head in the current multi-head attention submodule, where i = 1, 2, ..., n. q, k, and v are the key, value, and query vector input to the current multi-head attention submodule, respectively. v and k are the l-th feature signal, and q is the u-th feature signal. f is the first calculation function, whose expression is:

[0221]

[0222]

[0223] Where d is a preset calculation constant, and s(q2, q1) is an intermediate calculation quantity.

[0224] The training process of the cognitive impairment screening model includes:

[0225] A training dataset is constructed using historical feature parameter sequences and corresponding cognitive impairment probability values; the training data in the training dataset consists of historical feature parameter sequences, and the cognitive impairment probability values ​​corresponding to the historical feature parameter sequences are the label information corresponding to the training data.

[0226] Initialize the number of training iterations;

[0227] The training data in the training dataset is used as input data and input into the cognitive impairment screening model.

[0228] The input data is processed using the cognitive impairment screening model to obtain predicted values;

[0229] The difference between the predicted value and the label information corresponding to the input data is calculated to obtain the difference value.

[0230] Determine whether the difference value satisfies the convergence condition to obtain the first determination result;

[0231] When the first judgment result is negative, it is determined whether the training iteration count value is equal to the training count threshold to obtain the second judgment result;

[0232] When the second judgment result is negative, the model training state is determined to be that the termination training condition is not met.

[0233] When the second judgment result is yes, it is determined that the model training state meets the termination training condition;

[0234] When the first judgment result is yes, it is determined that the model training state meets the termination training condition;

[0235] When the training state of the model does not meet the termination training condition, the parameters of the cognitive impairment screening model are updated using the parameter update model, the training iteration count is increased by 1, and the training data in the training dataset is used as input data to the cognitive impairment screening model.

[0236] When the model training state meets the termination training condition, the training process of the cognitive impairment screening model is completed, and the trained cognitive impairment screening model is obtained.

[0237] The parameter update model is as follows:

[0238]

[0239] θ←θ+v;

[0240] In the formula, Let v be the difference value calculated for the i-th training data in the training dataset, θ be the parameter update value, η be the parameters of the module to be updated, η be the initial parameter learning rate, and α be the momentum angle parameter, where 0 ≤ α ≤ π / 4. This indicates taking the partial derivative with respect to the variable θ;

[0241] The difference value satisfies the convergence condition when it is less than a preset convergence threshold; the difference value does not satisfy the convergence condition when it is not less than a preset convergence threshold.

[0242] The difference calculation process can be implemented using a loss function.

[0243] The loss function can be the cross-entropy loss function.

[0244] The normalization process maps data from different value ranges to a specified value range, and its mathematical expression is:

[0245]

[0246] Where t is a specific data value in the dataset, t max t represents the maximum value of all data in the dataset. min t is the minimum value of all data in the dataset. 归一化 The value of data t after normalization;

[0247] The data cleaning process includes filling in missing values, smoothing noisy data, and smoothing or deleting outliers. Smoothing noisy data involves first identifying the noisy data, and then smoothing it based on the data preceding and following it. The noisy data refers to values ​​whose values ​​are less than the sensor's detection sensitivity for the observed data, or greater than the sensor's measurement upper limit for the observed data. Outlier identification can be performed using a Kalman filter. The filling values ​​for missing values ​​can be determined by averaging the measurements within a certain sampling interval before and after the missing value.

[0248] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A cognitive impairment screening device based on a VR eye tracker and a brain-computer interface, characterized in that, include: Virtual reality scene generation module, eye tracker data acquisition module, brain-computer interface data acquisition module, data analysis and processing module, cognitive impairment screening module; The virtual reality scene generation module is used to generate virtual reality scenes that include cognitive tasks; The eye tracker data acquisition module is used to acquire in real time a set of eye movement data information when the user completes a cognitive task in a virtual reality scene; the set of eye movement data information includes a fixation point information sequence, a fixation time information sequence, and a saccade frequency information sequence; the fixation point information sequence includes fixation point location information; the fixation time information sequence includes fixation time information; the saccade frequency information sequence includes saccade frequency information. The brain-computer interface data acquisition module is used to acquire a set of brainwave signal information when the user completes a cognitive task in a virtual reality scene; the set of brainwave signal information includes a brainwave signal sequence. The data analysis and processing module is connected to the virtual reality scene generation module, the eye tracker data acquisition module, the brain-computer interface data acquisition module, and the cognitive impairment screening module, respectively. It is used to analyze and process the acquired eye movement data information set and brain electrical signal information set to extract feature parameter sequences. The feature parameter sequence includes an eye movement pattern abnormality parameter sequence and an EEG signal spectral feature sequence; The cognitive impairment screening module is used to perform cognitive impairment screening processing on the feature parameter sequence to obtain the user's cognitive impairment information; The cognitive impairment information includes whether cognitive impairment exists and the degree of cognitive impairment.

2. The cognitive impairment screening device based on VR eye tracker and brain-computer interface as described in claim 1, characterized in that, The data analysis and processing module is used to analyze and process the collected eye movement data set and EEG signal set, and extract feature parameter sequences, including: The data analysis and processing module performs preprocessing operations on the collected eye movement data information set and EEG signal information set respectively to obtain preprocessed eye movement data information set and EEG signal information set; performs abnormal feature analysis processing on the preprocessed eye movement data information set to obtain an eye movement pattern abnormal parameter sequence; and performs frequency domain abnormality identification processing on the preprocessed EEG signal information set to obtain an EEG signal spectral feature sequence. The abnormal eye movement pattern parameter sequence and the electroencephalogram (EEG) signal spectral feature sequence are fused to obtain the feature parameter sequence.

3. The cognitive impairment screening device based on VR eye tracker and brain-computer interface as described in claim 1, characterized in that, The cognitive impairment screening module is used to perform cognitive impairment screening processing on the feature parameter sequence to obtain the user's cognitive impairment information, including: The cognitive impairment screening module uses the trained cognitive impairment screening model to process the feature parameter sequence to obtain a cognitive impairment probability value. Determine whether the probability value of the cognitive impairment is greater than a preset impairment probability threshold to obtain screening result information; if the screening result information is yes, confirm that the user has a cognitive impairment, confirm the degree of cognitive impairment information as the probability value of the cognitive impairment, and obtain cognitive impairment information; if the screening result information is no, confirm that the user does not have a cognitive impairment.

Images