Methods and devices for screening cognitive impairment based on AR eye trackers and brain-computer interfaces

Abstract

This invention discloses a cognitive impairment screening device and method based on an AR eye tracker and a brain-computer interface. The device includes: an augmented reality scene generation module, an eye tracker data acquisition module, a brain-computer interface data acquisition module, and a cognitive impairment screening module. The augmented reality scene generation module is used to generate and display an augmented reality scene containing a cognitive task and acquire task scene image information. The eye tracker data acquisition module is used to acquire, in real time, a set of eye movement data information of the user when completing the cognitive task in the augmented reality scene. The brain-computer interface data acquisition module is used to acquire, in real time, a set of electroencephalogram (EEG) signal information of the user when completing the cognitive task in the augmented reality scene. The cognitive impairment screening module is used to comprehensively evaluate and process the acquired task scene image information, eye movement data information, and EEG signal information to obtain the user's cognitive impairment information.

Description

Technical Field

[0001] This invention relates to the fields of augmented reality and brain cognition, and specifically to a method and apparatus for screening cognitive impairment based on an AR eye tracker and a brain-computer interface. Background Technology

[0002] With the accelerating trend of an aging society, an increasing number of people are experiencing cognitive impairment. Cognitive impairment is a common neurological abnormality, including dementia and mild cognitive impairment. Early and accurate screening is crucial for the diagnosis and treatment of cognitive impairment. Currently, commonly used screening methods for cognitive impairment include neuropsychological tests and imaging examinations, but these methods have certain limitations, such as high subjectivity, complex operation, and insensitivity to early lesions.

[0003] How to achieve rapid, accurate, and simple early screening for cognitive impairment is an urgent problem that needs to be solved in the context of the current aging society. Summary of the Invention

[0004] This invention primarily addresses the problem of how to achieve rapid, accurate, and simple early screening for cognitive impairment. This invention discloses a method and device for screening cognitive impairment based on an AR eye tracker and a brain-computer interface.

[0005] The purpose of this invention is to provide a method and device for screening cognitive impairment based on augmented reality (AR) eye trackers and brain-computer interfaces, so as to overcome the limitations of existing cognitive impairment screening methods and improve the accuracy, objectivity and efficiency of screening.

[0006] In a first aspect, this application discloses a cognitive impairment screening device based on an AR eye tracker and a brain-computer interface, characterized in that it includes: an augmented reality scene generation module, an eye tracker data acquisition module, a brain-computer interface data acquisition module, and a cognitive impairment screening module;

[0007] The augmented reality scene generation module is used to generate and display augmented reality scenes containing cognitive tasks and to acquire task scene image information.

[0008] The eye tracker data acquisition module is used to collect a set of eye movement data information in real time when the user completes cognitive tasks in an augmented reality scene; the set of eye movement data information includes a fixation point information sequence, a fixation time information sequence, and a saccade frequency information sequence; the fixation point information sequence includes fixation point location information; the fixation time information sequence includes fixation time information; the saccade frequency information sequence includes saccade frequency information.

[0009] The brain-computer interface data acquisition module is used to acquire in real time a set of brainwave signal information when the user completes cognitive tasks in an augmented reality scenario; the set of brainwave signal information includes brainwave signal sequences.

[0010] The cognitive impairment screening module is connected to the augmented reality scene generation module, the eye tracker data acquisition module, and the brain-computer interface data acquisition module, respectively. It is used to comprehensively evaluate and process the acquired task scene image information, eye track data information set, and brain electrical signal information set to obtain the user's cognitive impairment information. The cognitive impairment information includes whether cognitive impairment exists and the degree of cognitive impairment.

[0011] The task scene image information includes real-time images of the task scene, target location information, and coordinates of the user's participating position in the task. The target location information includes the coordinate range information of several targets in the task scene. The real-time images of the task scene include images of the user participating in the cognitive task. The coordinates of the user's participating position in the task scene are the coordinates of the user's position when participating in the cognitive task. Each target location information has corresponding coordinates of the user's participating position in the task.

[0012] The augmented reality scene generation module includes an augmented reality scene display submodule, an image acquisition submodule, a location acquisition submodule, and a data acquisition submodule;

[0013] The location acquisition submodule is used to acquire the location coordinates of the user when participating in the cognitive task in the task scenario, and obtain the location coordinates of the user participating in the task.

[0014] The augmented reality scene display submodule is used to generate and display an augmented reality scene containing cognitive tasks and to collect target location information; the augmented reality scene contains targets that require user interaction when participating in completing cognitive tasks;

[0015] The image acquisition submodule is used to acquire real-time images of the task scene when the user participates in completing the cognitive task in the task scene;

[0016] The data acquisition submodule is used to fuse the acquired real-time images of the task scene, target location information, and coordinates of the locations of the participants in the task to obtain task scene image information, and then send the task scene image information to the cognitive impairment screening module.

[0017] The cognitive impairment screening module is used to comprehensively evaluate and process the collected task scene image information, eye movement data information set, and electroencephalogram (EEG) signal information set to obtain the user's cognitive impairment information, including:

[0018] The cognitive impairment screening module evaluates and processes the task scene image information to obtain user participation task evaluation result information;

[0019] The eye-tracking data set is evaluated and processed to obtain an eye-tracking evaluation result value;

[0020] The set of EEG signal information is evaluated and processed to obtain EEG evaluation result values;

[0021] Using a preset first weight vector, the user participation task evaluation result information, eye movement evaluation result value, and electroencephalogram evaluation result value are weighted and summed to obtain the cognitive impairment evaluation value;

[0022] Determine whether the cognitive impairment assessment value is greater than a set cognitive discrimination threshold to obtain a cognitive discrimination result; if the cognitive discrimination result is greater than, determine that the user has a cognitive impairment, and determine the degree of cognitive impairment information in the cognitive impairment information as the cognitive impairment assessment value; if the cognitive discrimination result is not greater than, determine that the user does not have a cognitive impairment.

[0023] Using the information on the degree of cognitive impairment and the presence or absence of cognitive impairment, the user's cognitive impairment information is constructed.

[0024] A second aspect of this invention discloses a cognitive impairment screening method based on an AR eye tracker and a brain-computer interface, implemented using the aforementioned cognitive impairment screening device based on an AR eye tracker and a brain-computer interface, comprising:

[0025] Using the augmented reality scene generation module, an augmented reality scene containing cognitive tasks is generated and displayed, and task scene image information is acquired.

[0026] The eye-tracking data acquisition module is used to collect a set of eye-tracking data information of the user in the augmented reality scene in real time;

[0027] Using the brain-computer interface data acquisition module, a set of brainwave signal information of the user when completing cognitive tasks is acquired;

[0028] The cognitive impairment screening module is used to comprehensively evaluate and process the collected task scene image information, eye movement data information set and EEG signal information set to obtain the user's cognitive impairment information.

[0029] The process of comprehensively evaluating and processing the collected task scene image information, eye movement data information set, and electroencephalogram (EEG) signal information set yields the user's cognitive impairment information, including:

[0030] The task scene image information is evaluated and processed to obtain the user participation task evaluation result information;

[0031] The eye-tracking data set is evaluated and processed to obtain an eye-tracking evaluation result value;

[0032] The set of EEG signal information is evaluated and processed to obtain EEG evaluation result values;

[0033] Using a preset first weight vector, the user participation task evaluation result information, eye movement evaluation result value, and electroencephalogram evaluation result value are weighted and summed to obtain the cognitive impairment evaluation value;

[0034] Determine whether the cognitive impairment assessment value is greater than a set cognitive discrimination threshold to obtain a cognitive discrimination result; if the cognitive discrimination result is greater than, determine that the user has a cognitive impairment, and determine the degree of cognitive impairment information in the cognitive impairment information as the cognitive impairment assessment value; if the cognitive discrimination result is not greater than, determine that the user does not have a cognitive impairment.

[0035] Using the information on the degree of cognitive impairment and the presence or absence of cognitive impairment, the user's cognitive impairment information is constructed.

[0036] The process of evaluating the eye movement data set to obtain an eye movement evaluation result value includes:

[0037] The gaze point information sequence in the eye movement data information set is subjected to deviation evaluation processing to obtain the deviation value of each gaze point information sequence;

[0038] The calculation expression for the deviation evaluation process is:

[0039]

[0040] Where t1 is a preset calculation parameter, a i Let (x0, y0, z0) be the deviation value of the i-th gaze point information sequence, and (x0, y0, z0) be the position coordinates of the preset target gaze point in the cognitive task. ij ,y ij ,z ij ) represents the coordinates of the j-th gaze point in the i-th gaze point information sequence, and M represents the number of gaze points contained in a gaze point information sequence;

[0041] The mean of the deviations of all fixation information sequences is used to obtain the fixation evaluation estimate;

[0042] A joint anomaly assessment process is performed on the fixation time information sequence and saccade frequency information sequence in the eye movement data information set to obtain a time-frequency assessment value;

[0043] The calculation expression for the joint anomaly assessment process is as follows:

[0044]

[0045]

[0046] Among them, c ij Let t0 and f0 be the time-frequency anomaly parameters of the i-th fixation time information in the j-th fixation time information sequence, respectively, and t0 and f0 be the standard fixation time value and standard saccade frequency value. ij and f ij Let pi be the ith fixation time information in the j-th fixation time information sequence and the ith saccade frequency information in the j-th saccade frequency information sequence, respectively, where pi is the constant of pi, and c is the saccade frequency information. j Let P be the time-frequency evaluation value of the j-th gaze time information sequence, and let P be the number of gaze time information sequences contained in a gaze time information sequence.

[0047] The mean time-frequency evaluation values ​​corresponding to all fixation time information sequences and saccade frequency information sequences are averaged to obtain the mean time-frequency evaluation value.

[0048] The eye movement assessment result is obtained by weighted summation of the gaze evaluation estimate and the mean of the time-frequency assessment.

[0049] The evaluation and processing of the task scene image information to obtain user participation task evaluation result information includes:

[0050] The real-time image of the task scene in the task scene image information is subjected to a first evaluation process to obtain first evaluation information;

[0051] A second evaluation process is performed on the participating task location coordinates and the target location information in the task scene image information to obtain second evaluation information;

[0052] Using a preset evaluation weighting vector, the first evaluation information and the second evaluation information are weighted and summed to obtain the user participation task evaluation result information.

[0053] The first evaluation process of the real-time image of the task scene in the task scene image information to obtain first evaluation information includes:

[0054] Human key point recognition is performed on the real-time image of the task scene in the task scene image information to obtain the key point spatial location matrix;

[0055] The difference calculation process is performed on the key point spatial location matrix and the standard participating task location matrix to obtain the first evaluation information;

[0056] The difference calculation process includes:

[0057] Subtract the key point spatial location matrix from the standard participating task location matrix to obtain the difference matrix;

[0058] The difference matrix is ​​subjected to a first standard calculation to obtain a first standard matrix;

[0059] The expression for the first standard calculation is:

[0060]

[0061] Where m represents the row dimension of the difference matrix, x ij z represents the element in the i-th row and j-th column of the difference matrix. ij This represents the element in the i-th row and j-th column of the first standard matrix;

[0062] Perform column optimization processing on the first standard matrix to obtain the optimal solution vector;

[0063] The optimal processing of the columns involves extracting the largest number from each column to form the optimal solution vector z. + The expression for the optimal solution vector is:

[0064]

[0065] Where n represents the column dimension of the difference matrix;

[0066] Perform column worst-case processing on the first standard matrix to obtain the worst-case solution vector;

[0067] The worst-case scenario processing described above involves extracting the smallest number from each column to form the worst-case solution vector z. - The expression for the worst-case solution vector is:

[0068]

[0069] The worst-case solution vector and the best-case solution vector are scored to obtain the first evaluation information; the expression for the scoring calculation is:

[0070]

[0071] In the formula, ω j ω is the preset importance weight for the j-th element; j s is the first evaluation information, obtained by pre-setting or by calculating the variance of each column of the first standard matrix.

[0072] The second evaluation process, which involves analyzing the coordinates of the participating task location and the target location information in the task scene image information to obtain second evaluation information, includes:

[0073] Determine the target location information corresponding to the coordinates of the participating task location;

[0074] Extract the set of coordinates of the boundary points from which the target location information is obtained;

[0075] Using the set of coordinates of the boundary points, a boundary matrix is ​​constructed;

[0076] Using the coordinates of the participating task locations, a target vector is constructed;

[0077] The boundary matrix and the target vector are subjected to difference evaluation processing to obtain the second evaluation information.

[0078] The beneficial effects of this invention are as follows:

[0079] This invention introduces augmented reality (AR) technology and eye-tracking and brain-computer interface (BCI) technology into the field of cognitive impairment screening. AR technology can create realistic virtual environments that simulate various daily life scenarios, increasing the fun and engagement of the tests while reducing adverse reactions such as dizziness associated with virtual reality technology.

[0080] This invention uses an eye tracker to monitor the user's eye movement trajectory in real time and obtain information such as visual attention; brain-computer interface technology can directly detect the neural activity of the brain and reflect the state of cognitive function; by establishing special evaluation variables based on the characteristics of brain-computer interface data, it achieves rapid and accurate evaluation of the user's brain data.

[0081] This invention targets eye-tracking data and employs a multi-level evaluation approach. It obtains a gaze evaluation estimate by performing deviation evaluation processing on the gaze point information sequence; it obtains a time-frequency evaluation value by performing joint anomaly evaluation processing on the gaze time information sequence and saccade frequency information sequence in the eye-tracking data information set; and finally, it fuses the multiple evaluation values ​​to obtain the eye-tracking evaluation result value.

[0082] This invention fully leverages the advantages of augmented reality technology, which can acquire user images and location coordinates. By collecting real-time images of the task scene, target location information, and the location coordinates of participants in the task, and establishing specialized first and second evaluation processing methods, the invention evaluates the standard of action and positional accuracy of the user when participating in the cognitive task, respectively, and obtains the evaluation result information of the user's participation in the task, thus realizing the evaluation of cognitive impairment from the macro-action domain.

[0083] This invention achieves automated screening for cognitive impairment by effectively integrating the assessment results of three types: eye, electroencephalogram (EEG), and motor function. This eliminates the influence of subjective factors and improves the accuracy, objectivity, and efficiency of the screening. Attached Figure Description

[0084] Figure 1 This is a schematic diagram of the device of the present invention;

[0085] Figure 2This is a flowchart illustrating the implementation of the method of the present invention. Detailed Implementation

[0086] To better understand the content of this invention, an embodiment is provided here.

[0087] Figure 1 This is a schematic diagram of the device of the present invention; Figure 2 This is a flowchart illustrating the implementation of the method of the present invention.

[0088] In a first aspect, this application discloses a cognitive impairment screening device based on an AR eye tracker and a brain-computer interface, comprising: an augmented reality scene generation module, an eye tracker data acquisition module, a brain-computer interface data acquisition module, and a cognitive impairment screening module;

[0089] The augmented reality scene generation module is used to generate and display augmented reality scenes containing cognitive tasks and to acquire task scene image information; the cognitive tasks include object recognition, spatial navigation, memory testing, etc.

[0090] The eye-tracking data acquisition module is used to collect a set of eye-tracking data information of the user in the augmented reality scene in real time. The set of eye-tracking data information includes a fixation point information sequence, a fixation time information sequence, and a saccade frequency information sequence. The fixation point information sequence includes fixation point location information; the fixation time information sequence includes fixation time information; and the saccade frequency information sequence includes saccade frequency information.

[0091] The brain-computer interface data acquisition module is used to acquire a set of electroencephalogram (EEG) signal information when the user completes a cognitive task. The set of EEG signal information includes an EEG signal sequence.

[0092] The cognitive impairment screening module is connected to the augmented reality scene generation module, the eye tracker data acquisition module, and the brain-computer interface data acquisition module, respectively. It is used to comprehensively evaluate and process the acquired task scene image information, eye track data information set, and brain electrical signal information set to obtain the user's cognitive impairment information. The cognitive impairment information includes whether cognitive impairment exists and the degree of cognitive impairment.

[0093] The task scene image information includes a real-time image of the task scene, target location information, and coordinates of the user's participating position. The target location information includes the coordinate range information of several targets in the task scene. The real-time image of the task scene includes the user's participating image. The coordinates of the participating position are the coordinates of the user's position when participating in the cognitive task in the task scene.

[0094] The augmented reality scene generation module includes an augmented reality scene display submodule, an image acquisition submodule, a location acquisition submodule, and a data acquisition submodule;

[0095] The location acquisition submodule is used to acquire the location coordinates of the user when participating in the cognitive task in the task scenario, and obtain the location coordinates of the user participating in the task.

[0096] The augmented reality scene display submodule is used to generate and display an augmented reality scene containing cognitive tasks and to collect target location information; the augmented reality scene contains targets that require user interaction when participating in completing cognitive tasks;

[0097] The image acquisition submodule is used to acquire real-time images of the task scene when the user participates in completing the cognitive task in the task scene;

[0098] The data acquisition submodule is used to fuse the acquired real-time images of the task scene, target location information, and coordinates of the locations of the participants in the task to obtain task scene image information, and then send the task scene image information to the cognitive impairment screening module.

[0099] The cognitive impairment screening module is used to comprehensively evaluate and process the collected task scene image information, eye movement data information set, and electroencephalogram (EEG) signal information set to obtain the user's cognitive impairment information, including:

[0100] The task scene image information is evaluated and processed to obtain the user participation task evaluation result information;

[0101] The eye-tracking data set is evaluated and processed to obtain an eye-tracking evaluation result value;

[0102] The set of EEG signal information is evaluated and processed to obtain EEG evaluation result values;

[0103] Using a preset first weight vector, the user participation task evaluation result information, eye movement evaluation result value, and electroencephalogram evaluation result value are weighted and summed to obtain the cognitive impairment evaluation value;

[0104] Determine whether the cognitive impairment assessment value is greater than a set cognitive discrimination threshold to obtain a cognitive discrimination result; if the cognitive discrimination result is greater than, determine that the user has a cognitive impairment, and determine the degree of cognitive impairment information in the cognitive impairment information as the cognitive impairment assessment value; if the cognitive discrimination result is not greater than, determine that the user does not have a cognitive impairment.

[0105] The user participation task evaluation result information, eye movement evaluation result value, and electroencephalogram evaluation result value are weighted and summed using a preset first weight vector. The preset first weight vector can be 0.3, 0.4, or 0.3.

[0106] The location acquisition submodule can be implemented using differential GPS;

[0107] The process of evaluating the eye movement data set to obtain an eye movement evaluation result value includes:

[0108] The gaze point information sequence in the eye movement data information set is subjected to deviation evaluation processing to obtain the deviation value of each gaze point information sequence;

[0109] The calculation expression for the deviation evaluation process is:

[0110]

[0111] Where t1 is a preset calculation parameter, a i Let (x0, y0, z0) be the deviation value of the i-th gaze point information sequence, and (x0, y0, z0) be the position coordinates of the preset target gaze point in the cognitive task. ij ,y ij ,z ij ) represents the coordinates of the j-th gaze point in the i-th gaze point information sequence, and M represents the number of gaze points contained in a gaze point information sequence;

[0112] The mean of the deviations of all fixation information sequences is used to obtain the fixation evaluation estimate;

[0113] A joint anomaly assessment process is performed on the fixation time information sequence and saccade frequency information sequence in the eye movement data information set to obtain a time-frequency assessment value;

[0114] The calculation expression for the joint anomaly assessment process is as follows:

[0115]

[0116]

[0117] Among them, c ij Let t0 and f0 be the time-frequency anomaly parameters of the i-th fixation time information in the j-th fixation time information sequence, respectively, and t0 and f0 be the standard fixation time value and standard saccade frequency value. ij and f ij Let pi be the ith fixation time information in the j-th fixation time information sequence and the ith saccade frequency information in the j-th saccade frequency information sequence, respectively, where pi is the constant of pi, and c is the saccade frequency information. jLet P be the time-frequency evaluation value of the j-th gaze time information sequence, and let P be the number of gaze time information sequences contained in a gaze time information sequence.

[0118] The mean time-frequency evaluation values ​​corresponding to all fixation time information sequences and saccade frequency information sequences are averaged to obtain the mean time-frequency evaluation value.

[0119] The eye movement assessment result is obtained by weighted summation of the gaze evaluation estimate and the mean of the time-frequency assessment.

[0120] The weight values ​​for the weighted summation of the gaze evaluation value and the time-frequency evaluation value are 0.7 and 0.3, respectively.

[0121] The evaluation and processing of the set of EEG signal information to obtain EEG evaluation result values ​​includes:

[0122] For each EEG signal sequence in the EEG signal information set, an evaluation value is calculated to obtain the corresponding evaluation value.

[0123] The average value of the evaluation values ​​of all EEG signal sequences is calculated to obtain the EEG evaluation result value;

[0124] The expression for calculating the evaluation quantity is:

[0125]

[0126] g=(G 23 -G 34 ) / (G 23 +G 34 ),

[0127] Where g is the evaluation value of the EEG signal sequence, Gabor() represents the Gabor transform operation, and x(n) represents the nth element of the EEG signal sequence. Let G represent the conjugate element of x(n), and let G represent the length of the EEG signal sequence. 23 G represents the first specific quantity of the EEG signal sequence. 34 This represents the second specific quantity of the EEG signal sequence.

[0128] The evaluation and processing of the task scene image information to obtain user participation task evaluation result information includes:

[0129] The real-time image of the task scene in the task scene image information is subjected to a first evaluation process to obtain first evaluation information;

[0130] A second evaluation process is performed on the participating task location coordinates and the target location information in the task scene image information to obtain second evaluation information;

[0131] Using a preset evaluation weighting vector, the first evaluation information and the second evaluation information are weighted and summed to obtain the user participation task evaluation result information.

[0132] The preset evaluation weighting vector can be 0.3 or 0.7.

[0133] The first evaluation process of the real-time image of the task scene in the task scene image information to obtain first evaluation information includes:

[0134] Human key point recognition is performed on the real-time image of the task scene in the task scene image information to obtain the key point spatial location matrix;

[0135] The difference calculation process is performed on the key point spatial location matrix and the standard participating task location matrix to obtain the first evaluation information;

[0136] The difference calculation process includes:

[0137] Subtract the key point spatial location matrix from the standard participating task location matrix to obtain the difference matrix;

[0138] The difference matrix is ​​subjected to a first standard calculation to obtain a first standard matrix;

[0139] The expression for the first standard calculation is:

[0140]

[0141] Where m represents the row dimension of the difference matrix, x ij z represents the element in the i-th row and j-th column of the difference matrix. ij This represents the element in the i-th row and j-th column of the first standard matrix;

[0142] Perform column optimization processing on the first standard matrix to obtain the optimal solution vector;

[0143] The optimal processing of the columns involves extracting the largest number from each column to form the optimal solution vector z. + The expression for the optimal solution vector is:

[0144]

[0145] Where n represents the column dimension of the difference matrix;

[0146] Perform column worst-case processing on the first standard matrix to obtain the worst-case solution vector;

[0147] The worst-case scenario processing described above involves extracting the smallest number from each column to form the worst-case solution vector z. - The expression for the worst-case solution vector is:

[0148]

[0149] The worst-case solution vector and the best-case solution vector are scored to obtain the first evaluation information; the expression for the scoring calculation is:

[0150]

[0151] In the formula, ω j ω is the preset importance weight for the j-th element; j s is the first evaluation information, obtained by pre-setting or by calculating the variance of each column of the first standard matrix.

[0152] The expression for calculating the positional deviation value is:

[0153] a1=|x0-x1| / |x0+x1|+|1-y0 / y1|+|exp(z0 / z1)-p1| / |z0+z1|,

[0154] Where p1 is the calculation parameter, a1 is the position deviation value, (x0, y0, z0) is the coordinate of the position of the participating task, and (x1, y1, z1) is the coordinate of the target position.

[0155] The expression for calculating the angle deviation value is:

[0156] c1 = arctan(|x0-x1| / |y0-y1|)

[0157] c2=arctan(|x0-x1| / |z0-z1|),

[0158] Where c1 and c2 are the angle deviation values;

[0159] The second evaluation process, which involves analyzing the coordinates of the participating task location and the target location information in the task scene image information to obtain second evaluation information, includes:

[0160] Determine the target location information corresponding to the coordinates of the participating task location;

[0161] Extract the set of coordinates of the boundary points from which the target location information is obtained;

[0162] Using the set of coordinates of the boundary points, a boundary matrix is ​​constructed;

[0163] Using the coordinates of the participating task locations, a target vector is constructed;

[0164] The boundary matrix and the target vector are subjected to difference evaluation processing to obtain the second evaluation information.

[0165] The difference assessment process includes

[0166] For each row vector of the target vector and the boundary matrix, perform cross-correlation operation to obtain the corresponding cross-correlation sequence;

[0167] Each cross-correlation sequence is subjected to time-frequency transformation to obtain the corresponding time-frequency sequence;

[0168] The time-frequency sequence is processed by distribution calculation to obtain the distribution sequence;

[0169] Logarithmic summation is performed on all distribution sequences to obtain the second evaluation information.

[0170] The expression for the distributed computation processing is:

[0171]

[0172] Where, r k,i E represents the value distribution of the i-th value in the k-th time-frequency sequence. k (i) represents the i-th value of the k-th time-frequency sequence, and N0 is the total number of elements in the time-frequency sequence;

[0173] The expression for the logarithmic accumulation process is:

[0174]

[0175] Where H represents the second evaluation information, and m represents the number of time-frequency sequences.

[0176] The set of coordinates of the boundary points includes the position coordinates of each boundary point of the target; the row vectors of the boundary matrix are the position coordinates of the boundary points.

[0177] The user feature extraction can be achieved using SURF feature point detection or corner detection algorithms.

[0178] The key points of the human body include the head, hands, and body;

[0179] The human body key point recognition can be achieved using the OpenPose algorithm in OpenCV.

[0180] The time-frequency transformation processing can be implemented using wavelet transform or short-time Fourier transform.

[0181] The augmented reality scene generation module can be implemented using augmented reality glasses; the eye tracker data acquisition module can be implemented using an eye tracker or a VR eye tracker developed by Xi'an Blue Brain Technology Co., Ltd.; and the brain-computer interface data acquisition module can be implemented using a brain-computer interface device.

[0182] A second aspect of this application discloses a cognitive impairment screening method based on an AR eye tracker and a brain-computer interface, implemented using the aforementioned cognitive impairment screening device based on an AR eye tracker and a brain-computer interface, comprising:

[0183] Using the augmented reality scene generation module, an augmented reality scene containing cognitive tasks is generated and displayed, and task scene image information is acquired.

[0184] The eye-tracking data acquisition module is used to collect a set of eye-tracking data information of the user in the augmented reality scene in real time;

[0185] Using the brain-computer interface data acquisition module, a set of brainwave signal information of the user when completing cognitive tasks is acquired;

[0186] The cognitive impairment screening module is used to comprehensively evaluate and process the collected task scene image information, eye movement data information set and EEG signal information set to obtain the user's cognitive impairment information.

[0187] The process of comprehensively evaluating and processing the collected task scene image information, eye movement data information set, and electroencephalogram (EEG) signal information set yields the user's cognitive impairment information, including:

[0188] The task scene image information is evaluated and processed to obtain the user participation task evaluation result information;

[0189] The eye-tracking data set is evaluated and processed to obtain an eye-tracking evaluation result value;

[0190] The set of EEG signal information is evaluated and processed to obtain EEG evaluation result values;

[0191] Using a preset first weight vector, the user participation task evaluation result information, eye movement evaluation result value, and electroencephalogram evaluation result value are weighted and summed to obtain the cognitive impairment evaluation value;

[0192] Determine whether the cognitive impairment assessment value is greater than a set cognitive discrimination threshold to obtain a cognitive discrimination result; if the cognitive discrimination result is greater than, determine that the user has a cognitive impairment, and determine the degree of cognitive impairment information in the cognitive impairment information as the cognitive impairment assessment value; if the cognitive discrimination result is not greater than, determine that the user does not have a cognitive impairment.

[0193] Using the information on the degree of cognitive impairment and the presence or absence of cognitive impairment, the user's cognitive impairment information is constructed.

[0194] The user participation task evaluation result information, eye movement evaluation result value, and electroencephalogram evaluation result value are weighted and summed using a preset first weight vector. The preset first weight vector can be 0.3, 0.4, or 0.3.

[0195] The location acquisition submodule can be implemented using differential GPS;

[0196] The process of evaluating the eye movement data set to obtain an eye movement evaluation result value includes:

[0197] The gaze point information sequence in the eye movement data information set is subjected to deviation evaluation processing to obtain the deviation value of each gaze point information sequence;

[0198] The calculation expression for the deviation evaluation process is:

[0199]

[0200] Where p1 is the calculation parameter, a i Let (x0, y0, z0) be the deviation value of the i-th gaze point information sequence, and (x0, y0, z0) be the position coordinates of the preset target gaze point in the cognitive task. i ,y i ,z i ) represents the coordinates of the j-th gaze point in the i-th gaze point information sequence, and M represents the number of gaze points contained in a gaze point information sequence;

[0201] The mean of the deviations of all fixation information sequences is used to obtain the fixation evaluation estimate;

[0202] A joint anomaly assessment process is performed on the fixation time information sequence and saccade frequency information sequence in the eye movement data information set to obtain a time-frequency assessment value;

[0203] The calculation expression for the joint anomaly assessment process is as follows:

[0204]

[0205]

[0206] Among them, c ij Let t0 and f0 be the time-frequency anomaly parameters of the i-th fixation time information in the j-th fixation time information sequence, respectively, and t0 and f0 be the standard fixation time value and standard saccade frequency value. ij and f ij Let pi be the ith fixation time information in the j-th fixation time information sequence and the ith saccade frequency information in the j-th saccade frequency information sequence, respectively, where pi is the constant of pi, and c is the saccade frequency information. jLet P be the time-frequency evaluation value of the j-th gaze time information sequence, and let P be the number of gaze time information sequences contained in a gaze time information sequence.

[0207] The mean time-frequency evaluation value corresponding to each fixation time information sequence and saccade frequency information sequence is calculated to obtain the mean time-frequency evaluation value.

[0208] The eye movement assessment result is obtained by weighted summation of the gaze evaluation estimate and the mean of the time-frequency assessment.

[0209] The weight values ​​for the weighted summation of the gaze evaluation value and the time-frequency evaluation value are 0.7 and 0.3, respectively.

[0210] The evaluation and processing of the set of EEG signal information to obtain EEG evaluation result values ​​includes:

[0211] For each EEG signal sequence in the EEG signal information set, an evaluation value is calculated to obtain the corresponding evaluation value.

[0212] The average value of the evaluation values ​​of all EEG signal sequences is calculated to obtain the EEG evaluation result value;

[0213] The expression for calculating the evaluation quantity is:

[0214]

[0215] g=(G 23 -G 34 ) / (G 23 +G 34 ),

[0216] Where g is the evaluation value of the EEG signal sequence, Gabor() represents the Gabor transform operation, and x(n) represents the nth element of the EEG signal sequence. Let G represent the conjugate element of x(n), and let G represent the length of the EEG signal sequence. 23 G represents the first specific quantity of the EEG signal sequence. 34 This represents the second specific quantity of the EEG signal sequence.

[0217] The evaluation and processing of the task scene image information to obtain user participation task evaluation result information includes:

[0218] The real-time image of the task scene in the task scene image information is subjected to a first evaluation process to obtain first evaluation information;

[0219] A second evaluation process is performed on the participating task location coordinates and the target location information in the task scene image information to obtain second evaluation information;

[0220] Using a preset evaluation weighting vector, the first evaluation information and the second evaluation information are weighted and summed to obtain the user participation task evaluation result information.

[0221] The preset evaluation weighting vector can be 0.3 or 0.7.

[0222] The first evaluation process of the real-time image of the task scene in the task scene image information to obtain first evaluation information includes:

[0223] Human key point recognition is performed on the real-time image of the task scene in the task scene image information to obtain the key point spatial location matrix;

[0224] The difference calculation process is performed on the key point spatial location matrix and the standard participating task location matrix to obtain the first evaluation information;

[0225] The difference calculation process includes:

[0226] Subtract the key point spatial location matrix from the standard participating task location matrix to obtain the difference matrix;

[0227] The difference matrix is ​​subjected to a first standard calculation to obtain a first standard matrix;

[0228] The expression for the first standard calculation is:

[0229]

[0230] Where m represents the row dimension of the difference matrix, x ij z represents the element in the i-th row and j-th column of the difference matrix. ij This represents the element in the i-th row and j-th column of the first standard matrix;

[0231] Perform column optimization processing on the first standard matrix to obtain the optimal solution vector;

[0232] The optimal processing of the columns involves extracting the largest number from each column to form the optimal solution vector z. + The expression for the optimal solution vector is:

[0233]

[0234] Where n represents the column dimension of the difference matrix;

[0235] Perform column worst-case processing on the first standard matrix to obtain the worst-case solution vector;

[0236] The worst-case scenario processing described above involves extracting the smallest number from each column to form the worst-case solution vector z. - The expression for the worst-case solution vector is:

[0237]

[0238] The worst-case solution vector and the best-case solution vector are scored to obtain the first evaluation information; the expression for the scoring calculation is:

[0239]

[0240] In the formula, ω j ω is the preset importance weight for the j-th element; j s is the first evaluation information, obtained by pre-setting or by calculating the variance of each column of the first standard matrix.

[0241] The expression for calculating the positional deviation value is:

[0242] a1=|x0-x1| / |x0+x1|+|1-y0 / y1|+|exp(z0 / z1)-p1| / |z0+z1|,

[0243] Where p1 is the calculation parameter, a1 is the position deviation value, (x0, y0, z0) is the coordinate of the position of the participating task, and (x1, y1, z1) is the coordinate of the target position.

[0244] The expression for calculating the angle deviation value is:

[0245] c1 = arctan(|x0-x1| / |y0-y1|)

[0246] c2=arctan(|x0-x1| / |z0-z1|),

[0247] Where c1 and c2 are the angle deviation values;

[0248] The second evaluation process, which involves analyzing the coordinates of the participating task location and the target location information in the task scene image information to obtain second evaluation information, includes:

[0249] Determine the target location information corresponding to the coordinates of the participating task location;

[0250] Extract the set of coordinates of the boundary points from which the target location information is obtained;

[0251] Using the set of coordinates of the boundary points, a boundary matrix is ​​constructed;

[0252] Using the coordinates of the participating task locations, a target vector is constructed;

[0253] The boundary matrix and the target vector are subjected to difference evaluation processing to obtain the second evaluation information.

[0254] The difference assessment process includes

[0255] For each row vector of the target vector and the boundary matrix, perform cross-correlation operation to obtain the corresponding cross-correlation sequence;

[0256] Each cross-correlation sequence is subjected to time-frequency transformation to obtain the corresponding time-frequency sequence;

[0257] The time-frequency sequence is processed by distribution calculation to obtain the distribution sequence;

[0258] Logarithmic summation is performed on all distribution sequences to obtain the second evaluation information.

[0259] The expression for the distributed computation processing is:

[0260]

[0261] Where, r k,i E represents the value distribution of the i-th value in the k-th time-frequency sequence. k (i) represents the i-th value of the k-th time-frequency sequence, and N0 is the total number of elements in the time-frequency sequence;

[0262] The expression for the logarithmic accumulation process is:

[0263]

[0264] Where H represents the second evaluation information, and m represents the number of time-frequency sequences.

[0265] The set of coordinates of the boundary points includes the position coordinates of each boundary point of the target; the row vectors of the boundary matrix are the position coordinates of the boundary points.

[0266] The user feature extraction can be achieved using SURF feature point detection or corner detection algorithms.

[0267] The key points of the human body include the head, hands, and body;

[0268] The human body key point recognition can be achieved using the OpenPose algorithm in OpenCV.

[0269] The time-frequency transformation processing can be implemented using wavelet transform or short-time Fourier transform.

[0270] The augmented reality scene generation module can be implemented using augmented reality glasses; the eye tracker data acquisition module can be implemented using an eye tracker or a VR eye tracker developed by Xi'an Blue Brain Technology Co., Ltd.; and the brain-computer interface data acquisition module can be implemented using a brain-computer interface device.

[0271] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A cognitive impairment screening device based on AR eye tracker and brain-computer interface, characterized in that, include: Augmented reality scene generation module, eye tracker data acquisition module, brain-computer interface data acquisition module, and cognitive impairment screening module; The augmented reality scene generation module is used to generate and display augmented reality scenes containing cognitive tasks and to acquire task scene image information. The eye tracker data acquisition module is used to collect a set of eye movement data information in real time when the user completes cognitive tasks in an augmented reality scene; the set of eye movement data information includes a fixation point information sequence, a fixation time information sequence, and a saccade frequency information sequence; the fixation point information sequence includes fixation point location information; the fixation time information sequence includes fixation time information; the saccade frequency information sequence includes saccade frequency information. The brain-computer interface data acquisition module is used to acquire in real time a set of brainwave signal information when the user completes cognitive tasks in an augmented reality scenario; the set of brainwave signal information includes brainwave signal sequences. The cognitive impairment screening module is connected to the augmented reality scene generation module, the eye tracker data acquisition module, and the brain-computer interface data acquisition module, respectively. It is used to comprehensively evaluate and process the acquired task scene image information, eye track data information set, and brain electrical signal information set to obtain the user's cognitive impairment information. The cognitive impairment information includes whether cognitive impairment exists and the degree of cognitive impairment.

2. The cognitive impairment screening device based on AR eye tracker and brain-computer interface as described in claim 1, characterized in that, The task scene image information includes real-time images of the task scene, target location information, and coordinates of the locations of the participants in the task; the target location information includes the coordinate range information of several targets in the task scene. The real-time images of the task scene include images of the user participating in the cognitive task; the coordinates of the participating task location are the coordinates of the user's location in the task scene when participating in the cognitive task; each target location information has corresponding participating task location coordinates.

3. The cognitive impairment screening device based on AR eye tracker and brain-computer interface as described in claim 2, characterized in that, The augmented reality scene generation module includes an augmented reality scene display submodule, an image acquisition submodule, a location acquisition submodule, and a data acquisition submodule; The location acquisition submodule is used to acquire the location coordinates of the user when participating in the cognitive task in the task scenario, and obtain the location coordinates of the user participating in the task. The augmented reality scene display submodule is used to generate and display an augmented reality scene containing cognitive tasks and to collect target location information; the augmented reality scene contains targets that require user interaction when participating in completing cognitive tasks; The image acquisition submodule is used to acquire real-time images of the task scene when the user participates in completing the cognitive task in the task scene; The data acquisition submodule is used to fuse the acquired real-time images of the task scene, target location information, and coordinates of the locations of the participants in the task to obtain task scene image information, and then send the task scene image information to the cognitive impairment screening module.

4. The cognitive impairment screening device based on AR eye tracker and brain-computer interface as described in claim 2, characterized in that, The cognitive impairment screening module is used to comprehensively evaluate and process the collected task scene image information, eye movement data information set, and electroencephalogram (EEG) signal information set to obtain the user's cognitive impairment information, including: The cognitive impairment screening module evaluates and processes the task scene image information to obtain user participation task evaluation result information; The eye-tracking data set is evaluated and processed to obtain an eye-tracking evaluation result value; The set of EEG signal information is evaluated and processed to obtain EEG evaluation result values; Using a preset first weight vector, the user participation task evaluation result information, eye movement evaluation result value, and electroencephalogram evaluation result value are weighted and summed to obtain the cognitive impairment evaluation value; Determine whether the cognitive impairment assessment value is greater than a set cognitive discrimination threshold to obtain a cognitive discrimination result; if the cognitive discrimination result is greater than, determine that the user has a cognitive impairment, and determine the degree of cognitive impairment information in the cognitive impairment information as the cognitive impairment assessment value; if the cognitive discrimination result is not greater than, determine that the user does not have a cognitive impairment. Using the information on the degree of cognitive impairment and the presence or absence of cognitive impairment, the user's cognitive impairment information is constructed.

Images