Cognitive disorder screening method and device based on AR eye tracker and brain-computer interface

Abstract

The invention discloses a cognitive impairment screening device and method based on an AR eye tracker and a brain-computer interface. The device comprises an augmented reality scene generation module, an eye tracker data acquisition module, a brain-computer interface data acquisition module and a cognitive impairment screening module. The augmented reality scene generation module is used for generating and displaying an augmented reality scene containing a cognitive task and acquiring task scene image information; the eye tracker data acquisition module is used for acquiring an eye movement data information set when a user completes a cognitive task in an augmented reality scene in real time; the brain-computer interface data acquisition module is used for acquiring an electroencephalogram signal information set in real time when a user completes a cognitive task in an augmented reality scene; and the cognitive impairment screening module is used for carrying out comprehensive evaluation processing on the acquired task scene image information, the eye movement data information set and the electroencephalogram signal information set to obtain cognitive impairment information of the user.

Description

Technical Field

[0001] The present invention relates to the fields of augmented reality and brain cognition, and in particular to a method and device for screening cognitive impairment based on an AR eye tracker and a brain-computer interface. Background Art

[0002] With the acceleration of the trend of aging society, more and more people are suffering from cognitive impairment. Cognitive impairment is a common neurological abnormality, including dementia, mild cognitive impairment, etc. Early and accurate screening is crucial for the diagnosis and treatment of cognitive impairment. At present, commonly used screening methods for cognitive impairment include neuropsychological tests, imaging examinations, etc., but these methods have certain limitations, such as strong subjectivity, complex operation, and insensitivity to early lesions.

[0003] How to achieve rapid, accurate and simple early screening of cognitive impairment is an urgent problem that needs to be solved in the current context of an aging society. Summary of the invention

[0004] The present invention mainly solves the problem of how to achieve rapid, accurate and simple early screening of cognitive impairment. The present invention discloses a method and device for screening cognitive impairment based on an AR eye tracker and a brain-computer interface.

[0005] The purpose of the present invention is to provide a method and device for screening cognitive impairment based on an augmented reality (AR) eye tracker and a brain-computer interface, so as to address the limitations of existing cognitive impairment screening methods and improve the accuracy, objectivity and efficiency of screening.

[0006] In a first aspect of the embodiment of the present application, a cognitive impairment screening device based on an AR eye tracker and a brain-computer interface is disclosed, characterized in that it includes: an augmented reality scene generation module, an eye tracker data acquisition module, a brain-computer interface data acquisition module, and a cognitive impairment screening module;

[0007] The augmented reality scene generation module is used to generate and display an augmented reality scene containing a cognitive task, and collect task scene image information;

[0008] The eye tracker data acquisition module is used to acquire in real time a set of eye movement data information when the user completes a cognitive task in an augmented reality scene; the eye movement data information set includes a fixation point information sequence, a fixation time information sequence, and an eye saccade frequency information sequence; the fixation point information sequence includes fixation point position information; the fixation time information sequence includes fixation time information; the eye saccade frequency information sequence includes eye saccade frequency information;

[0009] The brain-computer interface data acquisition module is used to acquire in real time a set of EEG signal information when the user completes a cognitive task in an augmented reality scene; the set of EEG signal information includes an EEG signal sequence;

[0010] The cognitive impairment screening module is connected to the augmented reality scene generation module, the eye tracker data acquisition module, and the brain-computer interface data acquisition module respectively, and is used to comprehensively evaluate and process the collected task scene image information, eye movement data information set, and electroencephalogram signal information set to obtain the user's cognitive impairment information; the cognitive impairment information includes whether there is cognitive impairment and the degree of cognitive impairment.

[0011] The task scene image information includes a real-time image of the task scene, target position information and participating task position coordinates; the target position information includes position coordinate range information of several targets in the task scene; the task scene real-time image includes an image of the user participating in a cognitive task; the participating task position coordinates are the position coordinates of the user when participating in completing a cognitive task in the task scene; each target position information has corresponding participating task position coordinates.

[0012] The augmented reality scene generation module includes an augmented reality scene display submodule, an image acquisition submodule, a position acquisition submodule, and a data acquisition submodule;

[0013] The position acquisition submodule is used to acquire the position coordinates of the user when participating in the cognitive task in the task scene to obtain the position coordinates of the participating task;

[0014] The augmented reality scene display submodule is used to generate and display an augmented reality scene including a cognitive task, and collect target location information; the augmented reality scene includes a target that requires the user to interact when participating in completing the cognitive task;

[0015] The image acquisition submodule is used to acquire a real-time image of the task scene when the user participates in completing the cognitive task in the task scene;

[0016] The data acquisition submodule is used to fuse the collected real-time image of the task scene, the target position information and the coordinates of the participating task positions to obtain the task scene image information, and send the task scene image information to the cognitive impairment screening module.

[0017] The cognitive impairment screening module is used to comprehensively evaluate and process the collected task scene image information, eye movement data information set and EEG signal information set to obtain the user's cognitive impairment information, including:

[0018] The cognitive impairment screening module evaluates and processes the task scene image information to obtain user participation task evaluation result information;

[0019] Performing evaluation processing on the eye movement data information set to obtain an eye movement evaluation result value;

[0020] Evaluating and processing the EEG signal information set to obtain an EEG evaluation result value;

[0021] Using a preset first weight vector, weighted sum processing is performed on the user's task participation evaluation result information, the eye movement evaluation result value, and the electroencephalogram evaluation result value to obtain a cognitive impairment evaluation value;

[0022] Determine whether the cognitive impairment assessment value is greater than a set cognitive discrimination threshold value, and obtain a cognitive discrimination result; if the cognitive discrimination result is greater than, determine that the user has cognitive impairment, and determine the cognitive impairment degree information in the cognitive impairment information as the cognitive impairment assessment value; if the cognitive discrimination result is not greater than, determine that the user does not have cognitive impairment;

[0023] The cognitive impairment information of the user is constructed by using the cognitive impairment degree information and the information on whether cognitive impairment exists.

[0024] In a second aspect of an embodiment of the present invention, a method for screening cognitive impairment based on an AR eye tracker and a brain-computer interface is disclosed, which is implemented using the device for screening cognitive impairment based on an AR eye tracker and a brain-computer interface, and includes:

[0025] Using the augmented reality scene generation module, an augmented reality scene including a cognitive task is generated and displayed, and task scene image information is acquired;

[0026] Using the eye tracker data acquisition module, a collection of eye movement data information of the user in the augmented reality scene is acquired in real time;

[0027] Using the brain-computer interface data acquisition module, a collection of EEG signal information of the user when completing a cognitive task is acquired;

[0028] The cognitive impairment screening module is used to perform comprehensive evaluation and processing on the collected task scene image information, eye movement data information set and electroencephalogram signal information set to obtain the user's cognitive impairment information.

[0029] The step of comprehensively evaluating and processing the collected task scene image information, eye movement data information set, and EEG signal information set to obtain the user's cognitive impairment information includes:

[0030] Performing evaluation processing on the task scene image information to obtain user participation task evaluation result information;

[0031] Performing evaluation processing on the eye movement data information set to obtain an eye movement evaluation result value;

[0032] Evaluating and processing the EEG signal information set to obtain an EEG evaluation result value;

[0033] Using a preset first weight vector, weighted sum processing is performed on the user's task participation evaluation result information, the eye movement evaluation result value, and the electroencephalogram evaluation result value to obtain a cognitive impairment evaluation value;

[0034] Determine whether the cognitive impairment assessment value is greater than a set cognitive discrimination threshold value, and obtain a cognitive discrimination result; if the cognitive discrimination result is greater than, determine that the user has cognitive impairment, and determine the cognitive impairment degree information in the cognitive impairment information as the cognitive impairment assessment value; if the cognitive discrimination result is not greater than, determine that the user does not have cognitive impairment;

[0035] The cognitive impairment information of the user is constructed by using the cognitive impairment degree information and the information on whether cognitive impairment exists.

[0036] The step of evaluating the eye movement data information set to obtain an eye movement evaluation result value includes:

[0037] Performing deviation evaluation processing on the fixation point information sequence in the eye movement data information set to obtain a deviation value of each fixation point information sequence;

[0038] The calculation expression of the deviation evaluation process is:

[0039]

[0040] Among them, t1 is the preset calculation parameter, a i is the deviation value of the i-th fixation point information sequence, (x0, y0, z0) is the position coordinate of the preset target fixation point in the cognitive task, (x ij ,y ij ,z ij ) is the jth fixation point position coordinate of the ith fixation point information sequence, M is the number of fixations contained in a fixation point information sequence;

[0041] The deviation values ​​of all fixation point information sequences are averaged to obtain the fixation evaluation estimate;

[0042] Performing joint abnormality assessment processing on the fixation time information sequence and the eye movement frequency information sequence in the eye movement data information set to obtain a time-frequency assessment value;

[0043] The calculation expression of the joint anomaly assessment process is:

[0044]

[0045]

[0046] Among them, c ij is the time-frequency abnormality parameter of the ith fixation time information in the jth fixation time information sequence, t0 and f0 are the standard fixation time value and the standard eye saccade frequency value respectively, t ij and f ij are the i-th fixation time information in the j-th fixation time information sequence and the i-th eye saccade frequency information in the j-th eye saccade frequency information sequence, pi is the pi constant, c j is the time-frequency evaluation value of the jth fixation time information sequence, and P is the number of fixation time information contained in a fixation time information sequence;

[0047] The time-frequency evaluation values ​​corresponding to all fixation time information sequences and saccade frequency information sequences are averaged to obtain the time-frequency evaluation mean;

[0048] The gaze comment estimation value and the time-frequency evaluation mean are weightedly summed to obtain the eye movement evaluation result value.

[0049] The task scene image information is evaluated and processed to obtain the user participation task evaluation result information, including:

[0050] Performing a first evaluation process on the task scene real-time image in the task scene image information to obtain first evaluation information;

[0051] Performing a second evaluation process on the participating task position coordinates and the target position information in the task scene image information to obtain second evaluation information;

[0052] The first evaluation information and the second evaluation information are weighted and summed using a preset evaluation weight vector to obtain user participation task evaluation result information.

[0053] The performing a first evaluation process on the task scene real-time image in the task scene image information to obtain first evaluation information includes:

[0054] Performing human body key point recognition on the task scene real-time image in the task scene image information to obtain a key point spatial position matrix;

[0055] Performing difference calculation processing on the key point spatial position matrix and the standard participation task position matrix to obtain first evaluation information;

[0056] The difference calculation process includes:

[0057] Subtracting the key point spatial position matrix from the standard participation task position matrix to obtain a difference matrix;

[0058] Performing a first standard calculation process on the difference matrix to obtain a first standard matrix;

[0059] The expression of the first standard calculation process is:

[0060]

[0061] Where m represents the row dimension of the difference matrix, x ij represents the element of the i-th row and j-th column of the difference matrix, z ij represents the element of the i-th row and j-th column of the first standard matrix;

[0062] Performing column optimization processing on the first standard matrix to obtain an optimal solution vector;

[0063] The column optimal processing is to take out the largest number in each column to form the optimal solution vector z + ; The expression of the optimal solution vector is:

[0064]

[0065] Where n represents the column dimension of the difference matrix;

[0066] Performing column worst processing on the first standard matrix to obtain a worst solution vector;

[0067] The worst column processing is to take out the smallest number in each column to form the worst solution vector z - ; The expression of the worst solution vector is:

[0068]

[0069] The worst solution vector and the best solution vector are scored and calculated to obtain first evaluation information; the score calculation is expressed as:

[0070]

[0071] In the formula, ω j is the preset j-th importance weight; ω j It is obtained by presetting or by calculating the variance value of each column of the first standard matrix, and s is the first evaluation information.

[0072] The performing a second evaluation process on the participating task position coordinates and the target position information in the task scene image information to obtain second evaluation information includes:

[0073] Determine the target location information corresponding to the participating task location coordinates;

[0074] Extracting a coordinate set of boundary points of the target location information;

[0075] Using the coordinate set of the boundary points, a boundary matrix is ​​constructed;

[0076] Using the participating task position coordinates, construct a target vector;

[0077] A difference evaluation process is performed on the boundary matrix and the target vector to obtain second evaluation information.

[0078] The beneficial effects of the present invention are:

[0079] This invention introduces augmented reality technology, eye tracker, and brain-computer interface technology into the field of cognitive impairment screening. Augmented reality technology can create a realistic virtual environment, simulate various daily life scenes, increase the fun and participation of the test, and also reduce adverse reactions such as dizziness caused by virtual reality technology;

[0080] The present invention uses an eye tracker to monitor the user's eye movement trajectory in real time and obtain information such as visual attention; brain-computer interface technology can directly detect the brain's neural activity and reflect the state of cognitive function; by establishing special evaluation variables based on the characteristics of brain-computer interface data, a rapid and accurate evaluation of the user's brain data is achieved;

[0081] The present invention adopts a multi-level evaluation method for eye movement data, obtains a gaze comment estimation value by performing deviation evaluation processing on a gaze point information sequence; obtains a time-frequency evaluation value by performing a joint abnormality evaluation processing on a gaze time information sequence and an eye saccade frequency information sequence in the eye movement data information set; and finally, fuses multiple types of evaluation values ​​to obtain an eye movement evaluation result value.

[0082] The present invention gives full play to the advantage of augmented reality technology in obtaining the image and position coordinates of the user. By collecting the real-time image of the task scene, the target position information and the position coordinates of the participating task, a special first evaluation processing method and a second evaluation processing method are established to respectively evaluate the action standard and position accuracy of the user when participating in completing the cognitive task, and obtain the evaluation result information of the user's participation task, thereby realizing the cognitive impairment assessment from the macro action domain.

[0083] The present invention realizes automated screening for cognitive impairment assessment by effectively integrating the three types of assessment results: eye, EEG and movement, eliminates the influence of subjective factors, and improves the accuracy, objectivity and efficiency of screening. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 It is a schematic diagram of the composition of the device of the present invention;

[0085] Figure 2It is a flow chart for implementing the method of the present invention. DETAILED DESCRIPTION

[0086] In order to better understand the content of the present invention, an embodiment is given here.

[0087] Figure 1 It is a schematic diagram of the composition of the device of the present invention; Figure 2 It is a flow chart for implementing the method of the present invention.

[0088] In a first aspect of the embodiment of the present application, a cognitive impairment screening device based on an AR eye tracker and a brain-computer interface is disclosed, comprising: an augmented reality scene generation module, an eye tracker data acquisition module, a brain-computer interface data acquisition module, and a cognitive impairment screening module;

[0089] The augmented reality scene generation module is used to generate and display an augmented reality scene containing cognitive tasks, and collect task scene image information; the cognitive tasks include object recognition, spatial navigation, memory test, etc.;

[0090] The eye tracker data acquisition module is used to acquire the user's eye movement data information set in real time in the augmented reality scene; the eye movement data information set includes a fixation point information sequence, a fixation time information sequence, an eye saccade frequency information sequence, etc. The fixation point information sequence includes fixation point position information; the fixation time information sequence includes fixation time information; the eye saccade frequency information sequence includes eye saccade frequency information;

[0091] The brain-computer interface data acquisition module is used to acquire a collection of EEG signal information when the user completes a cognitive task. The collection of EEG signal information includes an EEG signal sequence;

[0092] The cognitive impairment screening module is connected to the augmented reality scene generation module, the eye tracker data acquisition module, and the brain-computer interface data acquisition module, respectively, and is used to perform comprehensive evaluation and processing on the collected task scene image information, eye movement data information set, and EEG signal information set to obtain the user's cognitive impairment information; the cognitive impairment information includes whether there is cognitive impairment and the degree of cognitive impairment;

[0093] The task scene image information includes a real-time image of the task scene, target location information and participating task location coordinates; the target location information includes location coordinate range information of several targets in the task scene; the real-time image of the task scene includes a user's participating task image; the participating task location coordinates are the location coordinates of the user when participating in completing the cognitive task in the task scene;

[0094] The augmented reality scene generation module includes an augmented reality scene display submodule, an image acquisition submodule, a position acquisition submodule, and a data acquisition submodule;

[0095] The position acquisition submodule is used to acquire the position coordinates of the user when participating in the cognitive task in the task scene to obtain the position coordinates of the participating task;

[0096] The augmented reality scene display submodule is used to generate and display an augmented reality scene including a cognitive task, and collect target location information; the augmented reality scene includes a target that requires the user to interact when participating in completing the cognitive task;

[0097] The image acquisition submodule is used to acquire a real-time image of the task scene when the user participates in completing the cognitive task in the task scene;

[0098] The data acquisition submodule is used to fuse the collected real-time image of the task scene, the target position information and the coordinates of the participating task positions to obtain the task scene image information, and send the task scene image information to the cognitive impairment screening module;

[0099] The cognitive impairment screening module is used to comprehensively evaluate and process the collected task scene image information, eye movement data information set and EEG signal information set to obtain the user's cognitive impairment information, including:

[0100] Performing evaluation processing on the task scene image information to obtain user participation task evaluation result information;

[0101] Performing evaluation processing on the eye movement data information set to obtain an eye movement evaluation result value;

[0102] Evaluating and processing the EEG signal information set to obtain an EEG evaluation result value;

[0103] Using a preset first weight vector, weighted sum processing is performed on the user's task participation evaluation result information, the eye movement evaluation result value, and the electroencephalogram evaluation result value to obtain a cognitive impairment evaluation value;

[0104] Determine whether the cognitive impairment assessment value is greater than a set cognitive discrimination threshold to obtain a cognitive discrimination result; if the cognitive discrimination result is greater than, determine that the user has cognitive impairment, and determine the cognitive impairment degree information in the cognitive impairment information as the cognitive impairment assessment value; if the cognitive discrimination result is not greater than, determine that the user does not have cognitive impairment.

[0105] The preset first weight vector is used to perform weighted summation processing on the user participation task evaluation result information, eye movement evaluation result value, and EEG evaluation result value, wherein the preset first weight vector can be 0.3, 0.4, or 0.3.

[0106] The position acquisition submodule can be implemented by using differential GPS;

[0107] The step of evaluating the eye movement data information set to obtain an eye movement evaluation result value includes:

[0108] Performing deviation evaluation processing on the fixation point information sequence in the eye movement data information set to obtain a deviation value of each fixation point information sequence;

[0109] The calculation expression of the deviation evaluation process is:

[0110]

[0111] Among them, t1 is the preset calculation parameter, a i is the deviation value of the i-th fixation point information sequence, (x0, y0, z0) is the position coordinate of the preset target fixation point in the cognitive task, (x ij ,y ij ,z ij ) is the jth fixation point position coordinate of the ith fixation point information sequence, M is the number of fixations contained in a fixation point information sequence;

[0112] The deviation values ​​of all fixation point information sequences are averaged to obtain the fixation evaluation estimate;

[0113] Performing joint abnormality assessment processing on the fixation time information sequence and the eye movement frequency information sequence in the eye movement data information set to obtain a time-frequency assessment value;

[0114] The calculation expression of the joint anomaly assessment process is:

[0115]

[0116]

[0117] Among them, c ij is the time-frequency abnormality parameter of the ith fixation time information in the jth fixation time information sequence, t0 and f0 are the standard fixation time value and the standard eye saccade frequency value respectively, t ij and f ij are the i-th fixation time information in the j-th fixation time information sequence and the i-th eye saccade frequency information in the j-th eye saccade frequency information sequence, pi is the pi constant, c jis the time-frequency evaluation value of the jth fixation time information sequence, and P is the number of fixation time information contained in a fixation time information sequence;

[0118] The time-frequency evaluation values ​​corresponding to all fixation time information sequences and saccade frequency information sequences are averaged to obtain the time-frequency evaluation mean;

[0119] The gaze evaluation value and the time-frequency evaluation mean are weighted and summed to obtain the eye movement evaluation result value.

[0120] The weight values ​​for weighted summation of the gaze comment estimation value and the time-frequency evaluation value are 0.7 and 0.3;

[0121] The step of evaluating and processing the EEG signal information set to obtain an EEG evaluation result value includes:

[0122] Performing evaluation amount calculation processing on each EEG signal sequence of the EEG signal information set to obtain a corresponding evaluation value;

[0123] The evaluation values ​​of all EEG signal sequences are averaged to obtain the EEG evaluation result value;

[0124] The expression for the evaluation amount calculation process is:

[0125]

[0126] g=(G 23 -G 34 ) / (G 23 +G 34 ),

[0127] Where g is the evaluation value of the EEG signal sequence, Gabor() represents the Gabor transform operation, x(n) represents the nth element of the EEG signal sequence, represents the conjugate element of x(n), G represents the length of the EEG signal sequence, G 23 Represents the first sub-specific quantity of the EEG signal sequence, G 34 Represents the second sub-specific quantity of the EEG signal sequence.

[0128] The task scene image information is evaluated and processed to obtain the user participation task evaluation result information, including:

[0129] Performing a first evaluation process on the task scene real-time image in the task scene image information to obtain first evaluation information;

[0130] Performing a second evaluation process on the participating task position coordinates and the target position information in the task scene image information to obtain second evaluation information;

[0131] The first evaluation information and the second evaluation information are weighted and summed using a preset evaluation weight vector to obtain user participation task evaluation result information.

[0132] The preset evaluation weight vector may be 0.3 and 0.7.

[0133] The performing a first evaluation process on the task scene real-time image in the task scene image information to obtain first evaluation information includes:

[0134] Performing human body key point recognition on the task scene real-time image in the task scene image information to obtain a key point spatial position matrix;

[0135] Performing difference calculation processing on the key point spatial position matrix and the standard participation task position matrix to obtain first evaluation information;

[0136] The difference calculation process includes:

[0137] Subtracting the key point spatial position matrix from the standard participation task position matrix to obtain a difference matrix;

[0138] Performing a first standard calculation process on the difference matrix to obtain a first standard matrix;

[0139] The expression of the first standard calculation process is:

[0140]

[0141] Where m represents the row dimension of the difference matrix, x ij represents the element of the i-th row and j-th column of the difference matrix, z ij represents the element of the i-th row and j-th column of the first standard matrix;

[0142] Performing column optimization processing on the first standard matrix to obtain an optimal solution vector;

[0143] The column optimal processing is to take out the largest number in each column to form the optimal solution vector z + ; The expression of the optimal solution vector is:

[0144]

[0145] Where n represents the column dimension of the difference matrix;

[0146] Performing column worst processing on the first standard matrix to obtain a worst solution vector;

[0147] The worst column processing is to take out the smallest number in each column to form the worst solution vector z - ; The expression of the worst solution vector is:

[0148]

[0149] The worst solution vector and the best solution vector are scored and calculated to obtain first evaluation information; the score calculation is expressed as:

[0150]

[0151] In the formula, ω j is the preset j-th importance weight; ω j It is obtained by presetting or by calculating the variance value of each column of the first standard matrix, and s is the first evaluation information.

[0152] The calculation expression of the position deviation value is:

[0153] a1=|x0-x1| / |x0+x1|+|1-y0 / y1|+|exp(z0 / z1)-p1| / |z0+z1|,

[0154] Wherein, p1 is the calculation parameter, a1 is the position deviation value, (x0, y0, z0) is the participating task position coordinate, and (x1, y1, z1) is the target position coordinate;

[0155] The calculation expression of the angle deviation value is:

[0156] c1=arctan(|x0-x1| / |y0-y1|)

[0157] c2=arctan(|x0-x1| / |z0-z1|),

[0158] Among them, c1 and c2 are angle deviation values;

[0159] The performing a second evaluation process on the participating task position coordinates and the target position information in the task scene image information to obtain second evaluation information includes:

[0160] Determine the target location information corresponding to the participating task location coordinates;

[0161] Extracting a coordinate set of boundary points of the target location information;

[0162] Using the coordinate set of the boundary points, a boundary matrix is ​​constructed;

[0163] Using the participating task position coordinates, construct a target vector;

[0164] A difference evaluation process is performed on the boundary matrix and the target vector to obtain second evaluation information.

[0165] The difference assessment process includes

[0166] Performing cross-correlation operations on the target vector and each row vector of the boundary matrix to obtain corresponding cross-correlation sequences;

[0167] Perform time-frequency transformation on each cross-correlation sequence to obtain the corresponding time-frequency sequence;

[0168] Perform distribution calculation on the time-frequency series to obtain a distribution sequence;

[0169] Perform logarithmic accumulation processing on all distribution sequences to obtain second evaluation information.

[0170] The expression of the distribution calculation process is:

[0171]

[0172] Among them, r k,i represents the distribution of the i-th value of the k-th time-frequency series, E k (i) represents the i-th value of the k-th time-frequency sequence, and N0 is the total number of elements in the time-frequency sequence;

[0173] The expression of the logarithmic accumulation process is:

[0174]

[0175] Wherein, H is the second evaluation information, and m is the number of time-frequency sequences.

[0176] The coordinate set of the boundary points includes the position coordinates of each boundary point of the target; the row vector of the boundary matrix is ​​the position coordinates of the boundary points.

[0177] The user feature extraction may be implemented by using a SURF feature point detection or a corner point detection algorithm.

[0178] The key points of the human body include the head, hands, and body;

[0179] The human body key point recognition can be implemented by using the OpenPose algorithm in OpenCV.

[0180] The time-frequency transformation process can be implemented by wavelet transformation or short-time Fourier transformation.

[0181] The augmented reality scene generation module can be implemented by augmented reality glasses; the eye tracker data acquisition module can be implemented by an eye tracker or a VR eye tracker developed by Xi'an Blue Brain Technology Co., Ltd.; the brain-computer interface data acquisition module can be implemented by a brain-computer interface device.

[0182] In a second aspect of the embodiment of the present application, a method for screening cognitive impairment based on an AR eye tracker and a brain-computer interface is disclosed, which is implemented using the device for screening cognitive impairment based on an AR eye tracker and a brain-computer interface, including:

[0183] Using the augmented reality scene generation module, an augmented reality scene including a cognitive task is generated and displayed, and task scene image information is acquired;

[0184] Using the eye tracker data acquisition module, a collection of eye movement data information of the user in the augmented reality scene is acquired in real time;

[0185] Using the brain-computer interface data acquisition module, a collection of EEG signal information of the user when completing a cognitive task is acquired;

[0186] The cognitive impairment screening module is used to perform comprehensive evaluation and processing on the collected task scene image information, eye movement data information set and electroencephalogram signal information set to obtain the user's cognitive impairment information.

[0187] The step of comprehensively evaluating and processing the collected task scene image information, eye movement data information set, and EEG signal information set to obtain the user's cognitive impairment information includes:

[0188] Performing evaluation processing on the task scene image information to obtain user participation task evaluation result information;

[0189] Performing evaluation processing on the eye movement data information set to obtain an eye movement evaluation result value;

[0190] Performing evaluation processing on the EEG signal information set to obtain an EEG evaluation result value;

[0191] Using a preset first weight vector, weighted sum processing is performed on the user's task participation evaluation result information, the eye movement evaluation result value, and the electroencephalogram evaluation result value to obtain a cognitive impairment evaluation value;

[0192] Determine whether the cognitive impairment assessment value is greater than a set cognitive discrimination threshold value, and obtain a cognitive discrimination result; if the cognitive discrimination result is greater than, determine that the user has cognitive impairment, and determine the cognitive impairment degree information in the cognitive impairment information as the cognitive impairment assessment value; if the cognitive discrimination result is not greater than, determine that the user does not have cognitive impairment;

[0193] The cognitive impairment information of the user is constructed by using the cognitive impairment degree information and the information on whether cognitive impairment exists.

[0194] The preset first weight vector is used to perform weighted summation processing on the user participation task evaluation result information, eye movement evaluation result value, and EEG evaluation result value, wherein the preset first weight vector can be 0.3, 0.4, or 0.3.

[0195] The position acquisition submodule can be implemented by using differential GPS;

[0196] The step of evaluating the eye movement data information set to obtain an eye movement evaluation result value includes:

[0197] Performing deviation evaluation processing on the fixation point information sequence in the eye movement data information set to obtain a deviation value of each fixation point information sequence;

[0198] The calculation expression of the deviation evaluation process is:

[0199]

[0200] Among them, p1 is the calculation parameter, a i is the deviation value of the i-th fixation point information sequence, (x0, y0, z0) is the position coordinate of the preset target fixation point in the cognitive task, (x i ,y i ,z i ) is the jth fixation point position coordinate of the ith fixation point information sequence, M is the number of fixations contained in a fixation point information sequence;

[0201] The deviation values ​​of all fixation point information sequences are averaged to obtain the fixation evaluation estimate;

[0202] Performing joint abnormality assessment processing on the fixation time information sequence and the eye movement frequency information sequence in the eye movement data information set to obtain a time-frequency assessment value;

[0203] The calculation expression of the joint anomaly assessment process is:

[0204]

[0205]

[0206] Among them, c ij is the time-frequency abnormality parameter of the ith fixation time information in the jth fixation time information sequence, t0 and f0 are the standard fixation time value and the standard eye saccade frequency value respectively, t ij and f ij are the i-th fixation time information in the j-th fixation time information sequence and the i-th eye saccade frequency information in the j-th eye saccade frequency information sequence, pi is the pi constant, c jis the time-frequency evaluation value of the jth fixation time information sequence, and P is the number of fixation time information contained in a fixation time information sequence;

[0207] The time-frequency evaluation values ​​corresponding to each fixation time information sequence and eye saccade frequency information sequence are averaged to obtain the time-frequency evaluation mean;

[0208] The gaze comment estimation value and the time-frequency evaluation mean are weightedly summed to obtain the eye movement evaluation result value.

[0209] The weight values ​​for weighted summation of the gaze comment estimation value and the time-frequency evaluation value are 0.7 and 0.3;

[0210] The step of evaluating and processing the EEG signal information set to obtain an EEG evaluation result value includes:

[0211] Performing evaluation amount calculation processing on each EEG signal sequence of the EEG signal information set to obtain a corresponding evaluation value;

[0212] The evaluation values ​​of all EEG signal sequences are averaged to obtain the EEG evaluation result value;

[0213] The expression for the evaluation amount calculation process is:

[0214]

[0215] g=(G 23 -G 34 ) / (G 23 +G 34 ),

[0216] Where g is the evaluation value of the EEG signal sequence, Gabor() represents the Gabor transform operation, x(n) represents the nth element of the EEG signal sequence, represents the conjugate element of x(n), G represents the length of the EEG signal sequence, G 23 Represents the first sub-specific quantity of the EEG signal sequence, G 34 Represents the second sub-specific quantity of the EEG signal sequence.

[0217] The task scene image information is evaluated and processed to obtain the user participation task evaluation result information, including:

[0218] Performing a first evaluation process on the task scene real-time image in the task scene image information to obtain first evaluation information;

[0219] Performing a second evaluation process on the participating task position coordinates and the target position information in the task scene image information to obtain second evaluation information;

[0220] The first evaluation information and the second evaluation information are weighted and summed using a preset evaluation weight vector to obtain user participation task evaluation result information.

[0221] The preset evaluation weight vector may be 0.3 and 0.7.

[0222] The performing a first evaluation process on the task scene real-time image in the task scene image information to obtain first evaluation information includes:

[0223] Performing human body key point recognition on the task scene real-time image in the task scene image information to obtain a key point spatial position matrix;

[0224] Performing difference calculation processing on the key point spatial position matrix and the standard participation task position matrix to obtain first evaluation information;

[0225] The difference calculation process includes:

[0226] Subtracting the key point spatial position matrix from the standard participation task position matrix to obtain a difference matrix;

[0227] Performing a first standard calculation process on the difference matrix to obtain a first standard matrix;

[0228] The expression of the first standard calculation process is:

[0229]

[0230] Where m represents the row dimension of the difference matrix, x ij represents the element of the i-th row and j-th column of the difference matrix, z ij represents the element of the i-th row and j-th column of the first standard matrix;

[0231] Performing column optimization processing on the first standard matrix to obtain an optimal solution vector;

[0232] The column optimal processing is to take out the largest number in each column to form the optimal solution vector z + ; The expression of the optimal solution vector is:

[0233]

[0234] Where n represents the column dimension of the difference matrix;

[0235] Performing column worst processing on the first standard matrix to obtain a worst solution vector;

[0236] The worst column processing is to take out the smallest number in each column to form the worst solution vector z - ; The expression of the worst solution vector is:

[0237]

[0238] The worst solution vector and the best solution vector are scored and calculated to obtain first evaluation information; the score calculation is expressed as:

[0239]

[0240] In the formula, ω j is the preset j-th importance weight; ω j It is obtained by presetting or by calculating the variance value of each column of the first standard matrix, and s is the first evaluation information.

[0241] The calculation expression of the position deviation value is:

[0242] a1=|x0-x1| / |x0+x1|+|1-y0 / y1|+|exp(z0 / z1)-p1| / |z0+z1|,

[0243] Wherein, p1 is the calculation parameter, a1 is the position deviation value, (x0, y0, z0) is the participating task position coordinate, and (x1, y1, z1) is the target position coordinate;

[0244] The calculation expression of the angle deviation value is:

[0245] c1=arctan(|x0-x1| / |y0-y1|)

[0246] c2=arctan(|x0-x1| / |z0-z1|),

[0247] Among them, c1 and c2 are angle deviation values;

[0248] The performing a second evaluation process on the participating task position coordinates and the target position information in the task scene image information to obtain second evaluation information includes:

[0249] Determine the target location information corresponding to the participating task location coordinates;

[0250] Extracting a coordinate set of boundary points of the target location information;

[0251] Using the coordinate set of the boundary points, a boundary matrix is ​​constructed;

[0252] Using the participating task position coordinates, construct a target vector;

[0253] A difference evaluation process is performed on the boundary matrix and the target vector to obtain second evaluation information.

[0254] The difference assessment process includes

[0255] Performing cross-correlation operations on the target vector and each row vector of the boundary matrix to obtain corresponding cross-correlation sequences;

[0256] Perform time-frequency transformation on each cross-correlation sequence to obtain the corresponding time-frequency sequence;

[0257] Perform distribution calculation on the time-frequency series to obtain a distribution sequence;

[0258] Perform logarithmic accumulation processing on all distribution sequences to obtain second evaluation information.

[0259] The expression of the distribution calculation process is:

[0260]

[0261] Among them, r k,i represents the distribution of the i-th value of the k-th time-frequency series, E k (i) represents the i-th value of the k-th time-frequency sequence, and N0 is the total number of elements in the time-frequency sequence;

[0262] The expression of the logarithmic accumulation process is:

[0263]

[0264] Wherein, H is the second evaluation information, and m is the number of time-frequency sequences.

[0265] The coordinate set of the boundary points includes the position coordinates of each boundary point of the target; the row vector of the boundary matrix is ​​the position coordinates of the boundary points.

[0266] The user feature extraction may be implemented by using a SURF feature point detection or a corner point detection algorithm.

[0267] The key points of the human body include the head, hands, and body;

[0268] The human body key point recognition can be implemented by using the OpenPose algorithm in OpenCV.

[0269] The time-frequency transformation process can be implemented by wavelet transformation or short-time Fourier transformation.

[0270] The augmented reality scene generation module can be implemented by augmented reality glasses; the eye tracker data acquisition module can be implemented by an eye tracker or a VR eye tracker developed by Xi'an Blue Brain Technology Co., Ltd.; the brain-computer interface data acquisition module can be implemented by a brain-computer interface device.

[0271] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A cognitive impairment screening device based on AR eye tracker and brain-computer interface, characterized in that: include: Augmented reality scene generation module, eye tracker data acquisition module, brain-computer interface data acquisition module, cognitive impairment screening module; The augmented reality scene generation module is used to generate and display an augmented reality scene containing a cognitive task, and collect task scene image information; The eye tracker data acquisition module is used to acquire in real time a set of eye movement data information when the user completes a cognitive task in an augmented reality scene; the eye movement data information set includes a fixation point information sequence, a fixation time information sequence, and an eye saccade frequency information sequence; the fixation point information sequence includes fixation point position information; the fixation time information sequence includes fixation time information; the eye saccade frequency information sequence includes eye saccade frequency information; The brain-computer interface data acquisition module is used to acquire in real time a set of EEG signal information when the user completes a cognitive task in an augmented reality scene; the set of EEG signal information includes an EEG signal sequence; The cognitive impairment screening module is connected to the augmented reality scene generation module, the eye tracker data acquisition module, and the brain-computer interface data acquisition module, respectively, and is used to perform comprehensive evaluation and processing on the collected task scene image information, eye movement data information set, and EEG signal information set to obtain the user's cognitive impairment information; The cognitive impairment information includes whether there is cognitive impairment and the degree of cognitive impairment.

2. The cognitive impairment screening device based on AR eye tracker and brain-computer interface according to claim 1, characterized in that: The mission scene image information includes a real-time mission scene image, target location information, and location coordinates of participating missions; the target location information includes location coordinate range information of several targets in the mission scene; The real-time image of the task scene includes an image of the user participating in the cognitive task; the participating task position coordinates are the position coordinates of the user when participating in completing the cognitive task in the task scene; each target position information has corresponding participating task position coordinates.

3. The cognitive impairment screening device based on AR eye tracker and brain-computer interface as claimed in claim 2, characterized in that: The augmented reality scene generation module includes an augmented reality scene display submodule, an image acquisition submodule, a position acquisition submodule, and a data acquisition submodule; The position acquisition submodule is used to acquire the position coordinates of the user when participating in the cognitive task in the task scene to obtain the position coordinates of the participating task; The augmented reality scene display submodule is used to generate and display an augmented reality scene including a cognitive task, and collect target location information; the augmented reality scene includes a target that requires the user to interact when participating in completing the cognitive task; The image acquisition submodule is used to acquire a real-time image of the task scene when the user participates in completing the cognitive task in the task scene; The data acquisition submodule is used to fuse the collected real-time image of the task scene, the target position information and the coordinates of the participating task positions to obtain the task scene image information, and send the task scene image information to the cognitive impairment screening module.

4. The cognitive impairment screening device based on AR eye tracker and brain-computer interface as claimed in claim 2, characterized in that: The cognitive impairment screening module is used to comprehensively evaluate and process the collected task scene image information, eye movement data information set and EEG signal information set to obtain the user's cognitive impairment information, including: The cognitive impairment screening module evaluates and processes the task scene image information to obtain user participation task evaluation result information; Performing evaluation processing on the eye movement data information set to obtain an eye movement evaluation result value; Performing evaluation processing on the EEG signal information set to obtain an EEG evaluation result value; Using a preset first weight vector, weighted sum processing is performed on the user's task participation evaluation result information, the eye movement evaluation result value, and the electroencephalogram evaluation result value to obtain a cognitive impairment evaluation value; Determine whether the cognitive impairment assessment value is greater than a set cognitive discrimination threshold value, and obtain a cognitive discrimination result; if the cognitive discrimination result is greater than, determine that the user has cognitive impairment, and determine the cognitive impairment degree information in the cognitive impairment information as the cognitive impairment assessment value; if the cognitive discrimination result is not greater than, determine that the user does not have cognitive impairment; The cognitive impairment information of the user is constructed by using the cognitive impairment degree information and the information on whether cognitive impairment exists.

5. A method for screening cognitive impairment based on AR eye tracker and brain-computer interface, characterized in that: The method is implemented by using the cognitive impairment screening device based on an AR eye tracker and a brain-computer interface according to any one of claims 1 to 4, comprising: Using the augmented reality scene generation module, an augmented reality scene including a cognitive task is generated and displayed, and task scene image information is acquired; Using the eye tracker data acquisition module, a collection of eye movement data information of the user in the augmented reality scene is acquired in real time; Using the brain-computer interface data acquisition module, a collection of EEG signal information of the user when completing a cognitive task is acquired; The cognitive impairment screening module is used to perform comprehensive evaluation and processing on the collected task scene image information, eye movement data information set and electroencephalogram signal information set to obtain the user's cognitive impairment information.

6. The cognitive impairment screening device based on AR eye tracker and brain-computer interface as claimed in claim 5, characterized in that: The step of comprehensively evaluating and processing the collected task scene image information, eye movement data information set, and EEG signal information set to obtain the user's cognitive impairment information includes: Performing evaluation processing on the task scene image information to obtain user participation task evaluation result information; Performing evaluation processing on the eye movement data information set to obtain an eye movement evaluation result value; Performing evaluation processing on the EEG signal information set to obtain an EEG evaluation result value; Using a preset first weight vector, weighted sum processing is performed on the user's task participation evaluation result information, the eye movement evaluation result value, and the electroencephalogram evaluation result value to obtain a cognitive impairment evaluation value; Determine whether the cognitive impairment assessment value is greater than a set cognitive discrimination threshold value, and obtain a cognitive discrimination result; if the cognitive discrimination result is greater than, determine that the user has cognitive impairment, and determine the cognitive impairment degree information in the cognitive impairment information as the cognitive impairment assessment value; if the cognitive discrimination result is not greater than, determine that the user does not have cognitive impairment; The cognitive impairment information of the user is constructed by using the cognitive impairment degree information and the information on whether cognitive impairment exists.

7. The method for screening cognitive impairment based on AR eye tracker and brain-computer interface as claimed in claim 6, characterized in that: The step of evaluating the eye movement data information set to obtain an eye movement evaluation result value includes: Performing deviation evaluation processing on the fixation point information sequence in the eye movement data information set to obtain a deviation value of each fixation point information sequence; The calculation expression of the deviation evaluation process is: Among them, t1 is the preset calculation parameter, a i is the deviation value of the i-th fixation point information sequence, (x0, y0, z0) is the position coordinate of the preset target fixation point in the cognitive task, (x ij ,y ij ,z ij ) is the jth fixation point position coordinate of the ith fixation point information sequence, M is the number of fixations contained in a fixation point information sequence; The deviation values ​​of all fixation point information sequences are averaged to obtain the fixation evaluation estimate; Performing joint abnormality assessment processing on the fixation time information sequence and the eye movement frequency information sequence in the eye movement data information set to obtain a time-frequency assessment value; The calculation expression of the joint anomaly assessment process is: Among them, c ij is the time-frequency abnormality parameter of the ith fixation time information in the jth fixation time information sequence, t0 and f0 are the standard fixation time value and the standard eye saccade frequency value respectively, t ij and f ij are the i-th fixation time information in the j-th fixation time information sequence and the i-th eye saccade frequency information in the j-th eye saccade frequency information sequence, pi is the pi constant, c j is the time-frequency evaluation value of the jth fixation time information sequence, and P is the number of fixation time information contained in a fixation time information sequence; The time-frequency evaluation values ​​corresponding to all fixation time information sequences and saccade frequency information sequences are averaged to obtain the time-frequency evaluation mean; The gaze comment estimation value and the time-frequency evaluation mean are weightedly summed to obtain the eye movement evaluation result value.

8. The method for screening cognitive impairment based on AR eye tracker and brain-computer interface as claimed in claim 7, characterized in that: The task scene image information is evaluated and processed to obtain the user participation task evaluation result information, including: Performing a first evaluation process on the task scene real-time image in the task scene image information to obtain first evaluation information; Performing a second evaluation process on the participating task position coordinates and the target position information in the task scene image information to obtain second evaluation information; The first evaluation information and the second evaluation information are weighted and summed using a preset evaluation weight vector to obtain user participation task evaluation result information.

9. The method for screening cognitive impairment based on AR eye tracker and brain-computer interface as claimed in claim 8, characterized in that: The performing a first evaluation process on the task scene real-time image in the task scene image information to obtain first evaluation information includes: Performing human body key point recognition on the task scene real-time image in the task scene image information to obtain a key point spatial position matrix; Performing difference calculation processing on the key point spatial position matrix and the standard participation task position matrix to obtain first evaluation information; The difference calculation process includes: Subtracting the key point spatial position matrix from the standard participation task position matrix to obtain a difference matrix; Performing a first standard calculation process on the difference matrix to obtain a first standard matrix; The expression of the first standard calculation process is: Where m represents the row dimension of the difference matrix, x ij represents the element of the i-th row and j-th column of the difference matrix, z ij represents the element of the i-th row and j-th column of the first standard matrix; Performing column optimization processing on the first standard matrix to obtain an optimal solution vector; The column optimal processing is to take out the largest number in each column to form the optimal solution vector z + ; The expression of the optimal solution vector is: Where n represents the column dimension of the difference matrix; Performing column worst processing on the first standard matrix to obtain a worst solution vector; The worst column processing is to take out the smallest number in each column to form the worst solution vector z - ; The expression of the worst solution vector is: The worst solution vector and the best solution vector are scored and calculated to obtain first evaluation information; the score calculation is expressed as: In the formula, ω j is the preset j-th importance weight, and s is the first evaluation information.

10. The method for screening cognitive impairment based on AR eye tracker and brain-computer interface as claimed in claim 8, characterized in that: The performing a second evaluation process on the participating task position coordinates and the target position information in the task scene image information to obtain second evaluation information includes: Determine the target location information corresponding to the participating task location coordinates; Extracting a coordinate set of boundary points of the target location information; Using the coordinate set of the boundary points, a boundary matrix is ​​constructed; Using the participating task position coordinates, construct a target vector; A difference evaluation process is performed on the boundary matrix and the target vector to obtain second evaluation information.

Images