Cognitive disorder recognition method, device, equipment and system based on eye movement data

Abstract

The invention relates to the technical field of cognitive impairment recognition, and discloses a cognitive impairment recognition method based on eye movement data. The method comprises the following steps: initializing a feature vector of each real node according to eye movement data related to fixation in to-be-processed eye movement data; initializing a feature vector of each edge according to eye movement data related to skipping and / or tracking in the to-be-processed eye movement data; constructing a global virtual node, and initializing feature vectors of the global virtual node according to the eye movement data appearing in the staring, skipping and tracking processes in the to-be-processed eye movement data; updating a graph structure formed by the real nodes, the edges and the global virtual nodes for more than one time, and fusing all the updated feature vectors to obtain a fused feature vector; and carrying out regression on the fused feature vector to obtain a cognitive impairment evaluation result. By adopting the method, a relatively fine cognitive impairment evaluation result can be obtained. The invention further discloses a cognitive disorder recognition device, equipment and system based on the eye movement data.

Description

Technical Field

[0001] The present application relates to the technical field of cognitive impairment identification, for example, to a method, device, equipment and system for cognitive impairment identification based on eye movement data. Background Art

[0002] Existing research has shown that different degrees of mild cognitive impairment (MCI) have distinct eye movement characteristics. For example, as the patient's condition worsens, gaze accuracy gradually decreases, and the start latency, initial acceleration, and accuracy of smooth pursuit decrease.

[0003] Some existing technologies can analyze saccade data (a type of eye movement data) to obtain abnormal saccade data. This can help doctors more quickly diagnose different types of saccades and predict Alzheimer's disease (AD) and Parkinson's disease (PD) based on different saccade types. This technology can be used for early screening of MCI.

[0004] Furthermore, other existing technologies have proposed eye movement data under different difficulty levels of eye movement paradigms, and used the Montreal Cognitive Assessment (MoCA) to conduct cognitive assessments on participants, obtaining MoCA scores. Finally, correlation analysis is performed between MoCA scores and eye movement characteristics under different difficulty levels of eye movement paradigms, obtaining correlation coefficients and significance between the two. Based on this, further refined assessments of MCI can be performed based on eye movement data.

[0005] During the implementation of the embodiments of the present application, it was found that at least the following problems exist in the related art:

[0006] During clinical treatment, improving MCI is a lengthy process. When eye movement features are limited, such as when using eye movement data from a highly significant paradigm, the granularity of MCI assessment results is relatively coarse, making it difficult to provide doctors with real-time feedback on treatment effectiveness.

[0007] On the contrary, if MCI is identified based on several eye movement features such as saccade distance, start delay time (latency), total fixation duration, fixation position, and dynamic tracking trajectory deviation, the existing neural network model has a serious black box effect, lacks interpretability, and the accuracy of the recognition results is biased. Summary of the Invention

[0008] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0009] The embodiments of the present application provide a method, apparatus, device, and system for identifying cognitive impairment based on eye movement data. Based on Graph Attention Networks (GATs), the system innovatively proposes global virtual nodes and redefines nodes and first-order neighbors. The patient's eye movement data is then extracted into a graph, and the improved GATs model is used to process the eye movement data. While ensuring the interpretability of the model, the system comprehensively considers multiple eye movement features to obtain more detailed cognitive impairment assessment results, thereby assisting doctors in making comprehensive decisions on treatment plans.

[0010] In some embodiments, a method for identifying cognitive impairment based on eye movement data includes:

[0011] Initialize the feature vector of each real node based on the gaze-related eye movement data in the processed eye movement data; where the real node represents the visual area of ​​interest (AOI);

[0012] Initializing a feature vector of each edge based on saccadic and / or pursuit-related eye movement data in the to-be-processed eye movement data; wherein an edge represents a saccadic and / or pursuit relationship between two real nodes;

[0013] Constructing a global virtual node and initializing a feature vector of the global virtual node based on the eye movement data to be processed, which appear in the gaze, saccade and pursuit processes;

[0014] The graph structure composed of real nodes, edges, and global virtual nodes is updated more than once, and all updated feature vectors are fused to obtain a fused feature vector. In each update, based on the multi-head attention mechanism, the attention score of the object to be updated and its first-order neighbors is obtained, and the object to be updated is updated according to the feature vectors and corresponding attention scores of each first-order neighbor. Among them, the object to be updated is the real node, the directed edge, or the global virtual node. The first-order neighbors include directly connected real nodes and real nodes, real nodes and edges, and edges and edges. The global virtual node and each real node and each edge are first-order neighbors of each other.

[0015] The fused feature vector was regressed to obtain the cognitive impairment assessment results.

[0016] In some embodiments, the apparatus for identifying cognitive impairment based on eye movement data includes:

[0017] A real node initialization module is used to initialize the feature vector of each real node according to the gaze-related eye movement data in the eye movement data to be processed; wherein the real node represents the AOI;

[0018] an edge initialization module, configured to initialize a feature vector of each edge based on saccadic and / or pursuit-related eye movement data in the processed eye movement data; wherein an edge represents a saccadic and / or pursuit relationship between two real nodes;

[0019] A global virtual node initialization module is used to construct a global virtual node and initialize the feature vector of the global virtual node based on the eye movement data to be processed, which appears in the gaze, saccade and tracking processes;

[0020] The update module is used to update the graph structure composed of real nodes, edges, and global virtual nodes more than once, and fuse all updated feature vectors to obtain a fused feature vector. In each update, based on the multi-head attention mechanism, the attention score of the object to be updated and its first-order neighbors is obtained, and the object to be updated is updated according to the feature vectors and corresponding attention scores of each first-order neighbor. The object to be updated is the real node, the directed edge, or the global virtual node. The first-order neighbors include directly connected real nodes and real nodes, real nodes and edges, and edges and edges. The global virtual node and each real node and each edge are first-order neighbors of each other.

[0021] The regression module is used to regress the fused feature vector to obtain the cognitive impairment assessment results.

[0022] In some embodiments, a cognitive impairment identification device based on eye movement data includes: a processor and a memory storing program instructions, and the processor is configured to execute the cognitive impairment identification method based on eye movement data provided by the aforementioned embodiment when executing the program instructions.

[0023] In some embodiments, the system for identifying cognitive impairment based on eye movement data includes the device for identifying cognitive impairment based on eye movement data provided in the aforementioned embodiments.

[0024] The methods, devices, and systems for identifying cognitive solutions based on eye movement data provided in the embodiments of the present application can achieve the following technical effects:

[0025] This application initializes the feature vectors of real nodes, edges, and global virtual nodes based on the eye movement data to be processed, and actually extracts the patient's eye movement performance into a graph composed of nodes and edges, which can reflect the patient's overall eye movement situation.

[0026] In this application, "first-order neighbors" are redefined: first-order neighbors include directly connected real nodes and real nodes, real nodes and edges, and edges and edges; global virtual nodes and each real node and each edge are first-order neighbors to each other.

[0027] The multi-head attention mechanism is to perform multiple attention calculations on the same object. When the multi-head attention mechanism is applied to this application, it means evaluating the similarity between the object to be updated and its first-order neighbors from multiple angles. After obtaining multiple attention scores based on the multi-head attention mechanism, the object to be updated is updated according to the feature vectors and corresponding attention scores of each first-order neighbor. That is, the similar features between the first-order neighbors and the object to be updated are extracted from multiple angles, and then the update results of the object to be updated are obtained by convergence.

[0028] In this way, when the object to be updated is a real node and the first-order neighbor is a real node, the two have the same type of features. Based on the attention mechanism, the attention score of the combination of the two is obtained. The attention score can reflect the similarity between the object to be updated and the real node. The higher the attention score corresponding to a certain-order neighbor, the more important the first-order neighbor is to the updated object. Moreover, compared with the case where the object to be updated and its first-order neighbor are of different types, the attention score indicates that the same type of object to be updated and its first-order neighbor have more similar features. When updating the real node, the real node as the first-order neighbor will contribute more features.

[0029] When the object to be updated is a real node and its first-order neighbor is an edge or global virtual node, they typically have different types of features. Based on the attention mechanism, an attention score is obtained by combining the two. This attention score reflects the similarity between the object to be updated and the edge or global virtual node. The higher the attention score corresponding to a given first-order neighbor, the more important that first-order neighbor is to the updated object. Because real nodes, edges, and global virtual nodes are three different perspectives for classifying eye movement data, this attention score indicates that the different types of objects to be updated have fewer similar features with their first-order neighbors compared to the case where the object to be updated and its first-order neighbors are of the same type. When updating the object to be updated, the edges or global virtual nodes that serve as first-order neighbors contribute fewer features, that is, they serve as a reference.

[0030] Because MCI patients will show MCI-related eye movement characteristics in different eye movement tasks, even if the eye movement feature types represented by edges and global virtual nodes are different from those represented by real nodes, the feature vectors of edges and global virtual nodes are still of reference significance during the real node update process.

[0031] In summary, when updating real nodes, the first-order neighbors of the real nodes provide the main features, and the first-order neighbors of the edges and global virtual nodes provide reference features. In this way, the updated real nodes are dominated by gaze-related eye movement features and integrate the eye movement features of the edges and global virtual nodes, which can better reflect the similar eye movement features of MCI patients in different tasks.

[0032] When the object to be updated is an edge and its first-order neighbor is an edge, the two have the same type of features. Based on the attention mechanism, an attention score is obtained after combining the two. The attention score can reflect the similarity between the object to be updated and the edge. The higher the attention score corresponding to a certain-order neighbor, the more important the first-order neighbor is to the updated object. Moreover, compared with the case where the object to be updated and its first-order neighbor are of different types, the attention score indicates that the same type of object to be updated and its first-order neighbor have more similar features. When updating the edge, the edge as a first-order neighbor will contribute more features.

[0033] When the object to be updated is an edge and its first-order neighbor is a real node or a global virtual node, they typically have different types of features. Based on the attention mechanism, an attention score is obtained by combining the two. This attention score reflects the similarity between the object to be updated and the real node or global virtual node. The higher the attention score corresponding to a given first-order neighbor, the more important that first-order neighbor is to the updated object. Because real nodes, edges, and global virtual nodes are three different perspectives for classifying eye movement data, this attention score indicates that the object to be updated of different types has fewer similar features with its first-order neighbors compared to the case where the object to be updated and its first-order neighbors are of the same type. When updating an edge, the real node or global virtual node as a first-order neighbor contributes fewer features, that is, it serves as a reference.

[0034] In summary, when updating edges, the first-order neighbors of the edges provide the main features, and the first-order neighbors of the real nodes and global virtual nodes provide reference features. The updated real nodes are dominated by eye movement features related to saccades and / or pursuits, and the eye movement features of the real nodes and global virtual nodes are integrated, which can better reflect the similar eye movement features of MCI patients in different tasks.

[0035] When the object to be updated is a global virtual node and the first-order neighbor is an edge or a real node, the two usually have different types of features. Based on the attention mechanism, an attention score is obtained by combining the two. This attention score reflects the similarity between the object to be updated and the edge or real node. The higher the attention score corresponding to a certain-order neighbor, the more important the first-order neighbor is to the updated object.

[0036] GATs consists of multiple layers. The newly proposed global virtual nodes are updated using the above method. On the basis of better reflecting the similar characteristics of MCI patients in different tasks, the updated feature vectors are consistent with the updated feature vectors of real nodes and edges, which facilitates the continued update or fusion of all feature vectors to obtain a fused feature vector, and then further regression.

[0037] It can be seen that the present application has better model interpretability. In addition, when updating the nodes defined under the standard GATs, the feature vectors of the edges and the feature vectors of the global virtual nodes are comprehensively considered, so that the updated nodes defined under the standard GATs can better reflect the characteristics of all eye movement data, which is not available in the standard GATs. In addition to the standard GATs, the edges are defined as "pseudo nodes". In accordance with the GATs inductive thinking, the characteristics of the edges themselves can be fully summarized and extracted to ensure that the eye movement data features are not missed. Of course, during the edge update process, both the global virtual nodes and the real nodes will further enrich the characteristics of the edges.

[0038] Standard GATs are mainly used for induction tasks. In the improved GATs, the update process of real nodes is actually an induction of all eye movement data, which can better summarize the consistent features of all eye movement data; similarly, global virtual nodes and edges can also summarize the consistent features of all eye movement data.

[0039] Furthermore, the fused feature vector obtained by finally fusing all the updated feature vectors can characterize the consistent features of all the eye movement data from three perspectives. Without improving the regression algorithm, the cognitive impairment assessment results obtained by regression are more accurate because the summarized eye movement data are more representative.

[0040] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] One or more embodiments are exemplarily described by corresponding drawings. These exemplary descriptions and drawings do not limit the embodiments. Elements with the same reference numerals in the drawings are considered similar elements, and wherein:

[0042] Figure 1 This is a schematic diagram of an application scenario of a method for identifying cognitive impairment based on eye movement data provided in an embodiment of the present application;

[0043] Figure 2a and Figure 2b This is a schematic diagram comparing the standard GATs graph structure and the improved GATs graph structure provided in the embodiment of the present application;

[0044] Figure 31 is a flow chart of a method for identifying cognitive impairment based on eye movement data provided in an embodiment of the present application;

[0045] Figure 4 1 is a flow chart of a method for identifying cognitive impairment based on eye movement data provided in an embodiment of the present application;

[0046] Figure 5 is a schematic diagram of a cognitive impairment identification device based on eye movement data provided by an embodiment of the present application;

[0047] Figure 6 This is a schematic diagram of a cognitive impairment identification device based on eye movement data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to be able to understand the features and technical contents of the embodiments of the present application in more detail, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference only and are not used to limit the embodiments of the present application. In the following technical description, for the sake of convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0049] In the description and claims of the embodiments of the present application and the accompanying drawings, the terms "first," "second," and the like are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate for the purposes of describing the embodiments of the present application. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions.

[0050] Unless otherwise stated, the term "plurality" means two or more.

[0051] In the embodiments of the present application, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0052] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0053] Figure 1 This is a schematic diagram of an application scenario of a method for identifying cognitive impairment based on eye movement data provided in an embodiment of the present application.

[0054] This method for identifying cognitive impairment based on eye movement data can quickly, non-invasively, and efficiently obtain a patient's cognitive impairment score. During clinical treatment, doctors can combine this score with the patient's symptoms to adaptively decide on treatment options.

[0055] Specifically, this method for identifying cognitive impairment based on eye movement data can be integrated into a hospital's backend server to support the cognitive impairment assessment process. After the patient completes the eye movement test, the doctor uploads the eye movement data to the hospital's backend server via a terminal. Once the backend server completes the calculations, the cognitive impairment assessment results are returned to the terminal for the doctor's reference.

[0056] It should be understood that this method for identifying cognitive impairment based on eye movement data is based on clinical treatment. Given its effectiveness, it can also be applied to early screening for MCI. For example, this method can be deployed in embedded devices, such as integrated into eye tracking devices, for use in primary care or routine home screening.

[0057] In addition, this cognitive impairment identification method based on eye movement data is proposed as an auxiliary treatment for MCI and can also be used to assist in the treatment of AD, PD, etc.

[0058] In this cognitive impairment identification method based on eye movement data, the concept of global virtual nodes is innovatively proposed, and the nodes and first-order neighbors are redefined, so that the improved GATs can process all eye movement parameters.

[0059] Combine Figure 2a As shown in Figure 1, the nodes and edges defined by the standard GATs are l, m, n, and o, and the edges are p, q, r, and s. In this application, the nodes in the standard GATs are mapped to "pseudo-edges", and the edges in the standard GATs are mapped to "pseudo-nodes", as shown in Figure 1. Figure 2b As shown, the nodes are p, q, r and s, and the edges are l, m, n and o.

[0060] In this way, the edge feature vectors can be summarized without departing from the standard GATs idea, maintaining the original inductive effect of the model.

[0061] In addition, regardless of the direction of the edge, the term "first-order neighbors" in standard GATs refers to two nodes that are directly connected, for example, Figure 2a In the example, node m and node n are first-order neighbors, node n and node o are first-order neighbors, and so on.

[0062] In this application, after mapping the edges in the standard GATs to "pseudo nodes", the edges can be used as "pseudo neighbors" of the nodes, and vice versa, the nodes can also be used as "pseudo neighbors" of the edges. The term "first-order neighbors" refers to the same and different objects that are directly connected to the object to be updated, as well as the special case: the global virtual node and other objects are all first-order neighbors. Figure 2b As shown, the first-order neighbors of the real node m include the real node l, the real node n, the edge s, the edge p and the global virtual node g; the first-order neighbors of the edge p include the edge s, the edge q, the real node m, the real node n and the global virtual node g; the first-order neighbors of the global virtual node g include all real nodes and all edges.

[0063] The above content uses the example of ignoring the direction of the edge to illustrate the first-order neighbors.

[0064] In this application, the direction of the saccade / tracking can be regarded as the direction of the edge. When considering the direction of the edge, the term "first-order neighbor" in standard GATs refers to other nodes that are directly connected and point to the current node. For example Figure 2a As shown, node l is a first-order neighbor of node m, but node m is not a first-order neighbor of node n.

[0065] In this application, the term "first-order neighbor" refers to objects of the same type and different types that are directly connected to the object to be updated and point to the object to be updated, as well as special cases: global virtual nodes and other objects have first-order neighbor relationships. Figure 2b As shown, the first-order neighbors of node m include node l, edge s, and the global virtual node g; the first-order neighbors of edge p include node m, edge s, and the global virtual node g; the first-order neighbors of the global virtual node include all real nodes and all edges.

[0066] For test images with a clear AOI order, such as "A, B, C, D...", the direction of the edge can be considered; for test images with an unclear AOI order, the direction of the edge can be ignored.

[0067] Figure 3 This is a flow chart of a method for identifying cognitive impairment based on eye movement data provided in an embodiment of the present application.

[0068] Combine Figure 3 As shown in Figure 2, methods for identifying cognitive impairments based on eye movement data include:

[0069] S301 : Initialize a feature vector of each real node according to gaze-related eye movement data in the eye movement data to be processed.

[0070] Among them, the real node represents the AOI.

[0071] When collecting eye movement data from patients, a test image is often used to stimulate the patient's eyes. The AOI can be fixed, such as a bright spot, or it can be fluid, such as a movie poster.

[0072] When the AOI is fixed, the AOI coordinates can be set in advance, and then according to the AOI coordinates, the gaze-related eye movement data belonging to the AOI coordinates can be extracted from the eye movement data to be processed; when the AOI is not fixed, a clustering method can be used to calculate multiple AOI coordinates in the eye movement data to be processed.

[0073] S302 : Initialize the feature vector of each edge according to the eye movement data related to saccades and / or pursuits in the eye movement data to be processed.

[0074] This step includes three cases: the first case is to initialize the feature vector of each edge according to the saccade-related eye movement data in the eye movement data to be processed; the second case is to initialize the feature vector of each edge according to the tracking-related eye movement data in the eye movement data to be processed; the third case is to initialize the feature vector of each edge according to the saccade- and tracking-related eye movement data in the eye movement data to be processed.

[0075] In this step, an edge indicates that there is a saccade and / or tracking relationship between two real nodes. Specifically, if a user saccades from one real node to another, the two real nodes have a saccade relationship, and a saccade-significant edge exists between them. Otherwise, there is no saccade relationship, and no saccade-significant edge exists between them.

[0076] For test images with tracking tasks, which contain moving targets, such as moving light spots, moving badminton and other objects, if two real nodes are on the actual motion trajectory of the moving target at the same time, then the two nodes have a tracking relationship, and there is an edge of tracking significance between the two nodes; if both real nodes are not on the actual motion trajectory of the moving target, or one real node is on the trajectory and the other real node is outside the trajectory, then the two real nodes do not have a tracking relationship, and there is no edge of tracking significance between the two real nodes.

[0077] When the eye movement data to be processed includes tracking-related eye movement data, the aforementioned real nodes must include two or more nodes located on the target movement trajectory.

[0078] In the specific application process, in order to simplify the processing, in the test screen, the moving target can be moved from one AOI to another AOI, so there is no need to set up independent real nodes for the tracking task.

[0079] S303: Construct a global virtual node, and initialize a feature vector of the global virtual node based on the eye movement data to be processed, which appear in the gaze, saccade and tracking processes.

[0080] S304: Update the graph structure composed of real nodes, edges, and global virtual nodes more than once, and fuse all updated feature vectors to obtain a fused feature vector.

[0081] If all real nodes, all edges, and global virtual nodes on the graph are updated once, it becomes a one-layer graph. If all are updated twice, it becomes a two-layer graph, and so on.

[0082] Optionally, the number of layers of the graph is two to five.

[0083] In each update, based on the multi-head attention mechanism, the attention score of the object to be updated and its first-order neighbors is obtained, and the object to be updated is updated according to the feature vector of each first-order neighbor and the corresponding attention score; among them, the object to be updated is the real node, the directed edge or the global virtual node, and the first-order neighbors include directly connected real nodes and real nodes, real nodes and edges, and edges and edges; the global virtual node and each real node and each edge are first-order neighbors to each other.

[0084] The multi-head attention mechanism performs multiple attention calculations on the same object. In the above technical solution, based on the multi-head attention mechanism, the attention score of the object to be updated and its first-order neighbors is obtained. Multiple attention calculations are performed on the combination of the object to be updated and its first-order neighbors.

[0085] Those skilled in the art can set the specific number of attention heads based on experience and verify it based on actual MCI data to determine the optimal number of attention heads.

[0086] In this embodiment of the present application, each graph layer includes real nodes, edges, and global virtual nodes, all of which need to be updated. The object to be updated is the real node, the directed edge, or the global virtual node. That is, when updating a real node, the object to be updated is the real node; when updating an edge, the object to be updated is the edge; when updating a global virtual node, the object to be updated is the global virtual node.

[0087] The method of fusing all updated feature vectors is an existing technology, including but not limited to: vector concatenation, vector summation, and corresponding element multiplication.

[0088] S305. Regress the fused feature vector to obtain a cognitive impairment assessment result.

[0089] For example, a fully connected layer can be used for regression: in, is the cognitive impairment assessment result, w is the weight vector of the fully connected layer obtained through training, T represents transposition, V G is the fusion feature vector, b is the bias scalar, obtained through training.

[0090] Alternatively, you can continue to connect the Sigmoid function: In this way, the results of cognitive impairment assessment can be The value of is limited to [0,1].

[0091] This application initializes the feature vectors of real nodes, edges, and global virtual nodes based on the eye movement data to be processed, and actually extracts the patient's eye movement performance into a graph composed of nodes and edges, which can reflect the patient's overall eye movement situation.

[0092] In this application, "first-order neighbors" are redefined: first-order neighbors include directly connected real nodes and real nodes, real nodes and edges, and edges and edges; global virtual nodes and each real node and each edge are first-order neighbors to each other.

[0093] The multi-head attention mechanism is to perform multiple attention calculations on the same object. When the multi-head attention mechanism is applied to this application, it means evaluating the similarity between the object to be updated and its first-order neighbors from multiple angles. After obtaining multiple attention scores based on the multi-head attention mechanism, the object to be updated is updated according to the feature vectors and corresponding attention scores of each first-order neighbor. That is, the similar features between the first-order neighbors and the object to be updated are extracted from multiple angles, and then the update results of the object to be updated are obtained by convergence.

[0094] In this way, when the object to be updated is a real node and the first-order neighbor is a real node, the two have the same type of features. Based on the attention mechanism, the attention score of the combination of the two is obtained. The attention score can reflect the similarity between the object to be updated and the real node. The higher the attention score corresponding to a certain-order neighbor, the more important the first-order neighbor is to the updated object. Moreover, compared with the case where the object to be updated and its first-order neighbor are of different types, the attention score indicates that the same type of object to be updated and its first-order neighbor have more similar features. When updating the real node, the real node as the first-order neighbor will contribute more features.

[0095] When the object to be updated is a real node and its first-order neighbor is an edge or global virtual node, they typically have different types of features. Based on the attention mechanism, an attention score is obtained by combining the two. This attention score reflects the similarity between the object to be updated and the edge or global virtual node. The higher the attention score corresponding to a given first-order neighbor, the more important that first-order neighbor is to the updated object. Because real nodes, edges, and global virtual nodes are three different perspectives for classifying eye movement data, this attention score indicates that the different types of objects to be updated have fewer similar features with their first-order neighbors compared to the case where the object to be updated and its first-order neighbors are of the same type. When updating the object to be updated, the edges or global virtual nodes that serve as first-order neighbors contribute fewer features, that is, they serve as a reference.

[0096] Because MCI patients will show MCI-related eye movement characteristics in different eye movement tasks, even if the eye movement feature types represented by edges and global virtual nodes are different from those represented by real nodes, the feature vectors of edges and global virtual nodes are still of reference significance during the real node update process.

[0097] In summary, when updating real nodes, the first-order neighbors of the real nodes provide the main features, and the first-order neighbors of the edges and global virtual nodes provide reference features. In this way, the updated real nodes are dominated by gaze-related eye movement features and integrate the eye movement features of the edges and global virtual nodes, which can better reflect the similar eye movement features of MCI patients in different tasks.

[0098] When the object to be updated is an edge and its first-order neighbor is an edge, the two have the same type of features. Based on the attention mechanism, an attention score is obtained after combining the two. The attention score can reflect the similarity between the object to be updated and the edge. The higher the attention score corresponding to a certain-order neighbor, the more important the first-order neighbor is to the updated object. Moreover, compared with the case where the object to be updated and its first-order neighbor are of different types, the attention score indicates that the same type of object to be updated and its first-order neighbor have more similar features. When updating the edge, the edge as a first-order neighbor will contribute more features.

[0099] When the object to be updated is an edge and its first-order neighbor is a real node or a global virtual node, they typically have different types of features. Based on the attention mechanism, an attention score is obtained by combining the two. This attention score reflects the similarity between the object to be updated and the real node or global virtual node. The higher the attention score corresponding to a given first-order neighbor, the more important that first-order neighbor is to the updated object. Because real nodes, edges, and global virtual nodes are three different perspectives for classifying eye movement data, this attention score indicates that the object to be updated of different types has fewer similar features with its first-order neighbors compared to the case where the object to be updated and its first-order neighbors are of the same type. When updating an edge, the real node or global virtual node as a first-order neighbor contributes fewer features, that is, it serves as a reference.

[0100] In summary, when updating edges, the first-order neighbors of the edges provide the main features, and the first-order neighbors of the real nodes and global virtual nodes provide reference features. The updated real nodes are dominated by eye movement features related to saccades and / or pursuits, and the eye movement features of the real nodes and global virtual nodes are integrated, which can better reflect the similar eye movement features of MCI patients in different tasks.

[0101] When the object to be updated is a global virtual node and the first-order neighbor is an edge or a real node, the two usually have different types of features. Based on the attention mechanism, an attention score is obtained by combining the two. This attention score reflects the similarity between the object to be updated and the edge or real node. The higher the attention score corresponding to a certain-order neighbor, the more important the first-order neighbor is to the updated object.

[0102] GATs consists of multiple layers. The newly proposed global virtual nodes are updated using the above method. On the basis of better reflecting the similar characteristics of MCI patients in different tasks, the updated feature vectors are consistent with the updated feature vectors of real nodes and edges, which facilitates the continued update or fusion of all feature vectors to obtain a fused feature vector, and then further regression.

[0103] It can be seen that the present application has better model interpretability. In addition, when updating the nodes defined under the standard GATs, the feature vectors of the edges and the feature vectors of the global virtual nodes are comprehensively considered, so that the updated nodes defined under the standard GATs can better reflect the characteristics of all eye movement data, which is not available in the standard GATs. In addition to the standard GATs, the edges are defined as "pseudo nodes". In accordance with the GATs inductive thinking, the characteristics of the edges themselves can be fully summarized and extracted to ensure that the eye movement data features are not missed. Of course, during the edge update process, both the global virtual nodes and the real nodes will further enrich the characteristics of the edges.

[0104] Standard GATs are mainly used for induction tasks. In the improved GATs, the update process of real nodes is actually an induction of all eye movement data, which can better summarize the consistent features of all eye movement data; similarly, global virtual nodes and edges can also summarize the consistent features of all eye movement data.

[0105] Furthermore, the fused feature vector obtained by finally fusing all the updated feature vectors can characterize the consistent features of all the eye movement data from three perspectives. Without improving the regression algorithm, the cognitive impairment assessment results obtained by regression are more accurate because the summarized eye movement data are more representative.

[0106] The following further explains the update process of the improved GATs.

[0107] Optionally, based on a multi-head attention mechanism, an attention score of the object to be updated and its first-order neighbors is obtained, and the object to be updated is updated according to the feature vector of each first-order neighbor and the corresponding attention score, including:

[0108] Repeat the following process multiple times to obtain multiple products: concatenate the feature vectors of the object to be updated and its first-order neighbors to obtain a concatenated feature vector; calculate the attention score of the concatenated feature vector based on the self-attention mechanism; calculate the product of the attention score and the feature vector of its corresponding first-order neighbor;

[0109] A nonlinear transformation is performed on all the products after the connection, or on the average value of all the products, and the nonlinear transformation is used as the update result of the object to be updated.

[0110] The connection operation in the above technical solution includes two cases: the dimension of the feature vectors of the object to be updated and its first-order neighbors is the same, and the dimension of the feature vectors of the object to be updated and its first-order neighbors is different.

[0111] When the eigenvector dimensions of the object to be updated and its first-order neighbor are the same, first concatenate the corresponding eigenmatrices of the two to obtain a new eigenvector, and then perform a linear transformation to obtain a concatenated eigenvector.

[0112] Alternatively, the weight matrix corresponding to the object to be updated and its first-order neighbor is used to perform a linear transformation on the eigenvectors of the two to obtain new eigenvectors corresponding to the two; and the new eigenvectors corresponding to the two are connected to obtain a connected eigenvector.

[0113] Specifically, when the dimension of the feature vectors of the object to be updated and its first-order neighbor is the same, the connection feature vector is obtained as follows:

[0114] or, Among them, k is the serial number of the attention head, is the connection feature vector in the k-th attention calculation, is the linear transformation matrix related to the real node in the kth attention calculation, i is the serial number of the object to be updated, h i Indicates that the object to be updated is a real node, and its eigenvector is h i ,|| represents the connection operation, j is the first-order neighbor number of the object to be updated, h j Indicates that the jth first-order neighbor is a real node, and its eigenvector is h j .

[0115] or,

[0116] or, or, or, in, The linear transformation matrix of the edge correlation in the k-th attention calculation, ξ j(pm) Indicates that the jth first-order neighbor is an edge, which is the edge between the pth node and the mth node, and its eigenvector is ξ j(pm) ξ j(pn) Indicates that the jth first-order neighbor is an edge, which is the edge between the pth node and the nth node, and its eigenvector is ξ j(pn) .

[0117] When the dimension of the eigenvector of the object to be updated and its first-order neighbor is different, the eigenvector of the first-order neighbor is linearly transformed using the dimension transformation matrix to obtain a new eigenvector of the first-order neighbor with the same dimension as the eigenvector of the object to be updated; the eigenvector of the object to be updated and the new eigenvector of the first-order neighbor are concatenated and then linearly transformed to obtain a concatenated eigenvector;

[0118] Alternatively, the weight matrices corresponding to the object to be updated and its first-order neighbor are used to perform linear transformation on the eigenvectors of the two to obtain new eigenvectors corresponding to the two respectively; and the new eigenvectors corresponding to the two are connected to obtain a connected eigenvector.

[0119] Among them, the cases where the dimension of the feature vector of the object to be updated and its first-order neighbor is different include: the object to be updated is a real node, and the first-order neighbor is an edge or a global virtual node; the object to be updated is an edge, and the first-order neighbor is a real node or a global virtual node; the object to be updated is a global virtual node, and the first-order neighbor is a real node or an edge.

[0120] When using a dimensionality transformation matrix to linearly transform the eigenvectors of first-order neighbors, the representation of those eigenvectors is also transformed. Because different objects (a collective term for real nodes, edges, and global virtual nodes) represent different types of eye movement parameters, their raw data is often difficult to reconcile. Using a dimensionality transformation matrix to alter the representation of the eigenvectors of first-order neighbors makes it easier to extract similarities between them when connecting them for attention calculations.

[0121] That is, the technical solution of this dimensional transformation has two effects: on the one hand, it realizes dimensional unification, which is the basis for connecting the two to form a new feature vector; on the other hand, it is conducive to extracting the similarity between the two when performing attention calculation after the two are connected.

[0122] Of course, first performing a linear transformation on the object to be updated and its first-order neighbors, followed by concatenation, also achieves these two effects. It’s worth noting that this linear transformation achieves higher-level expressiveness regardless of whether the feature vector dimensions of the object to be updated and its first-order neighbors are the same.

[0123] Specifically, when the dimension of the feature vectors of the object to be updated and its first-order neighbors is different, the connection feature vector is obtained as follows:

[0124] or, Among them, ξ j(mn) Indicates that the jth first-order neighbor is an edge, which is the edge between the mth node and the nth node, and its eigenvector is ξ j(mn) ; and The dimensions are the same; h is the dimension transformation matrix of the edge to the real node in the kth attention calculation; i and Same dimensions.

[0125] or,

[0126] or, Among them, ξ i(mn) Indicates that the object to be updated is an edge, and its eigenvector is ξ i(mn) ; and The dimensions are the same; is the dimension transformation matrix of the real node to edge in the k-th attention calculation; i(mn) and Same dimensions.

[0127] or,

[0128] or, in, is the linear transformation matrix related to the global virtual node in the k-th attention calculation, g j Indicates that the jth first-order neighbor is a global virtual node, and its eigenvector is g j ; and The dimensions are the same; is the dimension transformation matrix of the global virtual node to the real node in the kth attention calculation; k i and Same dimensions.

[0129] or,

[0130] or, in, and The dimensions are the same; is the dimension transformation matrix of the global node edge in the k-th attention calculation; i(mn) and Same dimensions.

[0131] or,

[0132] or, Among them, g i Indicates that the object to be updated is a global virtual node, and its eigenvector is g i ; and The dimensions are the same; is the dimensional transformation matrix of the real node to the global virtual node in the kth attention calculation; g i and Same dimensions.

[0133] or,

[0134] or, in, and The dimensions are the same; is the dimension transformation matrix of the edge to the global virtual node in the kth attention calculation; g i and Same dimensions.

[0135] The above describes the process of obtaining the connected feature vector in detail. The following is further explained: according to the self-attention mechanism, the attention score of the connected feature vector is calculated; and the product of the attention score and the feature vector of its corresponding first-order neighbor is calculated.

[0136] The attention score can be obtained in the following ways:

[0137] or,

[0138]

[0139] Among them, k is the sequence number of the attention head, i is the sequence number of the object to be updated, is the attention score of the jth first-order neighbor relative to the object to be updated in the kth attention calculation, LeakyReLU(·) is the activation function, and a k is the parameter of the single-layer feedforward neural network in the k-th attention calculation, a kT for a k The transpose of is the connection feature vector in the k-th attention calculation, is the attention score of the jth first-order neighbor after regularization in the kth attention calculation, exp(·) is the exponential function with the real number e as the base, are all the sequence numbers of the first-order neighbors of the object to be updated.

[0140] Optionally, the product of the attention score and the feature vector of its corresponding first-order neighbor is calculated, a nonlinear transformation is performed on all the concatenated products, and the nonlinear transformation is used as the update result of the object to be updated, including:

[0141]

[0142] Among them, v i is the feature vector of the object to be updated, v′ i is the update result of the object to be updated, K is the total number of attention heads, σ(·) is the activation function, is the linear transformation matrix of the jth first-order neighbor in the kth attention calculation, v j is the eigenvector of the first-order neighbor, specifically the eigenvector of a real node, the eigenvector of an edge, or the eigenvector of a global virtual node.

[0143] Alternatively, a nonlinear transformation is performed on the average of all the products after the connection, and the nonlinear transformation is used as the update result of the object to be updated, including:

[0144]

[0145] Among them, v i is the feature vector of the object to be updated, v′ i is the update result of the object to be updated, K is the total number of attention heads, σ(·) is the activation function, is the linear transformation matrix of the jth first-order neighbor in the kth attention calculation, v jis the eigenvector of the first-order neighbor, specifically the eigenvector of the real node, the eigenvector of the edge, or the eigenvector of the global virtual node, is the product.

[0146] The following further explains the entire process of the cognitive impairment identification method based on eye movement data.

[0147] Optionally, before regressing the fused feature vector to obtain the cognitive impairment assessment result, it also includes: dividing the eye movement data to be processed into multiple time segments according to the time window; for part of the eye movement data in each time segment, obtaining its corresponding fused feature vector, and forming a fused feature vector sequence.

[0148] On this basis, the fused feature vector is regressed to obtain the cognitive impairment assessment result, including: inputting the fused feature vector sequence into a long short-term memory neural network (Long Short-Term Memory, LSTM) or its derivative neural network, and using the output result of the LSTM or its derivative neural network as the cognitive impairment assessment result; or inputting the fused feature vector sequence into a Transformer-based neural network, and using the output result of the Transformer-based neural network as the cognitive impairment assessment result.

[0149] Among them, the neural networks derived from LSTM include but are not limited to: Gated Recurrent Unit (GRU), Simple Recurrent Units (SRU), Densely-connected Bi-directional LSTM for human action recognition (DB-LSTM), Bidirectional Long Short-Term Memory Network (BidirectionalLSTM, BiLSTM), etc., which are not listed here one by one.

[0150] like Figure 4 As shown, after combining the above technical solutions, the method for identifying cognitive impairment based on eye movement data includes:

[0151] S401: Divide the eye movement data to be processed into multiple time segments according to the time window.

[0152] S402 : Initializing a feature vector of each real node according to the gaze-related eye movement data in the eye movement data to be processed.

[0153] S403 : Initialize the feature vector of each edge according to the eye movement data related to saccades and / or pursuits in the eye movement data to be processed.

[0154] S404: Construct a global virtual node, and initialize a feature vector of the global virtual node based on the eye movement data to be processed, which appear in the gaze, saccade and tracking processes.

[0155] S405: In each time segment, the graph structure composed of real nodes, edges and global virtual nodes is updated more than once, all updated feature vectors are fused, and the fused feature vectors in each time segment are obtained to form a fused feature vector sequence.

[0156] S406. Input the fused feature vector sequence into the LSTM or its derivative neural network, and use the output result of the LSTM or its derivative neural network as the cognitive impairment assessment result; or, input all the fused feature vectors into the Transformer-based neural network, and use the output result of the Transformer-based neural network as the cognitive impairment assessment result.

[0157] LSTM or its derivative neural networks, Transformer, all have time series processing capabilities, but their time series processing capabilities are not infinite. In the embodiment of the present application, the eye movement data to be processed is first divided into multiple time segments according to the time window. Then, the fused feature vector within each time segment is extracted using the improved GATs. This can reduce the amount of data and the length of the time series to within the time series processing capabilities of LSTM or its derivative neural networks, Transformer, and thus obtain more accurate cognitive impairment assessment results.

[0158] Finally, the eye movement data related to fixation, the eye movement data related to saccade and / or pursuit, and the eye movement data occurring during fixation, saccade and pursuit are further explained.

[0159] Optionally, the gaze-related eye movement data includes one or more of the following: gaze point position, distance error between the gaze point and the target position, total gaze duration, number of gazes, average single gaze duration, timestamp of the first gaze, and eye gaze stability.

[0160] The explanation of each type of data is as follows:

[0161] Gaze point location: This is the screen coordinate of the gaze point, reflecting gaze distribution. Patients with MCI have more dispersed and chaotic gaze patterns. This is especially true when performing tasks, where it's difficult for them to focus on the target. For example, in a countersaccade task, patients with MCI are more likely to fixate on distractors than the target.

[0162] Normally, the superior colliculus inhibits reflexive saccades toward distractors, generating countersaccades under the control of the frontal eye field (FEF). However, in patients with MCI, the inhibitory function of the dorsolateral prefrontal cortex (DLPFC) is weakened, resulting in a preponderance of reflexive saccades toward distractors.

[0163] The distance error between the fixation point and the target location is the distance between the fixation point coordinates and the coordinates representing the AOI. The fixation point coordinates of patients with MCI may deviate further from the AOI coordinates. This error may be exacerbated by the presence of weakened functional connectivity in the cerebellar-parietal circuit in patients with MCI.

[0164] Total fixation duration: The total amount of time spent fixating on the AOI. MCI patients experience longer total fixations, especially during complex tasks, as they require more time to integrate information. Reduced myelination of white matter fiber tracts (such as the superior and inferior longitudinal fasciculi) results in less efficient transmission of visual information to the anterior frontal and temporal lobes.

[0165] Fixation count: The number of times a fixation is made on the AOI. Patients with MCI exhibit significantly increased fixation counts, manifesting as frequent, repetitive saccades, particularly during tasks requiring memory or decision-making. Damage to the entorhinal cortex-hippocampus circuit in the temporal lobe impairs episodic memory retrieval, leading to the need for repetitive fixations to compensate for memory deficits.

[0166] The increase in fixation times is associated with an increase in the working memory load of the DLPFC and a decrease in the efficiency of visual feature integration in the inferior parietal lobule (IPL), as well as an enhancement of the conflict detection signal in the anterior cingulate cortex (ACC), leading to an increase in driven corrective saccades.

[0167] Average single fixation duration: the ratio of total fixation duration to the number of fixations. Individual fixation durations in MCI patients may be shortened or prolonged, depending on the type of task. In tasks requiring rapid decision-making, fixation durations are shortened due to low information extraction efficiency; whereas in tasks requiring the integration of complex information, fixation durations are prolonged due to increased cognitive load. The weakened cholinergic neurotransmission in MCI patients leads to rigid allocation of attention resources, making it difficult to flexibly adjust fixation strategies to meet different task demands.

[0168] Timestamp of first fixation: The moment when the fixation point first reaches the AOI range. The timestamps of first fixations in different AOIs can reflect the cognitive order of MCI patients.

[0169] Gaze stability refers to the phenomenon that the duration of fixed gaze gradually increases and the amplitude of saccades gradually decreases until the gaze is fixed on a certain point. Gaze stability is significantly reduced in patients with MCI. Gaze stability depends on the coordinated regulation of brainstem gaze centers (such as the superior colliculus) and the parietal-prefrontal attention network. The weakened brainstem-cortical connection and parietal lobe atrophy in MCI patients may lead to difficulty in maintaining gaze. The functional integrity of the horizontal gaze center in the pons (Paramedian Pontine Reticular Formation, PPRF) and the vertical gaze center in the midbrain (involved by the interstitial nucleus of Cajal) also affects gaze stability.

[0170] Optionally, the saccade-related eye movement data includes one or more of the following: saccade amplitude, saccade angle, and saccade duration.

[0171] The explanation of each type of data is as follows:

[0172] Saccadic amplitude: This refers to the length of the path the gaze point moves between two real nodes. Patients with MCI may experience overshoots or undershoots during goal-directed saccade tasks, especially for distant targets. Abnormal saccade amplitude is associated with motor planning errors in the cerebellar hemispheres (dentate nucleus-thalamic pathway) and visual space coordinate transformation disorders in the superior parietal lobule (SPL). Purkinje cell degeneration leads to insufficient feedback correction, causing overshoots or undershoots.

[0173] Saccadic angle: The angle of gaze movement relative to the line connecting two real nodes. Patients with MCI have increased saccade angle deviation, potentially saccades toward distractors rather than the target, and their angular correction is slow. This angular deviation stems from the failure of the DLPFC to inhibit the basal ganglia-superior colliculus reflex pathway (reduced dopaminergic transmission) and abnormalities in active saccade direction encoding by neurons deep in the superior colliculus, resulting in a predominance of distractor-driven reflex saccades.

[0174] Saccadic duration: The time it takes for the gaze point to move from one real node to another. Saccadic duration is significantly prolonged in MCI patients, including prolonged latency (initiation time) and execution time (time required to complete the saccade). Prolonged latency is associated with delayed goal-directed saccade instructions in the FEF, decreased memory location efficiency in the hippocampal-parietal circuit, and an increased threshold for behavioral initiation due to reduced dopaminergic transmission in the basal ganglia. Reduced activity of PPRF neurons or decreased white matter integrity of the frontopontine tract leads to delayed conduction of saccade pulse signals, resulting in prolonged execution time.

[0175] Optionally, the tracking-related eye movement data includes one or more of the following: angular difference between the gaze point and the target trajectory, velocity difference, trajectory deviation distance, smooth tracking delay, smooth tracking acceleration, tracking path offset angle, and average tracking position error.

[0176] The explanation of each type of data is as follows:

[0177] The angular difference between the gaze point and the target trajectory is the difference in angle between the gaze point and the target trajectory. This angular difference is significantly increased in patients with MCI. This angular difference is associated with decreased predictive coding of motion trajectory in the posterior parietal lobe (angular gyrus) and dysregulation of the vestibulo-ocular reflex (VOR), leading to a decreased ability to proactively compensate for changes in target direction.

[0178] The speed difference between the fixation point and the target trajectory: The speed difference between the fixation point and the target trajectory is measured relative to the target trajectory. The eye movement velocity of patients with MCI often lags behind the target velocity, especially when the target accelerates or decelerates, which manifests as a speed chase phenomenon. This speed lag is associated with decreased dopaminergic gain regulation in the ventral basal ganglia pathway and delayed motor velocity signal transmission from the pontine nuclei to the anterior cerebellum, resulting in an inability for eye movement velocity to match target changes in real time.

[0179] Trajectory deviation distance: In the test screen, the moving target has a preset trajectory. The trajectory deviation distance indicates the deviation between the actual coordinates of the gaze point and the coordinates of the moving target when the moving target moves within a unit of time. The trajectory deviation distance of MCI patients is significantly increased, especially when tracking curved or broken line trajectories, the deviation of the gaze point from the target path is greater. Trajectory deviation is associated with motion vector calculation defects in the middle temporal area (MT) / medial superior temporal area (MST) of the intraparietal sulcus (IPS) and path integration errors in the cerebellar hemispheres (through the superior fronto-occipital fasciculus). Parietal lobe atrophy leads to a decrease in the real-time modeling ability of the target motion trajectory.

[0180] Smooth pursuit latency (SPL) is the time difference between the start of movement of a moving target and the start of gaze tracking. Patients with MCI have significantly prolonged SPL. This delay is associated with delayed tracking planning instructions in the supplementary eye field (SEF), decreased efficiency of cortical-vestibular signal integration in the pontine nuclei, and slower signal conduction due to demyelination in white matter pathways (such as the frontopontine fasciculus).

[0181] Smooth pursuit acceleration refers to the acceleration experienced when gaze tracks a moving target. Individuals with MCI have a reduced ability to regulate eye movement acceleration, manifesting as insufficient or excessive acceleration / deceleration. Acceleration dysregulation is associated with abnormal temporal coding in the dentate nucleus-thalamus pathway and diminished inhibitory control of the dentate nucleus by Purkinje cells, resulting in a mismatch between the timing of eye movement velocity changes and target acceleration.

[0182] Tracking path deviation angle: This represents the angular difference between the direction of gaze movement and the direction of the moving target per unit time. Patients with MCI exhibit increased overall tracking path deviation angles, particularly under complex background interference, with the tracking path deviating toward the distractor. Tracking path deviation is associated with decreased efficiency of the ACC-IPL conflict resolution loop and increased interference of the default mode network on the dorsal attention network, making attention more susceptible to distraction during tracking.

[0183] Mean tracking position error: The average distance deviation between the gaze point and the moving target during the entire process of tracking the moving target. The mean tracking position error is significantly increased in patients with MCI.

[0184] Optionally, the eye movement data occurring during gaze, saccade and pursuit processes include one or more of the following: number of blinks, longest blink interval, average pupil diameter, pupil change speed, eye wandering ratio or frequency, total eye wandering time, task completion time and total movement distance for task completion.

[0185] The explanation of each type of data is as follows:

[0186] Number of blinks: The average number of blinks during the statistical period. The statistical period depends on the actual situation and can be the entire test time or the time to complete a single task during the test. The blinking frequency of MCI patients is significantly higher than that of normal people. Functional changes in the basal ganglia of the brain (especially the D2 receptors in the striatum) can affect the blink rate, causing MCI patients to blink more frequently. Because people inhibit blinking when concentrating, weakened attention control can also lead to increased blinking. When the brain areas responsible for maintaining attention (such as the frontal and parietal attention networks) are damaged, patients find it difficult to inhibit blinking, resulting in an increase in blinking.

[0187] Average blink interval: The average of the time intervals between multiple blinks. The average blink interval of MCI patients is significantly shorter than that of normal people. Functional changes in the basal ganglia (especially the D2 receptors in the striatum) can affect the blink rate, shortening the average blink interval of MCI patients. Because people inhibit blinking when concentrating, weakened attention control can also lead to a shortened average blink interval. When the brain areas responsible for maintaining attention (such as the frontal and parietal attention networks) are damaged, patients find it difficult to inhibit blinking, resulting in a shorter average blink interval.

[0188] Average pupil diameter: The pupil diameter within the statistical period. The statistical period depends on the actual situation and can be the entire test time or the time to complete a single task during the test. The average pupil diameter of MCI patients is significantly smaller than that of normal people. The locus coeruleus (LC)-norepinephrine system controls the basal tension of the pupil. Early AD pathology affects the LC, causing it to degenerate, which in turn causes the pupil base diameter of MCI patients to become smaller and the average pupil diameter during the task becomes smaller. In addition, pupil size is also regulated by the parasympathetic nervous system (Edinger-Westphal nucleus of the oculomotor nerve). An imbalance in the central autonomic regulation of MCI can also lead to a decrease in the average pupil diameter.

[0189] Pupil change speed: The rate of change in pupil diameter per unit time, or the average rate of change within a statistical period. MCI patients experience slower pupil change speeds. MCI patients experience dysfunction in the locus coeruleus-pretectal nucleus pathway, leading to insufficient sympathetic drive and slower pupil dilation. Because cognitive processing is slowed in MCI patients, pupil change delays after stimulation may also increase, resulting in slower pupil change speeds.

[0190] Eye wandering rate or frequency: When tracking a moving target, the gaze suddenly deviates from the moving target and then returns. Each time the gaze "leaves the moving target and quickly returns to the moving target" occurs, it is counted as one eye wandering event. The eye wandering frequency is the number of times the eyes wander per unit time, and the eye wandering rate is the ratio of eye wandering to all saccades. MCI patients usually have a higher eye wandering rate or frequency. The function of the frontal and parietal attention networks in MCI patients is reduced, resulting in an increased eye wandering rate or frequency. The default mode network in MCI is overactive, which occupies cognitive resources, causing eye wandering, resulting in an increased eye wandering rate or frequency.

[0191] Total eye wandering duration: The cumulative time the gaze point deviates from the real node and / or tracks the meaningful edge. MCI patients typically have longer eye wandering durations. Impaired function in the prefrontal cortex (such as the dorsolateral prefrontal cortex) and parietal cortex, which are responsible for maintaining attention and working memory, makes it difficult for MCI patients to focus on a single target for a long time. MCI patients have weakened sustained attention and insufficient working memory, forcing them to frequently interrupt operations to obtain information. The brains of MCI patients may enter a wandering state, prolonging the time the eyes wander.

[0192] Task completion time: refers to the time required to complete the task specified in the test screen (such as completing a certain fixation sequence). The task completion time of MCI patients is significantly increased. MCI patients experience frontal lobe dysfunction (affecting execution and decision-making speed) and white matter pathway damage (affecting brain processing speed) in the early stages, which leads to slower motor reactions; reduced coordination of the frontal lobe-basal ganglia-cerebellum circuit will also prolong completion time. The weakened function of the prefrontal cortex (responsible for planning and decision-making) and the parietal lobe (responsible for spatial processing) will also slow down MCI patients in tasks such as understanding tasks, formulating strategies, and executing operations.

[0193] Total distance traveled during task completion: This refers to the total distance the gaze moved during a task (e.g., completing a gaze sequence) specified in the test image. This overall measure of eye wandering is provided. MCI patients showed a significant increase in total distance traveled during task completion.

[0194] Figure 5 Schematic diagram of a device for identifying cognitive impairment based on eye movement data provided by an embodiment of the present application. The device for identifying cognitive impairment based on eye movement data can be implemented through software, hardware, or a combination of software and hardware.

[0195] Combine Figure 5 As shown, the cognitive impairment identification device based on eye movement data includes a real node initialization module 51, an edge initialization module 52, a global virtual node initialization module 53, an update module 54 and a regression module 55.

[0196] The real node initialization module 51 is used to initialize the feature vector of each real node according to the gaze-related eye movement data in the eye movement data to be processed; wherein the real node represents the AOI.

[0197] The edge initialization module 52 is used to initialize the feature vector of each edge according to the eye movement data related to saccades and / or pursuits in the eye movement data to be processed; wherein an edge represents a saccade and / or pursuit relationship between two real nodes.

[0198] The global virtual node initialization module 53 is used to construct a global virtual node and initialize the feature vector of the global virtual node according to the eye movement data to be processed, which appear in the gaze, saccade and tracking processes.

[0199] The update module 54 is used to update the graph structure composed of real nodes, edges and global virtual nodes more than once, fuse all the updated feature vectors, and obtain a fused feature vector; in each update, based on the multi-head attention mechanism, obtain the attention score of the object to be updated and its first-order neighbors, and update the object to be updated according to the feature vectors and corresponding attention scores of each first-order neighbor; wherein, the object to be updated is the real node, the directed edge or the global virtual node, and the first-order neighbors include directly connected real nodes and real nodes, real nodes and edges, and edges and edges; the global virtual node and each real node and each edge are first-order neighbors to each other.

[0200] The regression module 55 is used to regress the fused feature vector to obtain a cognitive impairment assessment result.

[0201] Optionally, the cognitive impairment identification device based on eye movement data further includes a time window division module, which is used to divide the eye movement data to be processed into multiple time segments according to the time window.

[0202] For the partial eye movement data in each time segment, the real node initialization module 51, the edge initialization module 52, the global virtual node initialization module 53 and the update module 54 are executed to obtain the fused feature vector corresponding to each time window.

[0203] Optionally, the cognitive impairment identification device based on eye movement data further includes a sequence construction module, which is used to form a fused feature vector sequence from the fused feature vectors of all time windows.

[0204] Optionally, the regression module 55 includes a first regression unit or a second regression unit.

[0205] Among them, the first regression unit is used to input the fused feature vector sequence into the LSTM or its derivative neural network, and use the output result of the LSTM or its derivative neural network as the cognitive impairment assessment result.

[0206] The second regression unit is used to input the fused feature vector sequence into the Transformer-based neural network, and use the output result of the Transformer-based neural network as the cognitive impairment assessment result.

[0207] Regarding the update process of each layer of the improved GATs, please refer to the specific embodiment of the cognitive impairment identification method based on eye movement data, which will not be repeated here.

[0208] In some embodiments, a cognitive impairment identification device based on eye movement data includes a processor and a memory storing program instructions. The processor is configured to execute the cognitive impairment identification method based on eye movement data provided in the aforementioned embodiment when executing the program instructions.

[0209] like Figure 6 As shown, an embodiment of the present application provides a cognitive impairment identification device based on eye movement data, including:

[0210] Processor 61 and memory 62 may also include a communication interface 63 and a bus 64. Processor 61, communication interface 63, and memory 62 may communicate with each other via bus 64. Communication interface 63 may be used for information transmission. Processor 61 may invoke logic instructions in memory 62 to execute the method for identifying cognitive impairment based on eye movement data provided in the aforementioned embodiment.

[0211] In addition, the logic instructions in the memory 62 can be implemented in the form of software functional units and stored in a computer-readable storage medium when sold or used as an independent product.

[0212] Memory 62, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of the present application. Processor 61 executes the software programs, instructions, and modules stored in memory 62 to perform functional applications and data processing, thereby implementing the methods in the above-mentioned method embodiments.

[0213] The memory 62 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the terminal device. Furthermore, the memory 62 may include high-speed random access memory and non-volatile memory.

[0214] An embodiment of the present application provides a cognitive impairment identification system based on eye movement data, including the cognitive impairment identification device based on eye movement data provided by the aforementioned embodiment.

[0215] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the method for identifying cognitive impairment based on eye movement data provided in the aforementioned embodiment.

[0216] An embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the cognitive impairment identification method based on eye movement data provided by the aforementioned embodiment.

[0217] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0218] The technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of the embodiments of the present application. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code, or a transient storage medium.

[0219] The above description and accompanying drawings sufficiently illustrate the embodiments of the present application to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, process, and other changes. The embodiments represent only possible variations. Unless expressly required, individual components and functions are optional, and the order of operations may vary. Portions and features of some embodiments may be included in or substituted for portions and features of other embodiments. Furthermore, the terms used in this application are intended only to describe the embodiments and are not intended to limit the claims. As used in the embodiments and in the claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, when used in this application, the terms "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. In the absence of further limitations, the phrase "comprising a..." does not preclude the presence of other identical elements in the process, method, or apparatus comprising the elements. In this document, each embodiment may focus on the differences from other embodiments, and similar portions between the embodiments may refer to each other. For methods, products, etc. disclosed in the embodiments, if they correspond to the method portion disclosed in the embodiments, the relevant portions may refer to the description of the method portion.

[0220] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application. Technicians can clearly understand that for the convenience and brevity of description, the specific working process of the above-described systems, devices and units can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0221] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units can be merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, the functional units in the embodiments of the present application may be integrated into a processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

[0222] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the system, method and computer program product according to the embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or code, and a part of a module, program segment or code comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a special hardware-based system that performs the function or action of the specification, or can be implemented by a combination of special hardware and computer instructions.

Claims

1. A method for identifying cognitive impairment based on eye movement data, characterized in that: include: Initialize the feature vector of each real node according to the gaze-related eye movement data in the eye movement data to be processed; wherein the real node represents the AOI; Initializing a feature vector of each edge based on saccadic and / or pursuit-related eye movement data in the to-be-processed eye movement data; wherein an edge represents a saccadic and / or pursuit relationship between two real nodes; Constructing a global virtual node and initializing a feature vector of the global virtual node based on the eye movement data to be processed, which appear in the gaze, saccade and pursuit processes; The graph structure composed of real nodes, edges, and global virtual nodes is updated more than once, and all updated feature vectors are fused to obtain a fused feature vector. In each update, based on the multi-head attention mechanism, the attention score of the object to be updated and its first-order neighbors is obtained, and the object to be updated is updated according to the feature vectors and corresponding attention scores of each first-order neighbor. Among them, the object to be updated is the real node, the directed edge, or the global virtual node. The first-order neighbors include directly connected real nodes and real nodes, real nodes and edges, and edges and edges. The global virtual node and each real node and each edge are first-order neighbors of each other. The fused feature vector was regressed to obtain the cognitive impairment assessment results.

2. The method for identifying cognitive impairment according to claim 1, wherein: Based on the multi-head attention mechanism, the attention score of the object to be updated and its first-order neighbors is obtained, and the object to be updated is updated according to the feature vectors and corresponding attention scores of each first-order neighbor, including: Repeat the following process multiple times to obtain multiple products: concatenate the feature vectors of the object to be updated and its first-order neighbors to obtain a concatenated feature vector; calculate the attention score of the concatenated feature vector based on the self-attention mechanism; calculate the product of the attention score and the feature vector of its corresponding first-order neighbor; A nonlinear transformation is performed on all the products after the connection, or on the average value of all the products, and the nonlinear transformation is used as the update result of the object to be updated.

3. The method for identifying cognitive impairment according to claim 2, wherein: Connect the feature vectors of the object to be updated and its first-order neighbors to obtain the connected feature vector, including: When the eigenvector dimensions of the object to be updated and its first-order neighbor are the same, first concatenate the corresponding eigenmatrices of the two to obtain a new eigenvector, and then perform a linear transformation to obtain a concatenated eigenvector. Alternatively, a weight matrix corresponding to the object to be updated and its first-order neighbor is used to perform a linear transformation on the eigenvectors of the two to obtain new eigenvectors corresponding to the two; and the new eigenvectors corresponding to the two are connected to obtain a connected eigenvector; When the dimension of the eigenvector of the object to be updated and its first-order neighbor is different, the eigenvector of the first-order neighbor is linearly transformed using the dimension transformation matrix to obtain a new eigenvector of the first-order neighbor with the same dimension as the eigenvector of the object to be updated; the eigenvector of the object to be updated and the new eigenvector of the first-order neighbor are concatenated and then linearly transformed to obtain a concatenated eigenvector; Alternatively, the weight matrices corresponding to the object to be updated and its first-order neighbor are used to perform linear transformation on the eigenvectors of the two to obtain new eigenvectors corresponding to the two respectively; and the new eigenvectors corresponding to the two are connected to obtain a connected eigenvector.

4. The method for identifying cognitive impairment according to claim 3, wherein: When the dimension of the feature vectors of the object to be updated and its first-order neighbor is the same, the connection feature vector is obtained as follows: or, Among them, k is the serial number of the attention head, is the connection feature vector in the k-th attention calculation, is the linear transformation matrix related to the real node in the kth attention calculation, i is the serial number of the object to be updated, h i Indicates that the object to be updated is a real node, and its eigenvector is h i ,|| represents the connection operation, j is the first-order neighbor number of the object to be updated, h j Indicates that the jth first-order neighbor is a real node, and its eigenvector is h j ; or, or, or, or, in, The linear transformation matrix of the edge correlation in the k-th attention calculation, ξ j(mn) Indicates that the jth first-order neighbor is an edge, which is the edge between the mth node and the nth node, and its eigenvector is ξ j(m) ξ j(pm) Indicates that the jth first-order neighbor is an edge, which is the edge between the pth node and the mth node, and its eigenvector is ξ j(pm) ξ j(pn) Indicates that the jth first-order neighbor is an edge, which is the edge between the pth node and the nth node, and its eigenvector is ξ j(pn) ; When the dimension of the feature vector of the object to be updated and its first-order neighbor is different, the connection feature vector is obtained as follows: or, in, and The dimensions are the same; h is the dimension transformation matrix of the edge to the real node in the kth attention calculation; i and The dimensions are the same; or, or, in, and The dimensions are the same; is the dimension transformation matrix of the real node to edge in the k-th attention calculation; i(mn) and The dimensions are the same; or, or, in, is the linear transformation matrix related to the global virtual node in the k-th attention calculation, g j Indicates that the jth first-order neighbor is a global virtual node, and its eigenvector is g j ; and The dimensions are the same; h is the dimension transformation matrix of the global virtual node to the real node in the kth attention calculation; i and The dimensions are the same; or, or, in, and The dimensions are the same; is the dimension transformation matrix of the global node edge in the k-th attention calculation; i(mn) and The dimensions are the same; or, or, Among them, g i Indicates that the object to be updated is a global virtual node, and its eigenvector is g i ; and The dimensions are the same; is the dimensional transformation matrix of the real node to the global virtual node in the kth attention calculation; g i and The dimensions are the same; or, or, in, and The dimensions are the same; is the dimension transformation matrix of the edge to the global virtual node in the kth attention calculation; g i and Same dimensions.

5. The method for identifying cognitive impairment according to claim 2, wherein: According to the self-attention mechanism, the attention score of the connected feature vector is calculated, including: or, Among them, k is the sequence number of the attention head, i is the sequence number of the object to be updated, is the attention score of the jth first-order neighbor relative to the object to be updated in the kth attention calculation, LeakyReLU(·) is the activation function, and a k is the parameter of the single-layer feedforward neural network in the k-th attention calculation, a kT for a k The transpose of is the connection feature vector in the k-th attention calculation, is the attention score of the jth first-order neighbor after regularization in the kth attention calculation, exp(·) is the exponential function with the real number e as the base, are all the sequence numbers of the first-order neighbors of the object to be updated; Calculate the product of the attention score and the feature vector of its corresponding first-order neighbor, perform a nonlinear transformation on all the connected products, and use the nonlinear transformation as the update result of the object to be updated, including: Perform a nonlinear transformation on the average of all connected products, and use the nonlinear transformation as the update result of the object to be updated, including: Among them, v i is the feature vector of the object to be updated, v′ i is the update result of the object to be updated, K is the total number of attention heads, σ(·) is the activation function, is the linear transformation matrix of the jth first-order neighbor in the kth attention calculation, v j is the eigenvector of the first-order neighbor, specifically the eigenvector of the real node, the eigenvector of the edge, or the eigenvector of the global virtual node, is the product.

6. The method for identifying cognitive impairment according to any one of claims 1 to 5, characterized in that: Before regressing the fused feature vector to obtain the cognitive impairment assessment result, the method further includes: dividing the eye movement data to be processed into multiple time segments according to the time window; obtaining the corresponding fused feature vector for the partial eye movement data in each time segment, and forming a fused feature vector sequence; The fused feature vector is regressed to obtain a cognitive impairment assessment result, including: inputting the fused feature vector sequence into an LSTM or a derivative neural network thereof, and using the output result of the LSTM or the derivative neural network thereof as the cognitive impairment assessment result; or inputting the fused feature vector sequence into a Transformer-based neural network, and using the output result of the Transformer-based neural network as the cognitive impairment assessment result.

7. The method for identifying cognitive impairment according to any one of claims 1 to 5, characterized in that: Gaze-related eye movement data include one or more of the following: gaze point location, distance error between the gaze point and the target location, total gaze duration, number of fixations, average single fixation duration, timestamp of first fixation, and gaze stability; The eye movement data related to saccades include one or more of the following: saccade amplitude, saccade angle, and saccade duration; Tracking-related eye movement data includes one or more of the following: angular difference between the gaze point and the target trajectory, velocity difference, trajectory deviation distance, smooth pursuit delay, smooth pursuit acceleration, tracking path deviation angle, and average tracking position error; Eye movement data that appear during gaze, saccade, and pursuit include one or more of the following: number of blinks, average blink interval, average pupil diameter, pupil change speed, eye wandering ratio or frequency, total eye wandering time, task completion time, and total movement distance during task completion.

8. A device for identifying cognitive impairment based on eye movement data, characterized in that: include: A real node initialization module is used to initialize the feature vector of each real node according to the gaze-related eye movement data in the eye movement data to be processed; wherein the real node represents the AOI; an edge initialization module, configured to initialize a feature vector of each edge based on saccadic and / or pursuit-related eye movement data in the processed eye movement data; wherein an edge represents a saccadic and / or pursuit relationship between two real nodes; A global virtual node initialization module is used to construct a global virtual node and initialize the feature vector of the global virtual node based on the eye movement data to be processed, which appears in the gaze, saccade and tracking processes; The update module is used to update the graph structure composed of real nodes, edges, and global virtual nodes more than once, and fuse all updated feature vectors to obtain a fused feature vector. In each update, based on the multi-head attention mechanism, the attention score of the object to be updated and its first-order neighbors is obtained, and the object to be updated is updated according to the feature vectors and corresponding attention scores of each first-order neighbor. The object to be updated is the real node, the directed edge, or the global virtual node. The first-order neighbors include directly connected real nodes and real nodes, real nodes and edges, and edges and edges. The global virtual node and each real node and each edge are first-order neighbors of each other. The regression module is used to regress the fused feature vector to obtain the cognitive impairment assessment results.

9. A device for identifying cognitive impairment based on eye movement data, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to perform the method for identifying cognitive impairment based on eye movement data according to any one of claims 1 to 7 when executing the program instructions.

10. A cognitive impairment identification system based on eye movement data, characterized in that: It includes the cognitive impairment identification device based on eye movement data as described in claim 8.

Images