A system and method for identifying levels of cognitive impairment

Abstract

This invention provides a system and method for identifying cognitive impairment levels. The system includes: a data acquisition module for acquiring behavioral data and cognitive data from different modalities collected during a predetermined cognitive impairment test of a subject, both of which are time-series data with at least two variables, and the cognitive data is obtained based on EEG data from multiple brain regions; a first inference module for performing causal inference based on each variable within each modality to obtain a first causal correlation graph; a second inference module for using an encoder in a trained convolutional autoencoder to map the cognitive data to brain region features consistent with the variable data structure of the behavioral data, and performing cross-modal causal inference based on the variables of the behavioral data and the brain region features to obtain a second causal correlation graph between variables across modalities; and a cognitive impairment identification module for determining the cognitive impairment level based on the first and second causal correlation graphs for each modality.

Description

Technical Field

[0001] This invention relates to the field of computer science, more specifically to the field of causal inference technology for multimodal data, and more particularly to a system and method for identifying levels of cognitive impairment. Background Technology

[0002] In recent years, research on causal relationships has become a crucial part of the field of artificial intelligence. Based on directed acyclic graph structures and Bayesian models, various causal discovery algorithms have been proposed.

[0003] Traditional causal discovery (or causal inference) algorithms are generally classified into three categories: constraint-based algorithms, score-based algorithms, and algorithms based on functional causal models (FCMs). Peter-Clark (PC) is a leading expert in this field. [1] Algorithm, Greedy Equivalence Search (GES) [2] Linear Non-Gaussian Acyclic Model (LiNGAM) [3] The algorithms are representative algorithms among these three types of algorithms.

[0004] Constraint-based algorithms rely on statistical tests of conditional independence, typically satisfying Markov conditions and faithfulness assumptions. They also assume the absence of unobservable confounding factors, allowing them to quickly find a Markov equivalence class. Most constraint-based algorithms first find possible undirected graphs (i.e., the skeleton), then determine the collision-joining structures, also called V-structures, and finally search for as many other directed edges as possible. The representative PC (Peter-Clark) algorithm's processing steps are as follows:

[0005] ① Construct a completely undirected graph on the set of variables X. (That is, there is an undirected edge between every two variables) and take n = 0;

[0006] ② Select in an orderly manner In the context of adjacent variable pairs X and Y, X and Y need to satisfy... The number of elements in is greater than or equal to express (The set consisting of all nodes adjacent to X) also needs to be selected. A subset S of n elements. If, given S, X and Y are d-separated, then from... Remove the edge XY from the subset S and add the variables from the subset S to the sets Sepset(X,Y) and Sepset(Y,X); if X and Y are not d-separated given S, then keep the edge XY. This step requires checking all adjacent variable pairs X and Y that satisfy the conditions, and... A subset S of all elements having the number n.

[0007] ③ When n = n + 1, return to step ②, until for any pair of adjacent variables X and Y, The number of elements in each is less than n ( (This process changes continuously in steps ② and ③).

[0008] ④ Regarding the result obtained in step ③ For each triple of the form XYz (i.e., X and Y, Y and Z are adjacent, but X and Z are not adjacent), XYz is transformed into the V-structure X→Y←z if and only if Y is not in Sepset(X, Z).

[0009] ⑤ Finally, repeat the following steps until all edges are directed edges:

[0010] (i) If there is a directed edge X→Y between X and Y, Y and Z are adjacent, but X and Z are not adjacent, and there are no other arrows pointing to Y, then change Yz to Y→z.

[0011] (ii) If there is a directed path from X to Y, and X and Y are adjacent, then change XY to X→Y;

[0012] The PC algorithm has low computational complexity. Let n be the number of nodes and k be the maximum degree of all nodes. In the worst case, the time complexity of the PC algorithm does not exceed [a certain value].

[0013] Score-based algorithms find causal structures by optimizing a well-defined scoring function. Since verifying fidelity hypotheses requires a large number of samples, score-based algorithms use goodness-of-fit tests instead of conditional independence tests, weakening the importance of fidelity hypotheses and thus reducing sample size requirements. A representative algorithm, GES (Greedy Equivalence Search), starts with an empty graph and can be roughly divided into two steps:

[0014] ① Adding directed edges: When adding a directed edge increases the value of the scoring function, the directed edge that increases the value of the scoring function the most will be added. The resulting model will be mapped to the appropriate Markov equivalence class. Then, continue adding directed edges until the value of the scoring function can no longer increase.

[0015] ② Deletion of directed edges: The GES algorithm will traverse all directed edges. If deleting an edge can make the value of the score function continue to increase, the edge will be deleted until deleting any edge can no longer make the value of the score function larger.

[0016] Algorithms based on function-based causal models avoid the problems of indistinguishability of Markov equivalence classes and the need for a large number of samples to verify the fidelity hypothesis. These algorithms obtain a complete causal graph by restricting the type of function in the causal mechanism and utilizing the asymmetry between causal and anti-causal directions. Defined in the variable set X = {X1, X2, ..., X...} n The causal model of a function on} is a triple. This represents the following system of equations:

[0017] X i =f i (X Pa(i；G) E i E i ~ε, i=1, 2,..., n (1)

[0018] Each equation represents X i and X i parent variable set (X i ←X j If and only if X j ∈X Pa(i；G) The functional relationship between f and ) i It is the corresponding causal mechanism, E i It is a noise variable, following a distribution ε, which can include all unobservable situations (such as human interference, equipment failure, environmental influences, etc.), and is usually assumed to be E. i (i = 1, 2, ..., n) are mutually independent.

[0019] Among them, the LiNGAM (Linear Non-Gaussian Acyclic Model) algorithm is the most classic algorithm based on a function-causal model. It makes the following assumptions: (1) the data generation process is linear; (2) there are no unobservable confounding factors; (3) the interference terms follow a non-Gaussian distribution with a variance of non-zero and are mutually independent. The LiNGAM model can be written in the following form:

[0020] X = BX + ε (2)

[0021] Where B is the coefficient matrix representing the linear causal relationship between variables, and is a strictly lower triangular matrix, and ε represents the noise vector. Transforming equation (2), we can obtain:

[0022] (IB)X=ε (3)

[0023] Let A = (IB) -1 We can obtain:

[0024] X=Aε (4)

[0025] Here, A is a non-strict lower triangular matrix. A can be solved using independent component analysis for equation (4). -1 This allows us to obtain the correct causal relationship between variables.

[0026] Pervasive sensing devices generally possess long-term sensing capabilities, enabling them to monitor and record long-sequence, multi-state user behavior over extended periods. This data is typically presented in a heterogeneous, multimodal format. Learning causal relationships between heterogeneous, multimodal samples plays a significant role in medical diagnosis; for example, behavioral and cognitive multimodal causal association analysis is an effective approach for early assessment of cognitive disorders such as dementia.

[0027] Existing constraint-based, fraction-based, and function-based causal model-based traditional causal discovery algorithms often perform well when calculating causal relationships between homogeneous data. However, when calculating causal relationships between heterogeneous multimodal data, due to differences in data types, distributions, and structures among different modalities, the algorithms can usually only find causal relationships between the same modality, and cannot effectively find causal relationships between heterogeneous multimodalities, resulting in a low completeness of the causal relationship graph.

[0028] In fields such as early screening for cognitive disorders, a low rate of finding all relevant causal relationship diagrams will directly and significantly affect the accuracy of cognitive ability grading. Summary of the Invention

[0029] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a system and method for identifying levels of cognitive impairment.

[0030] The objective of this invention is achieved through the following technical solution:

[0031] According to a first aspect of the present invention, a system for identifying cognitive impairment levels is provided, comprising: a data acquisition module for acquiring behavioral data and cognitive data of different modalities collected by a subject during a predetermined cognitive impairment test, wherein the behavioral data and the cognitive data are time-series data each having at least two variables, the variables between different modalities are heterogeneous data, and the different variables within the same modality are isomorphic data, and the cognitive data is obtained based on electroencephalogram data of multiple brain regions; a first inference module for performing causal inference based on each variable within each modality to obtain a first causal relationship graph between variables within each modality; a second inference module for mapping the cognitive data to brain region features consistent with the variable data structure of the behavioral data using an encoder in a trained convolutional autoencoder, and performing cross-modal causal inference based on the variables of the behavioral data and the brain region features to obtain a second causal relationship graph between variables across modalities; and a cognitive impairment identification module for determining the cognitive impairment level of the subject based on the first causal relationship graph of each modality and the second causal relationship graph across modalities.

[0032] Optionally, each behavioral data point includes behavioral accuracy and behavioral speed for multiple specified test items, where behavioral accuracy and behavioral speed each correspond to a variable, and each cognitive data point includes average EEG data for multiple brain regions, where the average EEG data for one brain region corresponds to a variable.

[0033] Optionally, each behavioral data point includes multi-axis sensing data acquired based on multi-axis inertial sensors, where the sensing data for each axis corresponds to a variable, and each cognitive data point includes average EEG data for multiple brain regions, where the average EEG data for each brain region corresponds to a variable.

[0034] Optionally, the encoder of the trained convolutional autoencoder is trained in the following manner: multiple training cognitive data are acquired, collected from multiple different personnel, with each personnel collecting one or more training cognitive data; the average EEG data of each brain region in each training cognitive data is used as one training data, resulting in multiple training data; using the multiple training data, a convolutional autoencoder including an encoder and a decoder is trained to obtain a trained convolutional autoencoder, wherein, during training: the encoder is configured to encode the training data to output brain region features with a data structure consistent with the variables of the behavioral data, and the decoder is configured to reconstruct the average EEG data of the corresponding brain region based on the brain region features; the parameters of the encoder and decoder are updated based on the reconstruction loss determined by the reconstructed average EEG data of the brain region and the input corresponding training data.

[0035] Optionally, the cognitive data includes average EEG data for six brain regions, namely the brain regions corresponding to the left frontal cortex, right frontal cortex, left parietal cortex, right parietal cortex, left occipital cortex, and right occipital cortex. The average EEG data for each brain region is the average value of the EEG data collected by all electrodes in that brain region.

[0036] Optionally, the cognitive impairment identification module is configured to: fuse the first causal association graph of each modality and the second causal association graph across modalities to obtain a fused causal association graph; and use a trained graph neural network classification model to determine the cognitive impairment level of the subject based on the fused causal association graph, wherein the trained graph neural network classification model includes a graph neural network and a classifier, the graph neural network is used to extract the graph embedding vector corresponding to the subject based on the fused causal association graph, and the classifier is used to determine the cognitive impairment level of the subject based on the graph embedding vector.

[0037] Optionally, the trained graph neural network classification model is trained as follows: A classification training set is acquired, which is collected from multiple individuals and includes multiple fused causal association graphs and labels for training. Each fused causal association graph for training corresponds to a label, the label indicating the ground truth value of the cognitive impairment level of the individual to which the fused causal association graph belongs; using the classification training set, the graph neural network classification model is trained to identify the cognitive impairment level based on the input fused causal association graphs for training, wherein the classification loss is determined based on the identified cognitive impairment level and the corresponding label, and the gradient is calculated based on the classification loss and backpropagated to update the parameters of the graph neural network and the classifier.

[0038] Optionally, the cognitive impairment level is divided into levels 0 to N, where level 0 indicates no cognitive impairment, and levels 1 to N correspond to the degree of cognitive impairment. The higher the value, the more severe the cognitive impairment.

[0039] According to a second aspect of the present invention, a method for identifying a level of cognitive impairment is provided, the method comprising: acquiring behavioral data and cognitive data of different modalities collected by a subject during a predetermined cognitive impairment test, wherein the behavioral data and the cognitive data are time-series data each having at least two variables, the variables between different modalities are heterogeneous data, and the different variables within the same modality are isomorphic data; performing causal inference based on each variable within each modality to obtain a first causal relationship graph between variables within each modality; mapping the behavioral data and the cognitive data to data with a consistent data structure using an encoder in a convolutional autoencoder, and performing causal inference based on the data with the consistent data structure to obtain a second causal relationship graph between variables across modalities; and determining the level of cognitive impairment of the subject based on the first causal relationship graph of each modality and the second causal relationship graph across modalities.

[0040] Optionally, the cognitive impairment level of the subject is determined as follows: the first causal association graph of each modality and the second causal association graph across modalities are fused to obtain a fused causal association graph; the cognitive impairment level of the subject is determined based on the fused causal association graph using a trained graph neural network classification model, wherein the trained graph neural network classification model includes a graph neural network and a classifier, the graph neural network is used to extract the graph embedding vector corresponding to the subject based on the fused causal association graph, and the classifier is used to determine the cognitive impairment level of the subject based on the graph embedding vector.

[0041] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the steps of the method described in the second aspect by executing the executable instructions. Attached Figure Description

[0042] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0043] Figure 1 This is a schematic diagram of the overall process according to an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a system for identifying levels of cognitive impairment according to an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of electrode placement according to the existing 10-20 international standard lead system;

[0046] Figure 4 This is a schematic diagram illustrating causal inference between homogeneous and heterogeneous data according to an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of the fusion of causal relationships in Experiment 1 of the present invention, in which healthy individuals perform single task 1.

[0048] Figure 6 A schematic diagram of the causal relationship of a group with mild cognitive impairment performing single task 1 in Experiment 1 according to an embodiment of the present invention;

[0049] Figure 7 A schematic diagram of the causal relationship of a group with severe cognitive impairment performing a single task 1 in Experiment 1 according to an embodiment of the present invention;

[0050] Figure 8 This is a schematic diagram of data windowing in Experiment 2 according to an embodiment of the present invention;

[0051] Figure 9 This is a schematic diagram of the Dillum angle field in Experiment 2 according to an embodiment of the present invention;

[0052] Figure 10 This is a schematic diagram of the network architecture of the convolutional autoencoder used in Experiment 2 according to an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0054] As mentioned in the background section, in fields such as early screening for cognitive diseases, a low completeness of relevant causal association graphs directly impacts the accuracy of cognitive ability grading. While researching causal association learning methods for heterogeneous multimodal data, the inventors discovered that existing causal discovery algorithms, failing to consider the differences in structure, distribution, and representation of heterogeneous multimodal data, often only find causal associations within the same modality, failing to identify causal associations between multimodal data.

[0055] Therefore, this invention aims to improve upon the shortcomings of existing methods by adding a step of consistent feature extraction for heterogeneous multimodal data, based on traditional causal discovery algorithms. In this invention, a method is proposed to extract brain region features from cognitive data in heterogeneous multimodal data using a convolutional autoencoder, mapping the multimodal data to a unified feature space (i.e., consistent features, including variables of cognitive data represented by brain region features and the original variables of behavioral data, both with consistent data structures). Then, based on the consistent features, causal relationships between variables across modalities are inferred to obtain causal relationships between heterogeneous multimodal data. For causal relationships between data of the same modality, causal inference is directly performed based on the original variables of each modality. Finally, isomorphic and heterogeneous causal relationships are fused to obtain a fused causal relationship used to determine the level of cognitive impairment. This invention's method, by extracting features from heterogeneous data, mapping these features to a consistent space, and then performing causal discovery based on these consistent features, can effectively find causal relationships between heterogeneous multimodal data, thereby improving the accuracy of cognitive impairment level identification.

[0056] Let's combine the following... Figure 2 The overall process of the embodiments of the present invention will be simplified to provide an intuitive understanding of the technical principles:

[0057] (1) Multimodal data and problem definition

[0058] {D1,D2} represents data from two modalities, where D... i This is the data for mode i, where i = 1 or 2. k is in mode i i Observed data for each variable, for example, D i1 D1 and D2 represent multiple consecutive sampled point values ​​of the first variable in mode i (time series data composed of point values ​​at multiple time points), and D1 and D2 are heterogeneous; the present invention studies how to calculate the causal relationship M between variables in the same mode and across modes using observation data from two modes (causal relationship can be represented in the form of a matrix, and if the position (p, q) of the matrix is ​​non-zero, it indicates that there is a directed edge from variable p to variable q).

[0059] (2) A schematic process of causal association inference and cognitive impairment level identification based on feature extraction for heterogeneous multimodal data, as shown below. Figure 2 As shown, it includes:

[0060] K10. Obtain heterogeneous multimodal data {D1, D2}, where D1 represents behavioral data and D2 represents cognitive data, and proceed to steps K20 and K30 respectively;

[0061] K20. Perform causal inferences on the same modality, including:

[0062] K21, let i = 1,

[0063] K22, calculate the variables of the same mode. Causal relationship diagram M between them i Proceed to step K23.

[0064] K23, i = i + 1, go to step K23.

[0065] K24, determine if i > 2 is true; if yes, proceed to step K40; otherwise, proceed to step K22; K30, perform cross-modal causal inference, including:

[0066] K31. Using a trained convolutional autoencoder, brain region features D'2 are calculated based on cognitive data. The data structure of the brain region features is consistent with the data structure of the behavioral data variables, resulting in consistent features {D1, D'2}.

[0067] K32. Based on the consistency feature {D1, D'2}, calculate the causal relationship graph M0 between variables across modalities;

[0068] K40, Fusion of causal relationship graphs of the same modality M i The fused causal relationship graph M is obtained by combining the cross-modal causal relationship graph M0 with the causal relationship graph M0.

[0069] K50. Determine the level of cognitive impairment based on the fused causal relationship diagram M.

[0070] The technical solution of the system according to the embodiments of the present invention will be described in detail below.

[0071] According to one embodiment of the present invention, see Figure 2 A system for identifying levels of cognitive impairment is provided, comprising: a data acquisition module 100, a first inference module 200, a second inference module 300, and a cognitive impairment identification module 400. To better understand this invention, each module will be described in detail below with reference to specific embodiments.

[0072] I. Data Acquisition Module

[0073] According to one embodiment of the present invention, a data acquisition module is used to acquire behavioral data and cognitive data of different modalities collected by a subject during a predetermined cognitive impairment test, wherein the behavioral data and the cognitive data are time-series data each having at least two variables. The behavioral data and cognitive data are described below.

[0074] According to an optional embodiment of the present invention, each behavioral data point includes the accuracy and speed of behavior for multiple specified test items, wherein the accuracy and speed of behavior each correspond to a variable. Preferably, the accuracy of behavior is a vector composed of the accuracy of multiple behaviors for multiple specified test items, and the speed of behavior is a vector composed of the speed of multiple behaviors for multiple specified test items. Illustratively, a testing platform (such as preset testing software) can be provided, allowing subjects to perform a specified test item (such as dragging and dropping small blocks of different colors according to instructions, entering as many animal names as possible, countdown tests and count-up tests, or combinations thereof) at different time periods on the testing platform, and recording the corresponding accuracy and speed of behavior. The technical solution of this embodiment can at least achieve the following beneficial technical effects: The present invention collects data from subjects from both the aspects of behavioral accuracy and speed, which can better reflect the subject's behavioral ability, help to subsequently discover more general correlations between behavioral data and cognitive data, and improve the accuracy of cognitive impairment identification.

[0075] According to one embodiment of the present invention, a behavioral-EEG bimodal dataset collected at Guangxi Jiangbin Hospital can be used as a reference. [4]To obtain behavioral accuracy and behavioral speed. In the original method of reference [4], the behavioral data of the subjects were subdivided into 7 types of features, namely: block-based features, time-based features, path-based features, angle-based features, turning-based features, speed-based features and landing point distribution features. Block-based features include the number of blocks moved successfully, the number of blocks moved unsuccessfully, the number of blocks of blue, green, red and yellow in the blocks moved successfully, and the number of blocks of the four colors in the blocks moved unsuccessfully, for a total of 10 features. The other 6 types of features are also calculated from the four cases of successful movement, unsuccessful movement, successful movement with blue, green, red and yellow, and unsuccessful movement with blue, green, red and yellow (a total of 10 cases). Therefore, only one case of the latter 6 types of features will be described in detail. The other cases are similar and will not be described in detail. Time-based features include average time, median time, maximum time, minimum time, and standard deviation of time for successful movement and nine other scenarios, totaling 50 features; path-based features include average straight-line length, median straight-line length, standard deviation of straight-line length, average curve length, median curve length, and standard deviation of curve length for successful movement and nine other scenarios, totaling 60 features; angle-based features include average angle change at threshold θ1, average angle change at threshold θ2, and average angle change at threshold θ3 for successful movement and nine other scenarios, totaling 30 features; turning-based features include successful movement and its... It has 30 features: average number of turns at threshold β1, average number of turns at threshold β2, and average number of turns at threshold β3 in 9 scenarios; speed-based features include average speed, median speed, and standard deviation of speed in successful movement and the other 9 scenarios, totaling 30 features; and landing point distribution-based features include maximum pairwise distance, minimum pairwise distance, average distance from landing point to center point, median distance from landing point to center point, maximum distance from landing point to center point, minimum distance from landing point to center point, and standard deviation of distance from landing point to center point (center point refers to the center point of all landing points), totaling 70 features. This embodiment of the invention divides these 7 types of features into two main categories: behavioral accuracy and behavioral speed. Behavioral accuracy includes features based on block count, angle, turns, and landing point distribution; behavioral speed includes features based on time, path, and speed. The variables corresponding to behavioral accuracy and behavioral speed are both 140-dimensional. Therefore, the behavioral data of each subject in each task is 140×2 feature data.

[0076] According to an optional embodiment of the present invention, each piece of behavioral data includes multi-axis sensing data collected by a multi-axis inertial sensor, wherein the sensing data for each axis corresponds to a variable. For example, the behavioral data includes multiple sensing data or combinations thereof collected by a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer. The technical solution of this embodiment can achieve at least the following beneficial technical effects: it enables the rapid and convenient collection of multi-axis sensing data as behavioral data through wearable devices during testing, thereby improving the efficiency and accuracy of cognitive impairment level identification.

[0077] According to one embodiment of the present invention, the cognitive data is obtained based on electroencephalogram (EEG) data from multiple brain regions. Preferably, each piece of cognitive data includes average EEG data from multiple brain regions, wherein the average EEG data for each brain region corresponds to a variable. The cognitive data includes average EEG data from six brain regions, namely the brain regions corresponding to the left frontal cortex, right frontal cortex, left parietal cortex, right parietal cortex, left occipital cortex, and right occipital cortex, wherein the average EEG data for each brain region is the average of the EEG data collected by all electrodes located in that brain region. A schematic diagram of the electrode positions of the existing 10-20 international standard lead system is shown below. Figure 3 As shown, Figure 3The orientation markers (not electrodes) indicating head position have the following meanings: NASION indicates the root of the nose (anterior) and INION indicates the external occipital protuberance (posterior); the markings indicating electrode positions have the following meanings: A indicates an ear electrode, F indicates the frontal cortex (or frontal region, frontal lobe), C indicates the central region, T indicates the temporal region (or temporal lobe), P indicates the parietal cortex (or parietal region, parietal lobe), O indicates the occipital cortex (or occipital region, occipital lobe), z indicates the midline region, LC indicates the left cheek electrode, and RC indicates the right cheek electrode; the numbers indicate the electrode number for the corresponding region, with odd numbers on the left and even numbers on the right; the electrodes in the dotted lines are optional. Based on spatial location, the cerebral cortex is mainly divided into the frontal lobe, parietal lobe, temporal lobe, and occipital lobe. Multiple experiments suggest that to improve the recognition of cognitive impairment, the focus should be on six brain regions: the left frontal cortex (LFC), right frontal cortex (RFC), left parietal cortex (LPC), right parietal cortex (RPC), left occipital cortex (LOC), and right occipital cortex (ROC). These six regions involve 19 electrode locations. Specifically, F3, Fp1, and F7 belong to the LFC; F4, Fp2, and F8 belong to the RFC; P3 belongs to the LPC; P4 belongs to the RPC; O1 belongs to the LOC; and O2 belongs to the ROC. For each subject, the average EEG data collected from all electrodes in the same brain region was calculated to obtain the average EEG data for each of the six brain regions. Each subject has 6,000 data points in each brain region; it can be seen that its data structure is different from that of behavioral data. That is, variables between different modalities are heterogeneous data, while different variables within the same modality are isomorphic data. The technical solution of this embodiment can achieve at least the following beneficial technical effects: This embodiment of the invention selects the average EEG data of six brain regions that are more correlated with the identification of cognitive impairment levels as cognitive data, which can effectively improve the accuracy of subsequent cognitive impairment identification.

[0078] It should be understood that, without considering any loss of effectiveness, other brain regions could theoretically be added to create different implementation schemes. For example, adding a brain region corresponding to the left and right temporal lobes; or, reducing some brain regions to create other implementation schemes, such as reducing the brain regions corresponding to the left and right occipital cortex. Furthermore, to improve effectiveness, the average value of EEG data from the aforementioned six brain regions in a 64-lead EEG acquisition system could be used.

[0079] II. First Inference Module

[0080] According to one embodiment of the present invention, a first inference module is configured to perform causal inference based on each variable within each modality to obtain a first causal relationship graph between variables within each modality. Preferably, the causal inference based on each variable within each modality can employ a constraint-based causal inference algorithm (e.g., PC algorithm), a fraction-based causal inference algorithm (e.g., GES algorithm), or a function-based causal model-based causal inference algorithm (e.g., LiNGAM algorithm).

[0081] According to one example of the invention, the first inference module performs as follows: Figure 4 The diagram illustrates the causal inference process for isomorphic data. Data within the same modality, sharing the same structure and distribution, is considered isomorphic and can be directly analyzed using existing causal discovery algorithms. For example, for modality 1 data, a causal discovery algorithm is used to perform causal discovery on variables X1, X2, X3, and X4, resulting in the first causal relationship graph of modality 1, consisting of four causal edges: X1→X2, X1→X3, ​​X2→X4, and X3→X4. Similarly, for modality 2 data, a causal discovery algorithm is used to perform causal discovery on variables Y1, Y2, and Y3, resulting in the first causal relationship graph of modality 2, consisting of three causal edges: Y1→Y2, Y1→Y3, and Y2→Y3. This demonstrates the causal relationships between the variables within each modality.

[0082] III. Second Inference Module

[0083] According to one embodiment of the present invention, a second inference module is used to map the cognitive data into brain region features consistent with the variable data structure of the behavioral data using the encoder in a trained convolutional autoencoder, and to perform cross-modal causal inference based on the variables of the behavioral data and the brain region features to obtain a second causal relationship graph between the variables across modalities. An autoencoder (AE) is a fundamental structure widely used in the field of deep neural networks and is a type of unsupervised learning. It can automatically learn features from unlabeled data and is a neural network aimed at reconstructing input information, providing a better feature description than the original data. An autoencoder network consists of an encoder and a decoder. A multi-layered encoder can transform high-dimensional input data (original data) into low-dimensional codes (features), while the decoder restores the low-dimensional codes to high-dimensional data (reconstructed data). The training process for the encoder and decoder is a process of continuously adjusting the network parameters to minimize the difference between the original data and the reconstructed data. When the network training is complete, the low-dimensional code obtained by the encoder from the input data is the feature of the data. Correspondingly, when training using average EEG data of brain regions, the encoder obtains brain region features. Autoencoder networks have wide applications in many fields, such as image processing, speech recognition, and natural language processing. By training autoencoder networks, features of data can be acquired, and then these features can be used for other downstream tasks. Convolutional Autoencoder Network (CAE) is one such example. [5] It is a type of autoencoder network, whose encoder and decoder are convolutional and deconvolutional layers, respectively. It can preserve the spatiotemporal correlation of data and is more suitable for extracting features from time-series data. The technical solution of this embodiment can achieve at least the following beneficial technical effects: On the one hand, after unifying behavioral data and cognitive data into data with a consistent data structure, the accuracy of causal relationships inferred between heterogeneous data can be improved, which is conducive to improving the accuracy of cognitive impairment identification; on the other hand, as can be seen from the previous embodiments, since the spatial dimension of cognitive data is relatively high, after adjusting it to brain region features with a variable data structure consistent with the behavioral data, not only is the semantic information of behavioral data represented with a lower dimension, but the computational amount of inferring causal relationships between heterogeneous data is also reduced, improving the overall computational efficiency and reducing the total delay in the cognitive impairment identification process.

[0084] According to one embodiment of the present invention, the encoder of the trained convolutional autoencoder is trained in the following manner: acquiring multiple training cognitive data, collected from multiple different personnel, with each personnel collecting one or more training cognitive data; using the average EEG data of each brain region of each training cognitive data as one training data, resulting in multiple training data; using the multiple training data to train a convolutional autoencoder including an encoder and a decoder, resulting in a trained convolutional autoencoder, wherein, during training: the encoder is configured to encode the training data to output brain region features consistent with the data structure of the behavioral data variables, and the decoder is configured to reconstruct the average EEG data of the corresponding brain region based on the brain region features; updating the parameters of the encoder and decoder based on the reconstruction loss determined by the reconstructed average EEG data of the brain region and the input corresponding training data. Due to the differences in distribution, structure, and representation of heterogeneous multimodal data, existing causal discovery algorithms have low accuracy in obtaining causal associations when directly inferring causal relationships from them. Therefore, by extracting brain region features through the encoder of a trained convolutional autoencoder, heterogeneous multimodal data can be mapped to a unified feature space. Based on consistent features, causal discovery algorithms can be used to more accurately and effectively infer causal relationships between features and obtain causal relationships between heterogeneous multimodal data.

[0085] According to an embodiment of the present invention, the structure, training process, and application of a convolutional autoencoder are given below:

[0086] (1) Schematic diagram of the structure of a convolutional autoencoder

[0087] A convolutional autoencoder consists of an encoder (composed of multiple convolutional and pooling layers) and a decoder (composed of multiple deconvolutional layers). High-dimensional cognitive data can be processed through multiple convolutional and pooling layers to obtain low-dimensional brain region features, and then processed through multiple deconvolutional layers to recover the cognitive data. The structures of convolutional and deconvolutional layers are as follows:

[0088] a) Convolutional layer

[0089] Assume the input to the convolutional layer is a C in ×H in ×W in tensor, C in H represents the number of input channels. in and W in These represent the number of rows and columns of the two-dimensional tensor on each input channel, respectively, and the output is a C... out ×H out ×W out The tensor is M×N, and the convolution kernel is an M×N tensor. The output of the convolutional layer is obtained by performing operations on the two-dimensional tensors of different input channels using the convolution kernel, where the j-th (j=1,...,C) tensor is the first tensor. outThe schematic calculation process for each output channel is as follows:

[0090]

[0091] Among them, bias(C outj () represents the bias corresponding to the j-th output channel, and k represents the corresponding input channel. ★ This represents cross-correlation, which performs dot product calculations using a sliding convolution kernel. Different stride values ​​can be set, and you can also choose whether to pad the 2D vector with zeros, typically using zeros.

[0092] After computation by the convolutional layer, the number of rows H of the two-dimensional tensor on each output channel is... out And column number W out The values ​​are as follows:

[0093]

[0094]

[0095] Among them, 2×padding h and 2×padding w These refer to the number of rows and columns expanded vertically and horizontally of the two-dimensional vector (half vertically and half horizontally), respectively. h and Kernel w These are the number of rows and columns of the convolution kernel, respectively, and the stride. h and stride w These represent the number of sliding steps in the vertical and horizontal directions, respectively.

[0096] b) Pooling layer

[0097] Pooling layers reduce the dimensionality of input features, typically using methods such as average pooling and max pooling. For example, a pooling layer with a 2×2 kernel size, when the stride is set to 1, will shrink a 4×4 feature map to 2×2.

[0098] c) Deconvolutional layer

[0099] Assume the input to the deconvolution layer is a matrix with the number of channels × the number of rows × the number of columns, denoted as C. in ×H in ×W in The tensor outputs a channel number × row number × column number of C. out ×H out ×W outThe tensor has an M×N convolution kernel. Unlike convolutional layers, deconvolutional layers perform dimensionality upscaling by complementing the two-dimensional tensor in each input channel both vertically and horizontally, and by inserting new numbers between every two numbers in the two-dimensional tensor (the number of insertions in both the horizontal and vertical directions is stride-1). Then, a sliding convolution kernel performs dot product operations (moving one step at a time), thus achieving data dimensionality upscaling. After the deconvolutional layer's calculation, the number of rows H in the two-dimensional tensor of each output channel increases. out And column number W out The values ​​are as follows:

[0100] H out =(H in -1)×stride h -2×(Kernel h -1-padding h )+Kernel h (8)

[0101] W out =(W in -1)×stride w -2×(Kernel w -1-padding w )+Kemel w (9)

[0102] Therefore, the size of the convolution kernel, the stride of the convolution kernel, the size of the pooling kernel, and / or the stride of the pooling kernel can be set as needed to adjust the data structure of cognitive data to be consistent with that of behavioral data.

[0103] (2) Training diagram of convolutional autoencoder

[0104] The goal of training a convolutional autoencoder is to minimize the difference between the original and reconstructed data by minimizing a loss function (common loss functions include cross-entropy error and mean squared error). After training the convolutional autoencoder, EEG data from heterogeneous multimodal datasets can be processed by convolutional layers to obtain features with consistent dimensions, thereby mapping the heterogeneous multimodal data to a unified feature space.

[0105] (3) Application illustration of convolutional autoencoder

[0106] After extracting features from heterogeneous multimodal data using a convolutional autoencoder, the multimodal data has been mapped to a unified feature space. Based on the variables of brain region features and behavioral data with consistent dimensions, existing causal discovery algorithms are then used to perform causal inference on different variables across modalities.

[0107] See you again Figure 4The data for variables X1, X2, X3, and X4 belong to modality 1, with each variable's data in each sample being 1×N dimensional; the data for variables Y1, Y2, and Y3 belong to modality 2, with each variable's data in each sample being 1×M dimensional. After feature extraction via a convolutional autoencoder, each variable in modality 2 is mapped to 1×N dimensional EEG features. Therefore, existing causal discovery algorithms, such as PC, GES, and LiNGAM, can be used to perform causal discovery on the feature data of the seven variables. Only the causal associations between variables from different modalities are taken to represent the causal associations between heterogeneous data, resulting in three causal association edges: X2→Y1, Y1→X4, and Y3→X4.

[0108] IV. Cognitive Impairment Identification Module

[0109] According to one embodiment of the present invention, a cognitive impairment identification module is configured to determine the cognitive impairment level of a subject based on a first causal association graph for each modality and a second causal association graph across modalities. Preferably, the cognitive impairment identification module is configured to: fuse the first causal association graph for each modality and the second causal association graph across modalities to obtain a fused causal association graph; and determine the cognitive impairment level of the subject based on the fused causal association graph using a trained graph neural network classification model. The trained graph neural network classification model includes a graph neural network and a classifier. The graph neural network is used to extract a graph embedding vector corresponding to the subject based on the fused causal association graph, and the classifier is used to determine the cognitive impairment level of the subject based on the graph embedding vector. The graph neural network can be a conventional Graph Convolutional Network (GCN) or a Graph Attention Network (GAT). Therefore, this embodiment of the invention divides the causal association inference of heterogeneous multimodal data into two main parts: causal inference of homogeneous data and causal inference of heterogeneous data. Finally, by simply superimposing the causal relationships on the second causal association graph onto the first causal association graph, a complete fused causal association graph can be obtained. This not only leverages the strengths of existing causal discovery algorithms in handling causal associations of homogeneous data, but also finds causal associations between heterogeneous multimodal data based on the consistent characteristics of heterogeneous data, improving the algorithm's completeness and thus enhancing the accuracy and efficiency of cognitive impairment identification.

[0110] According to one embodiment of the present invention, the trained graph neural network classification model is trained as follows: A classification training set is acquired, collected from multiple individuals, and includes multiple fused causal association graphs and labels for training. Each fused causal association graph corresponds to a label, which indicates the ground truth value of the cognitive impairment level of the individual to which the fused causal association graph belongs. Using the classification training set, the graph neural network classification model is trained to identify the cognitive impairment level based on the input fused causal association graphs for training. Specifically, a classification loss is determined based on the identified cognitive impairment level and the corresponding label. The gradient is calculated based on the classification loss, and backpropagation is used to update the parameters of the graph neural network and the classifier. The classification loss can be determined using the cross-entropy loss function. Preferably, the cognitive impairment level is divided into levels 0 to N, where level 0 represents no cognitive impairment, and levels 1 to N correspond to different levels of cognitive impairment severity, with higher values ​​indicating more severe cognitive impairment. For example, the cognitive impairment level is divided into levels 0-2, where level 1 represents mild cognitive impairment and level 2 represents severe cognitive impairment. It should be understood that this is only for illustration, and those skilled in the art may also use other more detailed classifications, such as levels 0-5, etc., which would require the preparation of a corresponding classification training set to train the graph neural network classification model.

[0111] According to an embodiment of the present invention, a method for identifying cognitive impairment levels is also provided. The method includes: acquiring behavioral data and cognitive data of different modalities collected by a subject during a predetermined cognitive impairment test, wherein the behavioral data and the cognitive data are time-series data each having at least two variables, the variables between different modalities are heterogeneous data, and the different variables within the same modality are isomorphic data; performing causal inference based on each variable within each modality to obtain a first causal relationship graph between variables within each modality; using an encoder in a convolutional autoencoder to map the behavioral data and the cognitive data into data with a consistent data structure, and performing causal inference based on the data with the consistent data structure to obtain a second causal relationship graph between variables across modalities; and determining the cognitive impairment level of the subject based on the first causal relationship graph of each modality and the second causal relationship graph across modalities. The steps of the method and the functions of the various modules for identifying cognitive impairment levels are related, and their implementation can be referred to the embodiments of the aforementioned system, which will not be repeated here.

[0112] To verify the effectiveness of this invention, the inventors conducted experiments on a real dataset (behavioral-EEG bimodal dataset). On the real dataset, by extracting features from behavioral and EEG data, and then using the GES algorithm to perform causal discovery on these features, they obtained brain effect connectivity maps and causal association maps between brain regions and behavior for healthy, mild, and severe cognitive impairment groups. The causal maps of the three groups showed a certain degree of discriminative power. Comparative experiments were conducted on a publicly available benchmark dataset using constraint-based, fraction-based, and function-based causal model-based causal discovery algorithms. The experiments demonstrated that extracting features from heterogeneous multimodal data and then performing causal discovery on these features is more effective than directly performing causal discovery on heterogeneous multimodal data in finding causal relationships between multimodal groups, improving the recall rate, and verifying the effectiveness of this invention compared to existing technologies.

[0113] The two experiments will be explained below.

[0114] Experiment 1: Medical Application Investigation Experiment

[0115] The method of this invention was used to perform causal discovery on real behavioral-EEG bimodal datasets to explore the application value of this invention in medicine.

[0116] 1) Introduction to the real dataset

[0117] The real dataset used in this experiment was a behavioral-EEG bimodal dataset. The original data consisted of hand movement and brain neural activity data collected from patients with varying degrees of cognitive impairment at Guangxi Jiangbin Hospital. Hand movements were recorded using an electronic box and block test and a touchscreen, while neural activity was recorded using high-density EEG. The hand movement and brain neural activity data were recorded simultaneously. After initial screening, a total of 48 participants were selected, divided into a healthy group (18 participants), a mild cognitive impairment group (13 participants), and a severe cognitive impairment group (17 participants). The participant information form detailed each participant's name, age, gender, data collection date, left- or right-handedness, and group assignment.

[0118] During data collection, each participant completed five tasks, including two single tasks and three dual tasks. The first single task was a simple block drag-and-drop task. In the starting area, there were 25 blocks of each of four colors. Participants had to drag as many blocks as possible from the starting area to the target area within a specified time. The color could be chosen freely for each drag. If a participant hit a barrier or placed a block in the wrong area, the drag was considered unsuccessful; otherwise, it was considered successful. The second single task was a fixed-point drag-and-drop task. Participants had to accurately drag and drop as many blocks as possible to a designated location in the target area within a specified time. Compared to the first single task, the second task was more restrictive and challenging. Simultaneously with completing the first single task, participants performed three language-related tasks: counting backwards from 100, listing as many animal names as possible, and subtracting 7 from 100 sequentially, resulting in the three dual tasks. While each subject completed the five tasks, the object number and coordinates of the object being held were recorded every 17 milliseconds to obtain the subject's behavioral data. Simultaneously, based on the international 10-20 system, EEG signals from 19 electrode locations were recorded every 3.3 milliseconds to obtain the subject's EEG data. The 19 electrode locations measured are... Figure 3 The parameters shown are P3, C3, F3, Fz, F4, C4, P4, Cz, A1, Fp1, Fp2, T3, T5, O1, O2, F7, F8, A2, and T6. Behavioral and EEG data for each subject were measured five times for each task.

[0119] 2) Data Feature Extraction

[0120] 2.1) Extraction of behavioral data:

[0121] According to the literature [8] The method extracted seven types of features from the behavioral data: block-based features, time-based features, path-based features, angle-based features, turn-based features, speed-based features, and landing point distribution features. These seven types of features were further divided into two main categories: behavioral accuracy and behavioral speed, both of which are 140-dimensional. Thus, the behavioral data of each subject in each task was transformed into 140×2 feature data after feature extraction. Implementation details have been mentioned earlier and will not be repeated here.

[0122] 2.2) Feature extraction from EEG data

[0123] Because the EEG data is sampled very frequently, with 6000 data points per brain region per subject, the data dimensionality is high, and the total training sample size is 1440, there is no need to perform windowing or GAF transformation to obtain more samples. A convolutional autoencoder was used to extract features from the EEG data: the input data was in 1×1×6000 format. First, a 1×64 convolutional kernel transformed the data into a 16×1×2969 format. Then, a max pooling layer further reduced the dimensionality to 16×1×990, and then a 1×31 convolutional kernel transformed it into a 1×1×322 format. Finally, a linear layer further reduced the dimensionality to 140 dimensions. The decoder section is symmetrical to the encoder. First, a linear layer upscales the features into a 322-dimensional vector. Then, the vector is rewritten into a 1×1×322 format. Next, three deconvolutional layers follow: first, a 1×31 convolutional kernel transforms the data into a 16×1×990 format; then, a 1×4 convolutional kernel further upscales it to 16×1×2969; finally, a 1×64 convolutional kernel restores the data to a 1×1×6000 format. Thus, the average EEG data from each subject's six brain regions in each task is transformed into 140×6 features after feature extraction, effectively converting the average EEG data from each brain region into 140-dimensional brain region features, consistent with the dimension of each variable in the behavioral data. Merging the behavioral data and EEG features yields 140×8 bimodal feature data (i.e., consistent features) for each subject in each task. It should be understood that the specific encoder and decoder structures described here are for illustrative purposes only; implementers can adjust some parameters to achieve a similar effect with a convolutional autoencoder.

[0124] 3) Experimental Results

[0125] After extracting consistent features, the GES algorithm was used to perform causal discovery on these features. Since the participants were divided into three groups, the causal relationship graphs of each group were integrated to obtain causal relationship graphs for each of the three groups across the five tasks. The results for single task 1 are illustrated below. Figure 5 , Figure 6 , Figure 7 As shown, schematic diagrams of causal relationships between the healthy group, the mild cognitive impairment group, and the severe cognitive impairment group in the task are presented respectively. These include: a brain effect connectivity diagram (the first causal relationship diagram corresponding to cognitive data) and a second causal relationship diagram between behavioral accuracy, behavioral speed, and six brain regions.

[0126] Since the dataset in Experiment 1 did not actually have labels for the causal relationship graph, it was not easy to intuitively demonstrate the improvement in the accuracy of the inferred causal relationships. Therefore, Experiment 2 was also conducted.

[0127] Experiment 2: Experiments based on public benchmark datasets

[0128] 1) Experimental Evaluation

[0129] To indirectly verify the effectiveness of the causal correlation technical solution proposed in the embodiments of the present invention, experimental verification was conducted on a publicly available benchmark dataset, and quantitative evaluation indicators were calculated.

[0130] 2) Publicly available benchmark datasets and preprocessing

[0131] 2.1) Introduction to Public Benchmark Datasets

[0132] Since there are currently no publicly available heterogeneous multimodal datasets that provide true causal relationship graphs between multimodal data, a homogeneous dataset that provides true causal graphs was chosen. Then, while preserving the causal relationships, the homogeneous dataset was transformed into a heterogeneous dataset. The homogeneous dataset used was generated by Smith et al. [6] The dataset contains functional magnetic resonance imaging (fMRI) data. This dataset is a simulation dataset with a standard network, generated based on a dynamic causal model. It contains 28 simulations, each with 50 objects, with slight differences between the simulation datasets. Taking the first simulation dataset as an example, it contains 50 objects, each with 5 nodes. Each node records BOLD signal data at 200 time points, with a sampling frequency of 20Hz and a sampling duration of 10 minutes. Below, we will use the first simulation dataset to generate a causally correlated heterogeneous dataset.

[0133] 2.2) Generation of Heterogeneous Datasets

[0134] An exponential transformation g(x) = e was performed on each data point of the 5th node of the first simulated dataset. x (Including all objects), after the transformation, the data distribution of the 5th node changes, thus becoming heterogeneous (distribution heterogeneous) with the data of the previous 4 nodes. From equation (1), it can be seen that the original causal function model corresponding to the 5 nodes of each object in the first dataset is shown in equation (10). After performing an exponential transformation on the data of the 5th node, the corresponding causal function model is shown in equation (11), where X... i (i = 1, 2, ..., 5) represents 5 nodes, f i (i = 1, 2, ..., 5) represents the corresponding causal mechanism, ε i (i = 1, 2, ..., 5) are noise variables. Let g(x) represent the new variable obtained after exponential transformation of variable X5. As can be seen from equations (10) and (11), the causal relationship between the five nodes has not changed, thus obtaining the heterogeneous multimodal dataset with known causal relationship graph.

[0135]

[0136]

[0137] 2.3) Data Windowing

[0138] Since the first dataset only contained data from 50 objects, which was too small a sample size, a windowing operation was performed on the data in this experiment to increase the sample size for training the convolutional autoencoder. For each object, 50 data points were taken from each node as a segment, with 40 data points overlapping between adjacent segments. Thus, each node of an object could obtain 16 segments containing data from 50 time points. The entire data windowing process is as follows: Figure 8 As shown, after windowing, a total of 4000 50-dimensional training samples were obtained.

[0139] 2.4) Processing based on Gram's angle field

[0140] Since each dataset currently only has 50 dimensions, which is not conducive to dimensionality reduction and feature extraction, the data underwent further transformation. This was achieved using a Gramian Angular Field (GAF). [7] This is a method for converting time-series data into image data, specifically described as follows: Given a time-series data X = {x1, ..., x...} n}, perform the following transformation on X:

[0141] ① By scaling and translating X according to equation (12), we obtain Make each data point It can fall within the interval [-1, 1];

[0142]

[0143] ② Represented using polar coordinates according to equation (13) because Therefore each Corresponding to a unique φ i ;

[0144]

[0145] ③ Calculate the GAF matrix according to equation (14);

[0146]

[0147] After the above three steps, time series data can be transformed into image data. For example... Figure 9As shown, for each 50-dimensional data, by performing a scaling and translation transformation (as in equation (12)) and a Gram angle field transformation (as in equations (13) and (14)), 4000 50×50-dimensional image data can be obtained, thereby increasing the data dimension while retaining the original information, which is beneficial for feature extraction.

[0148] 3) Experimental setup

[0149] 3.1) Network Architecture Setup

[0150] After determining the shape of the input data, the architecture of the convolutional autoencoder in Experiment 2 was set up, referring to... Figure 10 The structure shown is as follows. Where:

[0151] The input data is a 1×50×50 image. First, a 4×4 convolutional kernel transforms the image data into 6×24×24 data. Then, a max-pooling layer further reduces the dimensionality to 6×13×13 data, followed by a 3×3 convolutional kernel to 1×13×13. Next, it is stretched into a 169-dimensional vector, and then a linear layer reduces the dimensionality to 100-dimensional features. The decoder is symmetrical to the encoder. First, a linear layer enlarges the features into a 169-dimensional vector, then rewrites the vector into a 1×13×13 format. Next, three deconvolutional layers: first, a 3×3 convolutional kernel transforms the data into 6×13×13 format; then, a 4×4 convolutional kernel further enlarges the data to 6×24×24; finally, a 4×4 convolutional kernel restores the data to a 1×50×50 image format, thus successfully recovering the image from the features.

[0152] 3.2) Experimental Design

[0153] Based on the proposed causal association learning method, we first use a causal discovery algorithm to directly process the causal associations between homogeneous data. Then, we use a convolutional autoencoder to extract features from heterogeneous data and perform causal discovery on these features. However, it's important to note that for each object, we can obtain a causal graph between 16 features from heterogeneous data. Therefore, we consider ensemble processing: we count the frequency of directed edges between heterogeneous data found at different times. If the frequency of a directed edge is higher than a set value α, we consider that there is a correlation between heterogeneous multimodal data. Finally, we combine the causal associations between heterogeneous and homogeneous data to obtain a complete causal graph for each object.

[0154] The PC algorithm, GES algorithm, and LiNGAM algorithm were used to perform causal discovery on heterogeneous data according to the above schemes. Then, the obtained causal graphs were compared with the real causal graphs to evaluate the effectiveness of the experimental scheme. The evaluation indicators were accuracy, recall, F1 score, and Structural Hamming distance (SHD). Accuracy refers to the proportion of correctly retrieved directed edges in the actual retrieved directed edges (as shown in equation (15)). Recall refers to the proportion of correctly retrieved directed edges in the expected retrieved directed edges (as shown in equation (16)). The F1 score is calculated as shown in equation (17). The SHD score is the sum of the number of redundant directed edges and the number of missed directed edges. Therefore, the higher the accuracy, recall, and F1 score, and the lower the SHD score, the closer the found causal graph is to the real causal graph, and the better the effect of the experimental scheme. In addition, we used these three algorithms to directly perform causal discovery on heterogeneous data and compared the results with those obtained using our method to explore whether using convolutional autoencoders to extract features would improve the experimental results.

[0155]

[0156]

[0157]

[0158] 3.3) Parameter Settings

[0159] Table 1 shows the parameter settings during the training of the convolutional autoencoder. During training, the mean squared error function (MSE) was chosen and optimized using the Adam algorithm. When ensembling the causal graph between features of each object, a frequency threshold α = 0.5 was used.

[0160] Table 1 CAE Training Parameter Settings

[0161]

[0162] 3.4) Experimental Results

[0163] Experiment 2 was conducted on data from 50 objects, and the results are presented in Table 2 in the form of mean ± standard deviation. The bolded data indicates that the mean or standard deviation obtained using the method of this invention is better than that obtained by directly using the causal discovery algorithm.

[0164] Table 2. Experimental results for 50 subjects

[0165]

[0166] Among these, the higher the values ​​of precision, recall, and F1 scores, the better, while the lower the value of SHD scores, the better.

[0167] 4) Experimental Analysis and Conclusions

[0168] Based on the experimental results of 50 objects, after using convolutional autoencoders to extract features, the recall rate of all three algorithms is higher than that of directly performing causal discovery on heterogeneous data. Furthermore, the accuracy, F1 score, and SHD score of the LiNGAM algorithm are all better than those obtained by directly performing causal discovery.

[0169] The above experiments and analyses demonstrate that, compared to directly using traditional causal discovery algorithms to find causal relationships between heterogeneous multimodal data, using convolutional autoencoders to extract features from heterogeneous data and then performing causal discovery on those features is more effective in finding causal relationships between multimodal data and increases the recall rate. Among the three algorithms, the GES algorithm, when used for causal discovery of features from heterogeneous data, yielded relatively good and stable values ​​for all four evaluation metrics, and showed a certain improvement compared to directly using the GES algorithm for causal discovery.

[0170] For details of some prior art, please refer to the following references. For brevity, their full technical solutions can be cited as supplementary information to the corresponding parts of this application:

[0171] [1]P.Spirtes,CNGlymour,R.Scheines,and D.Heckerman.Causeation,prediction,and search[M].MIT press,2000.

[0172] [2]DMChickering.Optimal Structure Identification with Greedy Search[J].Journal of Machine Learning Research,2002,3(Nov):507-554.

[0173] [3]S.Shimizu,POHoyer,et al.A Linear Non-Gaussian Acyclic Model forCausal Discovery[J].Journal of Machine Learning Research,2006,7(10).

[0174] [4]Y.Zhang,et al.What can“drag&drop”tell?Detecting mild cognitiveimpairment by hand motor function assessment under dual-task paradigm[J].International Journal of Human-Computer Studies,2021,145:102547.

[0175] [5]J.Masci,et al.Stacked Convolutional Auto-encoders for HierarchicalFeature Extraction[C] / / Artificial Neural Networks and Machine Learning–ICANN2011:21st International Conference on Artificial Neural Networks,Espoo,Finland,June 14-17,2011,Proceedings,Part I 21.Springer Berlin Heidelberg,2011:52-59.

[0176] [6]S.M.Smith,et al.Network Modelling Methods for FMRI[J].Neuroimage,2011,54(2):875-891.

[0177] [7]Z.Wang,T.Oates.Encoding Time Series as Images for VisualInspection and Classification Using Tiled Convolutional Neural Networks[C] / / Workshops at the twenty-ninth AAAI conference on artificialintelligence.Menlo Park,CA,USA:AAAI,2015,1.

[0178] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order, as long as the required function can be achieved.

[0179] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0180] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0181] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A system for identifying levels of cognitive impairment, characterized in that, include: The data acquisition module is used to acquire behavioral data and cognitive data of different modalities collected by the subject during a predetermined cognitive impairment test. The behavioral data and the cognitive data are time-series data with at least two variables. The variables between different modalities are heterogeneous data, while the different variables within the same modality are isogeneous data. The cognitive data is obtained based on EEG data from multiple brain regions. The first inference module is used to perform causal inference based on each variable in each modality, and obtain the first causal relationship graph between variables in each modality; The second inference module is used to map the cognitive data to brain region features consistent with the variable data structure of the behavioral data using the encoder in the trained convolutional autoencoder, and to perform cross-modal causal inference based on the variables of the behavioral data and the brain region features to obtain a second causal relationship graph between variables across modalities. A cognitive impairment identification module is used to determine the cognitive impairment level of the subject based on a first causal relationship graph for each modality and a second causal relationship graph across modalities.

2. The system according to claim 1, characterized in that, Each behavioral data point includes the accuracy and speed of behavior for multiple specified test items, with each behavioral accuracy and speed corresponding to a variable. Each cognitive data point also includes the average EEG data for multiple brain regions, with the average EEG data for one brain region corresponding to a variable. or Each behavioral data point includes multi-axis sensing data collected by a multi-axis inertial sensor, where the sensing data for each axis corresponds to a variable, and each cognitive data point includes average EEG data for multiple brain regions, where the average EEG data for each brain region corresponds to a variable.

3. The system according to claim 2, characterized in that, The encoder of the trained convolutional autoencoder is trained in the following manner: Multiple sets of cognitive data for training are acquired, collected from multiple different individuals, with each individual collecting one or more sets of cognitive data for training. The average EEG data of each brain region in each training cognitive data is used as one training data to obtain multiple training data; Using multiple training data sets, a convolutional autoencoder including an encoder and a decoder is trained to obtain a trained convolutional autoencoder. During training: the encoder is configured to encode the training data to output brain region features with a data structure consistent with the variables of the behavioral data; the decoder is configured to reconstruct the average EEG data of the corresponding brain region based on the brain region features; the parameters of the encoder and decoder are updated based on the reconstruction loss determined by the reconstructed average EEG data of the brain region and the corresponding input training data.

4. The system according to claim 2 or 3, characterized in that, The cognitive data includes average EEG data from six brain regions: the left frontal cortex, right frontal cortex, left parietal cortex, right parietal cortex, left occipital cortex, and right occipital cortex. The average EEG data for each brain region is the average of the EEG data collected by all electrodes in that brain region.

5. The system according to claim 4, characterized in that, The cognitive impairment recognition module is configured as follows: The first causal relationship graph for each modality and the second causal relationship graph across modalities are fused to obtain a fused causal relationship graph; The cognitive impairment level of the subject is determined by using a trained graph neural network classification model based on a fused causal relationship graph. The trained graph neural network classification model includes a graph neural network and a classifier. The graph neural network is used to extract the graph embedding vector corresponding to the subject based on the fused causal relationship graph, and the classifier is used to determine the cognitive impairment level of the subject based on the graph embedding vector.

6. The system according to claim 5, characterized in that, The trained graph neural network classification model was trained in the following manner: A classification training set is obtained, which is collected from multiple individuals and includes multiple training fusion causal relationship graphs and labels. Each training fusion causal relationship graph corresponds to a label, and the label indicates the true value of the cognitive impairment level of the individual to which the training fusion causal relationship graph belongs. Using the classification training set, the graph neural network classification model is trained to identify the level of cognitive impairment based on the fused causal relationship graph used for training. The classification loss is determined based on the identified level of cognitive impairment and the corresponding label. The gradient is calculated based on the classification loss and backpropagated to update the parameters of the graph neural network and the classifier.

7. The system according to any one of claims 1-3, characterized in that, The cognitive impairment level is divided into levels 0 to N, where level 0 indicates no cognitive impairment and levels 1 to N correspond to the degree of cognitive impairment. The higher the value, the more severe the cognitive impairment.

8. A method for identifying levels of cognitive impairment, characterized in that, The method includes: The method involves acquiring behavioral and cognitive data from different modalities collected during a predetermined cognitive impairment test of a subject. The behavioral and cognitive data are time-series data with at least two variables, respectively. The variables between different modalities are heterogeneous data, while the variables within the same modality are isomorphic data. Within each modality, causal inferences are made based on each variable to obtain the first causal relationship graph between variables within each modality; The encoder in the convolutional autoencoder is used to map the behavioral data and the cognitive data into data with a consistent data structure, and causal inference is performed based on the data with a consistent data structure to obtain a second causal relationship graph between variables across modalities; The cognitive impairment level of the subject is determined based on the first causal relationship diagram for each modality and the second causal relationship diagram across modalities.

9. The method according to claim 8, characterized in that, Each piece of cognitive data includes average EEG data from multiple brain regions, with the average EEG data for one brain region corresponding to one variable; and each piece of behavioral data includes behavioral accuracy and behavioral speed for multiple specified test items, with behavioral accuracy and behavioral speed each corresponding to one variable. or Each piece of cognitive data includes average EEG data from multiple brain regions, with each brain region's average EEG data corresponding to a variable; each piece of behavioral data includes multi-axis sensing data collected based on multi-axis inertial sensors, with each axis's sensing data corresponding to a variable.

10. The method according to claim 9, characterized in that, The encoder of the convolutional autoencoder is a trained convolutional autoencoder, which is trained in the following manner: Multiple sets of cognitive data for training are acquired, collected from multiple different individuals, with each individual collecting one or more sets of cognitive data for training. The average EEG data of each brain region in each training cognitive data is used as one training data to obtain multiple training data; Using multiple training data sets, a convolutional autoencoder including an encoder and a decoder is trained to obtain a trained convolutional autoencoder. During training: the encoder is configured to output brain region features with a data structure consistent with the variables of the behavioral data, and the decoder is configured to reconstruct the average EEG data of the corresponding brain region based on the brain region features; the parameters of the encoder and decoder are updated based on the reconstruction loss determined by the reconstructed average EEG data of the brain region and the corresponding training data input.

11. The method according to claim 9 or 10, characterized in that, The cognitive data includes average EEG data from six brain regions: the left frontal cortex, right frontal cortex, left parietal cortex, right parietal cortex, left occipital cortex, and right occipital cortex. The average EEG data for each brain region is the average of the EEG data collected by all electrodes in that brain region.

12. The method according to claim 11, characterized in that, The level of cognitive impairment of the subjects was determined as follows: The first causal relationship graph for each modality and the second causal relationship graph across modalities are fused to obtain a fused causal relationship graph; The cognitive impairment level of the subject is determined by using a trained graph neural network classification model based on a fused causal relationship graph. The trained graph neural network classification model includes a graph neural network and a classifier. The graph neural network is used to extract the graph embedding vector corresponding to the subject based on the fused causal relationship graph, and the classifier is used to determine the cognitive impairment level of the subject based on the graph embedding vector.

13. The method according to claim 12, characterized in that, The trained graph neural network classification model is trained in the following manner: A classification training set is obtained, which is collected from multiple individuals and includes multiple training fusion causal relationship graphs and labels. Each training fusion causal relationship graph corresponds to a label, and the label indicates the true value of the cognitive impairment level of the individual to which the training fusion causal relationship graph belongs. Using the classification training set, the graph neural network classification model is trained to identify the level of cognitive impairment based on the fused causal relationship graph used for training. The classification loss is determined based on the identified level of cognitive impairment and the corresponding label. The gradient is calculated based on the classification loss and backpropagated to update the parameters of the graph neural network and the classifier.

14. The method according to any one of claims 8-10, characterized in that, The cognitive impairment level is divided into levels 0 to N, where level 0 indicates no cognitive impairment and levels 1 to N correspond to the degree of cognitive impairment. The higher the value, the more severe the cognitive impairment.

15. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 8 to 13.

16. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 8 to 13 by executing the executable instructions.

Images