Short-time-sequence Alzheimer disease progress prediction method based on multi-relation mixed perception

Abstract

The invention discloses a short time sequence Alzheimer disease progress prediction method based on multi-relation mixed perception, and belongs to the field of progress prediction, and the method comprises the steps: providing a time sequence evolution generative adversarial network, enhancing the dynamic fitting capability of data evolution, and improving the generalization of a long time sequence learning model to a short time sequence. Secondly, a multi-relation mixed sensing module embedded with global noise filtering is designed, complex relations among subjects can be captured, and pairwise and high-order interaction relations can be efficiently learned. Specifically, the module realizes joint modeling of a pairwise relationship and a high-order relationship by constructing a noise filtering population graph and a population hypergraph. Then, the mixed representation of the multiple relationships is fed to an extended long-short term memory module for time series learning.

Description

Technical Field

[0001] The present invention belongs to the field of progression prediction, and in particular relates to a short-time series Alzheimer's disease progression prediction method based on multi-relationship mixed perception. Background Art

[0002] Alzheimer's disease (AD) is a common neurodegenerative disease with long-term and irreversible characteristics. It mainly affects the elderly population aged 65 and above, causing significant damage to patients' cognitive abilities and daily living abilities, and may eventually lead to death. In the development of Alzheimer's disease, mild cognitive impairment (MCI) is a critical early stage and is divided into stable MCI (sMCI) and progressive MCI (pMCI). Without timely intervention, MCI may progress to AD, placing a heavy economic and social burden on patients and their families. Therefore, early identification and diagnosis of Alzheimer's disease is crucial to improving patient prognosis and alleviating pressure on the medical system.

[0003] Existing technologies for processing short time series data on Alzheimer's disease suffer from the following deficiencies: first, it is difficult to effectively handle missing data; second, existing time series models are not adaptable to short time series data; and third, they fail to fully exploit the complex relationships between subjects, especially high-order relationships. Therefore, there is an urgent need for an Alzheimer's disease progression prediction method that can effectively process short time series data and fully exploit the complex relationships between subjects. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a short-time series Alzheimer's disease progression prediction method based on multi-relationship hybrid perception, comprising:

[0005] The multimodal data of the subjects were preprocessed to obtain time series data including MRI features, demographic features, and clinical score features;

[0006] Interpolating the time series data to obtain complete time series data;

[0007] A short time series prediction model based on multi-relationship hybrid perception is constructed based on the multi-relationship hybrid perception module and the time series learning module;

[0008] The complete time series data is input into the short time series prediction model for calculation to obtain the prediction results of the subject's clinical status and clinical score.

[0009] Preferably, the process of obtaining complete time series data includes:

[0010] Noise is introduced and input into the imputation module together with the longitudinal MRI data with missing values ​​and the mask;

[0011] The interpolation module learns the distribution of the observed time series data, estimates the missing data, and replaces the missing MRI values ​​with the corresponding estimated values ​​through masking to obtain the complete time series data after interpolation.

[0012] Preferably, the process of obtaining the predicted results of the subject's clinical status and clinical score comprises:

[0013] The complete time series data is input into the short time series prediction model for calculation, and a multi-relationship hybrid perception module embedded with global spatiotemporal noise filtering is used to learn the pairwise relationships and high-order relationships between subjects to obtain a multi-relationship hybrid feature representation;

[0014] The multi-relationship mixed feature representation is input into a time series learning module to perform time series learning to obtain prediction results of the subject's clinical status and clinical score.

[0015] Preferably, the process of obtaining the multi-relation mixed feature representation includes:

[0016] The pairwise relationships are constructed with subjects as vertices, and the generation of relationships is guided by demographic characteristics. The global spatiotemporal noise filtering module is used to filter out invalid correlations between subjects caused by noise, and the pairwise relationship perception matrix is ​​obtained.

[0017] Construct a hypergraph with subjects as vertices, learn high-order relationships between subjects through node-to-edge convolution and edge-to-node convolution, and obtain a high-order relationship perception matrix;

[0018] The pairwise relation perception matrix and the high-order relation perception matrix are fused to obtain a multi-relation hybrid feature representation.

[0019] Preferably, the time series learning module adopts an xLSTM module, which is composed of sLSTM and mLSTM modules through residual stacking, and performs time series learning through the following steps:

[0020] Calculate the input features at each time point, the hidden state at the previous moment, and the unit state at the previous moment to obtain the hidden state and unit state at the current moment;

[0021] The obtained hidden state and unit state are residually stacked to obtain the final time series feature representation.

[0022] Preferably, the time series learning module further includes a generative adversarial network for enhancing the data fitting capability of the time series learning module, and the specific steps include:

[0023] The generator generates pseudo sample data;

[0024] The pseudo sample data generated by the generator and the real sample data are input into the discriminator, and the performance of the generator and discriminator are optimized through adversarial training.

[0025] Preferably, the discriminator is trained by the following steps:

[0026] Construct a pairwise relationship network between subjects;

[0027] Use graph convolutional networks to aggregate information on the features of subject nodes;

[0028] The aggregated features are input into the fully connected layer to obtain the discrimination results.

[0029] Preferably, the process of preprocessing the multimodal data of the subject includes:

[0030] Perform head correction and segmentation on MRI data to obtain ROI feature representation;

[0031] Demographic information and clinical score data were standardized.

[0032] On the other hand, the present invention further provides an electronic device, comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the method is implemented when the processor executes the computing program.

[0033] On the other hand, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the method when executed by a processor.

[0034] Compared with the prior art, the present invention has the following advantages and technical effects:

[0035] (1) To address the difficulty of fitting time series models with short time series data, this paper designs a time series evolution generative adversarial model. While efficiently interpolating missing data to ensure data volume, the idea of ​​generative adversarial network game makes the learned time series data more closely fit the real distribution.

[0036] (2) To address the inability to adequately represent complex relationships between subjects, the present invention designs a multi-relationship hybrid perception module embedded with global noise filtering. The global spatiotemporal noise filtering module filters out the global spatiotemporal irrelevance between subjects and embeds it into the multi-relationship hybrid perception module, effectively perceiving pairwise and many-to-many relationships in longitudinal data.

[0037] (3) We conducted experiments on 1,230 subjects collected from the ADNI dataset to verify the effectiveness of the proposed method and its submodules. The experimental results show that MMPSTS outperforms previous methods in classification and prediction performance in most cases. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0039] Figure 1 Schematic diagram of a method for dividing target time points according to an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of a short time series prediction model framework based on multi-relationship hybrid perception according to an embodiment of the present invention;

[0041] Figure 3 This is a diagram of a multi-relationship hybrid perception module according to an embodiment of the present invention;

[0042] Figure 4 A global spatiotemporal noise filtering diagram according to an embodiment of the present invention;

[0043] Figure 5 This is a diagram of the sLSTM module of an embodiment of the present invention;

[0044] Figure 6 This is a diagram of the mLSTM module of an embodiment of the present invention;

[0045] Figure 7 AUC result comparison chart of the proposed method and the comparative method in the embodiment of the present invention;

[0046] Figure 8 Schematic diagram of the locations of the top 10 most discriminative ROIs in the AD vs NC classification task according to an embodiment of the present invention;

[0047] Figure 9 Schematic diagram of the locations of the top 10 most discriminative ROIs in the pMCI vs sMCI classification task according to an embodiment of the present invention;

[0048] Figure 10 This is a schematic diagram of the results of verifying the effectiveness of the multi-relationship hybrid perception module according to an embodiment of the present invention;

[0049] Figure 11 This is a schematic diagram of the comparison results of time series learning variant classification according to an embodiment of the present invention;

[0050] Figure 12 This is a schematic diagram of classification comparison results of variants of the multi-relation hybrid perception module according to an embodiment of the present invention;

[0051] Figure 13 Schematic diagram of comparison results of classification between multi-task joint training and single-task separate training according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0053] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0054] Example 1

[0055] This embodiment provides a short-term Alzheimer's disease progression prediction method based on multi-relationship hybrid perception, including:

[0056] The data used in this example are from the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset. This example screened 1,256 subjects with time-series data from this dataset, including 333 subjects with NC, 204 subjects with AD, 300 subjects with progressive mild cognitive impairment (pMCI), and 419 subjects with stable mild cognitive impairment (sMCI). pMCI and sMCI were combined into MCI, resulting in three categories: NC, AD, and MCI.

[0057] The data studied in this example included magnetic resonance imaging (MRI), demographic information, and clinical scores. All images were T1-weighted. Demographic information included gender, age, weight, APOE A1 and APOE E2, and other information. Clinical scores included the FAQ, GDS, CDR, MMSE, and NPI-Q. Demographic information for all subjects is shown in Table 1.

[0058] Table 1

[0059]

[0060] The time point of the subject's first examination is taken as the "baseline", and the subsequent examination times are named based on the baseline. For example, M06 means that the subject was examined again 6 months after the first examination. At subsequent time points, there are missing data in the MRI images and clinical score data. This embodiment uses a 6-month time interval for model learning. The MRI, demographic information and clinical score data of the number of subjects fluctuating over time are shown in Table 2. It can be seen that due to various reasons, the number of subjects is gradually decreasing over time.

[0061] Table 2

[0062]

[0063] According to the fluctuation of the subject data over time, this embodiment defines the first five time points with more frequent records as "historical access time points", and the predicted time point is called the "target time point". Figure 1 As shown, this embodiment partitions the data. For participants whose time series are longer than five time points, this embodiment does not perform special processing. For participants whose time series are shorter than five time points, this embodiment uses the second-to-last available time point as the last historical time point data, and the last time point as the target time point for prediction. This embodiment excludes 26 participants whose target time points lack scoring data. Ultimately, this embodiment collects longitudinal data for 1,230 participants. In model design, this embodiment uses historical time points as input to the generator, including baseline (BL), 6-month (M06), 12-month (M12), 18-month (M18), and 24-month (M24) time points. The model's prediction targets include 30-month (M30), 36-month (M36), 42-month (M42), 48-month (M48), 54-month (M54), 60-month (M60), 66-month (M66), and 72-month (M72) time points.

[0064] This example uses a similar MRI data processing method as Liu et al., primarily for head correction and segmentation. The sMRI data preprocessing process was completed using the CAT12 toolkit in Matlab. This example segmented the preprocessed sMRI images using the Anatomical Automatic Labeling (AAL3) atlas to generate 170 ROI feature representations.

[0065] The input of the model is T historical visit records of N subjects. The time series data is divided into three modalities, MRI features, demographic features and clinical score features are represented as X (1) 、X (2) and X (3) , the data of the qth mode is expressed as d q represents the feature dimension of the qth mode, and q=1, 2, 3. Represents the characteristic matrix of the qth modality at time t, which includes N subjects. In this embodiment, represents the eigenvector of the qth mode of subject i at time t.

[0066] Because MRI features are missing at some time points, this embodiment defines a missing mask matrix To encode the missing data status of the subjects at each time point, 0 means missing and 1 means not missing.

[0067] For incomplete and variable-length short time series data, this embodiment proposes the MMPSTS model. Figure 2 As shown, this embodiment divides the model into two parts: generator and discriminator.

[0068] The generator consists of three modules: interpolation of missing data, multi-relationship hybrid perception of subjects, and time series learning. The interpolation module uses the generative network portion of a traditional GAN ​​network to interpolate longitudinal MRI data containing missing values. Demographic features are added to the multi-relationship extraction module to guide the generation of pairwise relationships, and global spatiotemporal noise filtering is used to filter out invalid associations between subjects caused by the noise of the interpolation module. The demographic features and the completed MRI data are then input into the multi-relationship hybrid perception module for feature learning. Finally, the features output by the multi-relationship hybrid perception module are concatenated with the scoring data as input to the time series learning module.

[0069] To address the problem of missing longitudinal MRI data of subjects, this embodiment introduces noise Z into the generator and adds it to the interpolation module for training. The interpolation module learns the distribution of the observed time series data and then estimates the missing data. First, input MRI data X with missing values (1) , mask M and noise Z, and finally output the complete time series after interpolation The calculation of the interpolation module can be described as:

[0070]

[0071] in Using the provided network, the function of estimating missing values ​​is calculated by learning the distribution of the observed time series data. Its output is is the missing value of the function estimate, ⊙ represents the Hadamard product. The missing values ​​of MRI are masked by M. The corresponding estimated values ​​are replaced by represents the complete time data after interpolation. Therefore, for the data interpolation part, the relevant loss can be expressed as:

[0072]

[0073] Among them, this embodiment hopes The generated data is as similar as possible to the real data, so the mean square error MSE is used as the loss of the interpolation module. represents the mean squared error between the imputed data and the true MRI features.

[0074] In traditional graph learning methods, this embodiment usually only considers paired relationships between research objects, with subjects as vertices, but in real life, there are often more complex many-to-many relationships between research subjects. In real life, when making clinical diagnoses, doctors often compare multiple subjects with similar age, test images, and clinical manifestations to obtain better clinical diagnosis results. Therefore, for a single time, this embodiment constructs paired relationships and high-order relationships with subjects as vertices, and extracts potential features through a multi-relationship hybrid perception module. The multi-relationship hybrid perception module is as follows: Figure 3 As shown. represents the interpolated complete MRI data at time t, represents the demographic characteristics at time t, represents the output of the pairwise relationship module, Represents the output of the high-order relation module, It represents the output of the multi-relation hybrid perception module after layer normalization. Next, the pairwise relationship and many-to-many relationship parts are introduced separately.

[0075] In the feature learning part of the paired relationship between subjects, this embodiment is different from the previous feature fusion. Instead, it uses the demographic feature to guide the relationship generation method to generate paired relationships between subjects. The paired relationship is generated by G p =(V p ,A p ) indicates that represents the vertex set consisting of subjects, is a weighted adjacency matrix, where a ij ∈A p represents the similarity between subject i and subject j. The pairwise relationship between subjects is obtained by interpolating the complete MRI features and demographic characteristics X (2) Build, the build process is as follows:

[0076]

[0077] Where δ represents the Kronecker delta function, d2 represents the demographic characteristics of the subjects, and θ k represents the similarity threshold of the kth demographic feature. represents the similarity value of the demographic characteristics of subject i and subject j. Then, this embodiment uses the demographic similarity as a weight to perform a weighted combination with the interpolated MRI features:

[0078]

[0079] Where ρ represents the calculation of the Pearson correlation coefficient. Based on the MRI characteristics and demographic characteristics of the subject, this embodiment can calculate the subject's preliminary pairwise relationship perception matrix A p .

[0080] Generally speaking, longitudinal data has temporal characteristics, and since a large amount of noise is introduced in the interpolation module, it has a great impact on the interconnection of subject nodes. To solve the above problem, inspired by [1], this embodiment uses a spatiotemporal attention mechanism for global noise filtering to filter out invalid correlations between subjects, such as Figure 4 As shown. First, use the completed MRI features of T time periods To calculate the spatiotemporal correlation matrix, as follows:

[0081]

[0082] in are learnable parameters, is the complete time data after interpolation, is the spatiotemporal correlation matrix.

[0083] Next, this embodiment embeds the noise filtering component into the spatiotemporal correlation matrix as shown below:

[0084]

[0085] in is the normalized spatiotemporal correlation matrix, represents the temporal correlation between nodes i and j, Represents the average value of the original spatiotemporal correlation matrix, reflecting the strength of the global temporal and spatial correlation. ε∈(0,1] is the filter factor. The larger the value of ε, the closer the noise-filtered spatiotemporal attention is to the global correlation, and the greater the correlation between subjects. If ε is smaller, the filtering strength of the noise-filtered temporal attention is stronger, and the correlation between subjects is smaller. Here, a combination of filter factors and global correlation is used to filter out invalid correlations between subjects. Figure 4 As shown, where A p is the pairwise relationship perception matrix, is the global spatiotemporal noise filtering matrix, is the pairwise relationship perception matrix after noise filtering.

[0086] Next, this embodiment uses a global spatiotemporal noise filtering module to filter out invalid correlations between some subjects due to noise:

[0087]

[0088] Where ⊙ represents the Hadamard product. After filtering out the invalid global correlation caused by noise, this embodiment uses K-order Chebyshev polynomials to perform convolution operations on the pairwise relationships between subjects:

[0089]

[0090] in Represents the updated node features after pairwise convolution operation, yes The normalized Laplacian matrix, is the incidence matrix The degree matrix of is a learnable weight parameter, It is calculated The i-th order Chebyshev polynomial of .

[0091] Based on the paired relationship of the subjects, this embodiment constructs a hypergraph with the subjects as vertices. In the process of constructing the hypergraph, this embodiment measures the similarity between the two subjects by the Euclidean distance between the subjects. Most previous studies have used the KNN method to construct the hypergraph. Specifically, with each subject as the central node, the K subjects closest to the central node are connected by a hyperedge. However, this method fixes the size of the hyperedge and cannot adaptively match the distribution of data. Based on this, this embodiment designs an objective function to construct the hypergraph. As with the assumptions explained, if the distance between subject i and subject j is small, the similarity S ij It should be large, but if subject i and subject j belong to different categories, the distance between them should be large and the similarity is approximately 0. Therefore, this embodiment defines the similarity of subjects as:

[0092]

[0093] where s ij represents the similarity between subject i and subject j, and accordingly, It is represented as the similarity matrix between subjects. 2,1 The norm is defined as This embodiment can convert the above problem (10) into a linear constrained quadratic programming and solve it through the KTT condition, thereby obtaining an adaptive hypergraph incident matrix.

[0094] Finally, this embodiment constructs N hyperedges for N subjects and obtains the subject's hypergraph incident matrix Different from the previous hypergraphs that use 0 and 1 to represent the adjacency relationship between vertices and edges, Indicates that vertex i is connected to hyperedge e j Compared with the previous hypergraph learning based on binary transition probability, for a given vertex, the hyperedge constructed in this embodiment explicitly assigns different importance to the incoming and outgoing information.

[0095] In this embodiment, convolution is performed on high-order relationships between subjects. The convolution layer of the hypergraph is divided into two parts: "node-to-edge convolution" and "edge-to-node convolution." First, let's discuss node-to-edge convolution. In this step, feature information associated with vertices on a hyperedge is propagated along the hyperedge through message propagation. From a disease perspective, information about subjects with high similarity is aggregated along the hyperedge.

[0096]

[0097] Among them, E represents the N hyperedge information obtained from the vertex information of the previous layer through convolution.

[0098] After obtaining the features of N hyperedges, the next step is to aggregate the features of the hyperedges associated with the vertex to update the node features. In this step, the node learns the information of all hyperedges associated with it, thereby updating the node's features. Through these two steps, the subject can learn the information of subjects with similar performance characteristics to itself. This step mainly allows the subject to learn the information of subjects who are very similar to itself.

[0099]

[0100] in After the hypergraph features are aggregated, the updated high-order node feature information is obtained. Finally, this embodiment fuses the updated features of the two perception branches to obtain the output of the multi-relation hybrid perception module:

[0101]

[0102] In order to speed up the training, the layer normalization layer LayerNorm module is executed.

[0103] After the multi-relational hybrid perception feature extraction module, this example uses xLSTM to capture the relationship between disease progression and time. xLSTM is composed of sLSTM and mLSTM modules stacked with residuals. The following describes these two modules and how they are combined and separated.

[0104] The input of sLSTM is x t 、h t-1 、C t-1 , respectively representing the input features at the current time t, the hidden state at the previous time, and the cell state at the previous time. sLSTM introduces exponentials, normalization, and stabilization based on LSTM, enabling LSTM to modify storage decisions. Figure 5 The sLSTM architecture is shown. The forward pass update formula of sLSTM is:

[0105] c t =f t c t-1 +i t z t cell state(14)

[0106] n t =f t n t-1 +i t normalizer state(15)

[0107]

[0108] where w z 、w i 、w f 、w o and r z 、r i 、r f 、r o and b z 、b i 、b f 、b o are the weight and bias of the sLSTM unit respectively. tanh(·), σ(·), exp(·) represent the tangent function, activation function and exponential function respectively. t Represents the normalized state at time point t. In order to provide numerical stability for the exponential gate, the forget gate and the input gate are merged into another state m t As shown below:

[0109] m t =max(log(f t )+m t-1 ,log(i t))stabilizer state(21)

[0110]

[0111] f' t =exp(log(f t )+m t-1 -m t )stabil.forget gate(23)

[0112] The matrix long short-term memory (mLSTM) model introduces matrix memory units and a covariance update mechanism for key-value pair storage, significantly improving the model's memory capacity. Figure 6 The architecture of mLSTM is shown. The forward pass formula of mLSTM is:

[0113]

[0114] n t =f t n t-1 +i t k t normalizer state(25)

[0115]

[0116] q t =W q x t +b q query input(27)

[0117]

[0118] v t =W v x t +b v value input(29)

[0119]

[0120] Where W q 、W k 、W v Calculate the learnable weights associated with self-attention, w i 、w f 、w o and b i 、b f 、b o are the learnable weights and biases respectively. C t is the matrix memory, q t、k t 、v t Similar to the Transformer architecture, they are query, key, value, i t 、f t 、o t They are input gate, forget gate and output gate respectively. sLSTM and mLSTM are constructed by stacking residual blocks as shown below to construct the xLSTM architecture:

[0121] h t =mLSTM(sLSTM(X t ,h t-1 )+X t )+X t (33)

[0122] For time series data, each time point is processed with xLSTM with the same parameters.

[0123] Next, this example will describe in detail how to use the xLSTM module to process time series data. For the T historical accesses input, after the above interpolation and multi-relation hybrid perception feature extraction module, the hidden feature representation of the subject has been obtained. in represents the output feature of the Mix-aware module at time point t. For each visit t, this embodiment combines the feature with the clinical cognitive score vector Put together, it is expressed as And input it as input data to the xLSTM module for feature learning. Therefore, the data fitting loss of the encoder at the first T time points is:

[0124]

[0125] in is the feature representation of subject i at time t after learning by the xLSTM module.

[0126] After learning the historical time series, this embodiment performs rating prediction and classification prediction for several subsequent time points. The decoder corresponding to the prediction module is also the above module. For the output of the prediction module at each time point, this embodiment obtains the corresponding predicted rating data. For the F time points that need to be predicted, the rating output of the prediction module can be expressed as Therefore, the loss for rating prediction is:

[0127]

[0128] Where B is similar to the mask M, indicating whether there is scoring data at a time point, B i,t =1 means that the i-th subject has scoring data at time point t, otherwise Bi,t =0.

[0129] The classification prediction and classification loss related formulas are as follows:

[0130]

[0131] in and Y represent the predicted and true subject categories, respectively, using cross entropy loss Calculating classification loss

[0132] The discriminator is used to distinguish the pseudo samples generated by the generator at time T from the real samples at time T. Specifically, the output of the last historical time point T-1 of the encoder part of the generator module is As a pseudo sample at time T, and the data S at the real historical time point T T The input is fed into the discriminator, which distinguishes real samples from fake ones. The discriminator's computational process consists of two main steps: graph convolution and linear combination. First, similar to Section 3.1.2, demographic features are combined with MRI data to construct a pairwise relationship network between subjects. In the discriminator, this example uses two GCN layers to aggregate information on the features of the subject nodes:

[0133]

[0134] Where Θ1 and Θ2 represent learnable parameters, Represents the input of the convolution, here is and S T . S D It means that after two graph convolutions, the updated feature matrix is ​​obtained.

[0135] Finally, the updated matrix S D Input to the fully connected layer:

[0136]

[0137] in Y l ,b l They represent the weight matrix, the updated features and bias after graph convolution, and L is the number of fully connected layers.

[0138] In the model of this embodiment, the generator and the discriminator are composed of two sets of parameters, and the loss function of the model is defined as follows.

[0139] First, the pseudo sample generated by the generator is input into the discriminator to obtain the result Y G, where the value of each element is a probability value between [0,1]. Accordingly, in this embodiment, the result of the real sample data calculated by the discriminator is recorded as Y D In this embodiment, the label lbl of the real result D And pseudo sample result label lbl G All elements of are recorded as 1 and 0 respectively, and the cross entropy of the two result vectors and the label vector is calculated respectively. Naturally, if the discriminator can correctly distinguish between real samples and fake samples, then the cross entropy between the result vector and the corresponding label should be small. Therefore, this embodiment defines the loss of the discriminator as for:

[0140]

[0141] In order to further estimate the performance of the generator, this embodiment calculates Y G and lbl D If the cross entropy between the two vectors is small, it means that the discriminator mistakenly judges the data generated by the generator as correct, and the generated pseudo samples are closer to the real samples, which is exactly what this embodiment expects. Therefore, this embodiment uses the above cross entropy value to obtain the partial loss of the generator

[0142]

[0143] In order to better train the model of this embodiment, this embodiment combines all the generator modules for training, and simultaneously performs score prediction and classification tasks, so that the model can learn the feature representation of each task. Combining the generator loss, interpolation loss, data fitting loss, score prediction loss and classification loss, the total loss of the final generator is:

[0144]

[0145] In experiments, this example predicts progression by predicting clinical scores and final clinical status at up to eight target time points for a given subject. To validate the performance of the proposed method, this example compared it with several advanced methods. Furthermore, this example conducted ablation experiments to verify the effectiveness of the modules in this example's method. The following sections describe the evaluation metrics, comparison models, and comparison results.

[0146] Clinical diagnosis is a classification problem, and model performance is evaluated using seven metrics. The specific calculation formulas for these metrics are shown below, where TP represents true positive, TN represents true negative, FP represents false positive, and FN represents false negative.

[0147]

[0148]

[0149] where a i , b i Represent the sum of the row elements and column elements of the confusion matrix respectively.

[0150] In the experiment of this embodiment, the mean absolute error MAE and the weighted correlation coefficient wR are used to measure the prediction performance. The two metrics are defined in formulas (51-53), where j represents five scoring data, b j,t represents the number of subjects corresponding to score j observed at time point t. is the calculation of the correlation coefficient. The average value of the true value and the predicted value is expressed as and Represent respectively.

[0151]

[0152] The dataset is split into 8:2 for training and testing respectively. The subjects with fixed batch size are used for training.

[0153] This example compares the model of this example with the currently most advanced methods:

[0154] LSTM-P: Impute missing data with the number 0 and use LSTM units with Peephole to predict cognitive scores and MRI biomarkers.

[0155] LSTM-T:,The missing values ​​of time series are filled by model training.,The model considers the intervals between time points and,the interactions between time points, thus enhancing the prediction,performance.

[0156] MinimalRNN: Clinical status and scores were predicted and MRI biomarkers were extracted by using the MinimalRNN method.

[0157] MIA: A multimodal sequence learning framework capable of handling incomplete variable-length data and proposing collaborative training for AD progression modeling.

[0158] Informer: Based on the Transformer architecture and using the ProbSparse self-attention mechanism, it effectively processes long time series. This ensures model performance while significantly reducing training time.

[0159] TPA-LSTM: An attention calculation method specifically proposed for time series, focusing on capturing the dependencies between time points.

[0160] SLA-EDLSTM: A sequence-length adaptive encoder-decoder long short-term memory deep learning model is proposed, which achieves high accuracy in predicting AD disease progression in individuals over the next 6 years.

[0161] WHGCN:,constructs hypergraphs for different time points and considers the impact of time on the disease,weighted combination of hypergraphs, and performs classification tasks using the many-to-many relationships between subjects in the complete time series.

[0162] LSTM-TSGAIN: Adopts the idea of ​​reducing "accumulated error", generates interpolation data through adversarial learning, and uses collaborative training to train multiple tasks.

[0163] In this example, in order to better compare the performance of the model, since the WHGCN model is only for classification tasks, the complete MRI data of this example is interpolated as the input of the model, and the results of the model classification and feature dimensionality reduction are spliced ​​with the demographic characteristics and score data and input into the xLSTM model to carry out the score prediction task. As the objective function, in this example, the hyperparameter filter factor ε is set to 0.6, and the number of fully connected layers L in the discriminator is 3. All deep learning methods use the Adam optimizer to train the model.

[0164] For the score prediction task, Table 3 shows the MAE and wR of all methods. From the results, it can be seen that the method proposed in this embodiment is superior to other comparison methods in all indicators. It is worth mentioning that although the SLA-ED LSTM method uses data enhancement to increase the diversity of data, the method of this embodiment is still better than the model, proving that this method still captures the long-range information of the data even when the time series length is short. For the clinical classification task, Table 4 shows the Accuracy, Percision, Recall, F1, MCC, Kappa, and CSI corresponding to all methods. The results show that the prediction and classification performance of the method of this embodiment are better than other comparison methods. In addition, this embodiment further compares the classification performance of each method by calculating AUC. The comparison results are as follows. Figure 7 As shown. It can be seen from the observation that the classification performance of the method of this embodiment is significantly better than all the comparison methods. Among them, the performance of the WHGCN model for disease classification tasks using hypergraphs in the three-classification tasks of AD, CN, and MCI is much lower than that of this method, which also reveals the effectiveness of the multi-relation hybrid perception module and time series learning module of this method. The results of the Informer method are relatively low in all indicators. This may be because the Informer method is aimed at long time series modeling and is not as effective for shorter sequence modeling.

[0165] Table 3

[0166]

[0167] Table 4

[0168]

[0169] Extracting discriminative pathogenic biomarkers has great biological significance for AD. In order to evaluate the impact of different brain regions on clinical classification and diagnosis tasks, this example masks one brain region at a time, performs a binary classification task, and calculates the average accuracy a i , the importance score of brain region i is 1-a i This process was repeated for all 170 ROIs, resulting in an importance score for each ROI. Finally, the importance scores were ranked in descending order. Based on the AAL3 atlas, the 10 ROIs with the highest scores were identified as important brain regions that significantly contributed to the classification task.

[0170] The 10 ROI regions that contribute most to the AD and NC classification tasks are as follows: Figure 8 As shown. The amygdala (AMYG) is associated with emotion, and damage to this area can cause cognitive impairments induced by specific emotions. The middle temporal gyrus (MTG) is a region associated with cognitive abilities such as face recognition and reading comprehension. In patients with psychiatric disorders, increased leftward asymmetry in the inferior temporal gyrus (ITG) is observed. This region is also involved in cognitive processes such as social cognition, semantic processing, and concept retrieval. The temporal pole (TPOmid) is associated with multiple higher-level cognitive processes, such as face recognition and social emotion processing, and is particularly evident in patients with neurodegenerative diseases. The hippocampus (HIP) is a key structure for memory formation, interacting with the amygdala to regulate emotion and stress. The superior frontal gyrus (SFG) is believed to contribute to higher-level cognitive functions such as working memory. Damage to the anterior cingulate gyrus (ACC) is often reported to be strongly associated with schizophrenia, obsessive-compulsive disorder, and affective disorders. The Precentralgyrus (PreCG) is located in the frontal lobe of the cerebral cortex, where the motor center is located. The physical symptoms such as limb weakness and fatigue shown by patients with mental disorders are related to this area.

[0171] The 10 regions that contributed most to the pMCI and sMCI classification tasks are as follows: Figure 9As shown. In addition to the Amygdala, Superiorfrontalgyrus and Hippocampus introduced above, this embodiment also extracts another 6 discriminative brain regions. The Olfactorycortex (OLF) is a brain region involved in processing olfaction. In some studies, it has been found that it has similar functions to the hippocampus, involving spatial navigation and memory ability, and the Amygdala involved in emotional perception and social interaction is part of it. The SuperiorTemporalGyrus (TPOsup) is located around the Hippocampus and is one of the earliest brain regions to show pathology and neuronal loss in the development of AD. During the patient's cognitive decline, the neuronal cytoskeletal proteins in the Parahippocampalgyrus (PHG) will change, so this region is one of the key factors leading to memory loss and cognitive impairment. The SupraMarginalgyrus (SMG) is the main brain region of the cholinergic pathway. Cholinergic neuroregulation of the distal cortical motor center plays a key role in coordinating motor learning. The development of mental illness will change the brain network pattern in this region, resulting in reduced functional connectivity with other regions. Researchers have found that IFG parsorbitalis (IFGorb) is significantly associated with subjective cognitive decline (SCD), making it an early marker in the progression of AD. Pathological studies of AD have shown that Putamen (PUT) is crucial for feedback acquisition and has significant diagnostic value in the transition from MCI to AD.

[0172] pMCI is a patient with progressive mild cognitive impairment, while sMCI is a patient with stable mild cognitive impairment. Therefore, in order to observe the significant changes in the connection of ROIs between pMCI and sMCI over time, this embodiment splices the values ​​of the ROIs of the four types of subjects in BL and the predicted first period M30, and calculates the Pearson correlation coefficient of 170 ROIs of the four groups of subjects in the two periods. As the disease progresses, the ROI connections of pMCI patients are significantly reduced and weakened (the chords become fewer and thinner), which indicates that the connections of many ROIs are destroyed during the development of AD. However, the brain area connections of sMCI do not change significantly in terms of strength and density. Such experimental results are consistent with the pathological research of AD development, proving the effectiveness of the method of this embodiment in diagnosing AD disease progression.

[0173] In the generator of MMPSTS, multi-relationship mixed perception is proposed to learn the complex relationship between subjects. In order to verify the effectiveness of this module, the following Figure 10 For each time point, the mean value of MRI features of the subjects in each category group was calculated. i, calculate the Pearson correlation (PC) between the feature and the average feature. The above calculations are performed on the output of the multi-relationship hybrid perception module and the original feature representation. It can be observed from the figure that even if the subjects are in the same category, there is still a certain fluctuation between the original MRI features and the category average features, and the correlation is not particularly strong. However, after the multi-relationship hybrid perception module performs complex relationship learning, the characteristic curves of the subjects in the same category are relatively smooth, indicating that the module of this embodiment has a certain filtering effect on noise. And the learned features have a strong correlation, which proves the effectiveness of the relationship perception module designed in this embodiment for feature classification. As can be seen from the figure, there are some differences between the original data of the AD category and the original data of other categories in the first two periods (BL, M06). This is because AD patients have data missing in the later period due to reasons such as death. However, since the method of this embodiment aligns the historical time and the predicted time during data preprocessing, there is more data missing for AD patients in the first two periods.

[0174] This example conducts some ablation experiments to verify the effectiveness of various parts of the method, namely the short time series learning module, pairwise relationship perception module, high-order relationship perception module, multi-relationship hybrid perception module, and joint training of prediction and classification tasks.

[0175] This embodiment aims to solve the problem of short AD time series data length by designing a GAN-based enhanced time series learning module. Based on the model framework and several time series learning models, this embodiment designs several variants as shown below. LSTM, GRU, and RNN respectively represent replacing the time series learning module in Section 3.1.3 with the corresponding model, while the suffix wo-GAN in the table means removing the GAN framework and not adding pseudo samples and real samples to the discriminator for learning. The loss function of the model is changed to Directly study the time series. The results are shown in Table 5 and Figure 11 As shown in the figure, the clinical prediction and classification diagnosis results of this embodiment show that the xLSTM model used in this method outperforms the other three methods in most indicators, demonstrating the effectiveness of methods such as adding exponential gating to the xLSTM. Furthermore, it can be seen that after removing the framework for enhancing short-time series learning performance set in this embodiment, the performance of various indicators of the four methods decreases, indicating that the framework of this embodiment has significantly contributed to the improvement of short-time series learning performance.

[0176] Table 5

[0177]

[0178] In order to study the effectiveness of the multi-relation hybrid perception part, this embodiment designs several variants of MMPSTS, as shown below:

[0179] MMPSTS-wo-pair: This embodiment deletes the pairwise relationship perception module and only retains the high-order relationship perception module.

[0180] MMPSTS-wo-high: The high-order relation perception module is removed and only the pairwise relation module is retained.

[0181] MMPSTS-wo-mix: Delete all mixed perception modules and add the interpolated features directly to the time series learning module for learning.

[0182] Table 6 and Figure 12 The prediction and clinical classification performance of different variants of the method of this embodiment are shown. In this embodiment, it can be observed that compared with the significant performance degradation caused by deleting all the multi-relationship hybrid perception modules, the paired relationship perception and high-order relationship perception have brought certain performance improvements in both prediction and classification tasks, proving that there are one-to-one or many-to-many relationships between the subjects. Compared with the paired relationship perception module, the performance degradation caused by deleting the high-order relationship module is greater, indicating that the many-to-many relationship of the subjects is more common. However, adding the paired relationship module will still bring performance improvement, indicating that the multi-relationship hybrid perception module in the method of this embodiment learns the complex relationship between the subjects more fully and efficiently, and obtains better performance.

[0183] Table 6

[0184]

[0185]

[0186] In order to improve the effectiveness of joint training, this embodiment denotes the score prediction task and the diagnosis classification task as task1 and task2 respectively. The two tasks are trained and tested separately, and the results are shown in Table 7 and Figure 13 From this example, it can be seen that the best effect is achieved by training the two tasks together.

[0187] Table 7

[0188]

[0189] This embodiment proposes MMPSTS, which effectively models multiple relationships between multiple subjects and predicts short time series data based on the idea of ​​generating adversarial games. The results demonstrate the good performance of MMPSTS. However, the method of this embodiment has certain limitations. First, this embodiment aligns the MRI time series data with an interval of 6 months, which may ignore the irregular and asynchronous information of scattered time series data. Secondly, although this embodiment experiments on three types of data for each subject, only MRI is used as the image modality data, which may ignore the impact of modal data such as genes. In response to the first problem, in future work, this embodiment considers using a generalized discrete-time autoregressive (AR) model, integrated into the time series learning framework, to model sporadic time series data. In response to the second problem, this embodiment considers constructing the subject's brain network based on the subject's SNP or fMRI data, and improving the prediction performance from the perspective of multiple relationships and multiple views.

[0190] In this example, we address the short length of AD time series by proposing a short time series prediction model based on multi-relationship hybrid perception for clinical diagnosis and progression prediction of Alzheimer's disease. We first interpolate missing data, then design a multi-relationship hybrid perception module embedded with global spatiotemporal noise filtering to learn the various relationships of the subjects. Finally, we utilize a time series learning module composed of xLSTMs to perform clinical diagnosis and prediction on the subjects. To enable the time series learning module to better fit short time series, we design a time series evolution generative adversarial network. Pseudo-samples generated by the generator and real samples are fed into a designed discriminator. Using game theory, we enhance the data fitting capability of the time series learning module. Experiments conducted on the ADNI dataset validate the effectiveness and superiority of the proposed model for short time series learning.

[0191] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A short-time series Alzheimer's disease progression prediction method based on multi-relationship hybrid perception, characterized by: include: The multimodal data of the subjects were preprocessed to obtain time series data including MRI features, demographic features, and clinical score features; Interpolating the time series data to obtain complete time series data; A short time series prediction model based on multi-relationship hybrid perception is constructed based on the multi-relationship hybrid perception module and the time series learning module; The complete time series data is input into the short time series prediction model for calculation to obtain the prediction results of the subject's clinical status and clinical score.

2. The method according to claim 1, characterized in that The process of obtaining complete time series data includes: Noise is introduced and input into the imputation module together with the longitudinal MRI data with missing values ​​and the mask; The interpolation module learns the distribution of the observed time series data, estimates the missing data, and replaces the missing MRI values ​​with the corresponding estimated values ​​through masking to obtain the complete time series data after interpolation.

3. The method according to claim 1, characterized in that The process of obtaining the predicted results of the subject's clinical status and clinical score includes: The complete time series data is input into the short time series prediction model for calculation, and a multi-relationship hybrid perception module embedded with global spatiotemporal noise filtering is used to learn the pairwise relationships and high-order relationships between subjects to obtain a multi-relationship hybrid feature representation; The multi-relationship mixed feature representation is input into a time series learning module to perform time series learning to obtain prediction results of the subject's clinical status and clinical score.

4. The method according to claim 3, characterized in that The process of obtaining the multi-relation hybrid feature representation includes: The pairwise relationships are constructed with subjects as vertices, and the generation of relationships is guided by demographic characteristics. The global spatiotemporal noise filtering module is used to filter out invalid correlations between subjects caused by noise, and the pairwise relationship perception matrix is ​​obtained. Construct a hypergraph with subjects as vertices, learn high-order relationships between subjects through node-to-edge convolution and edge-to-node convolution, and obtain a high-order relationship perception matrix; The pairwise relation perception matrix and the high-order relation perception matrix are fused to obtain a multi-relation hybrid feature representation.

5. The method according to claim 3, characterized in that The time series learning module adopts the xLSTM module, which is composed of sLSTM and mLSTM modules through residual stacking, and performs time series learning through the following steps: Calculate the input features at each time point, the hidden state at the previous moment, and the unit state at the previous moment to obtain the hidden state and unit state at the current moment; The obtained hidden state and unit state are residually stacked to obtain the final time series feature representation.

6. The method according to claim 1, characterized in that The time series learning module also includes a generative adversarial network for enhancing the data fitting capability of the time series learning module. The specific steps include: The generator generates pseudo sample data; The pseudo sample data generated by the generator and the real sample data are input into the discriminator, and the performance of the generator and discriminator are optimized through adversarial training.

7. The method according to claim 6, characterized in that The discriminator is trained by the following steps: Construct a pairwise relationship network between subjects; Use graph convolutional networks to aggregate information on the features of subject nodes; The aggregated features are input into the fully connected layer to obtain the discrimination results.

8. The method according to claim 1, characterized in that The process of preprocessing the multimodal data of the subject includes: Perform head correction and segmentation on MRI data to obtain ROI feature representation; Demographic information and clinical score data were standardized.

9. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein: When the processor executes the computing program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images