System and method for detecting a disease state using MRI data

Abstract

A method comprising is described herein comprising receiving first magnetic resonance imaging (MRI) data of a first plurality of subjects, wherein the first MRI image data comprises a first plurality of two-dimensional (2D) images, pre-processing the first MRI data for analysis, converting each two-dimensional image of the first plurality of 2D images into first tokens as input for a transformer encoder, wherein the transformer encoder comprises a time series classification transformer, training the transformer encoder using the first input tokens, receiving second MRI data of a subject, wherein the second MRI data comprises a second plurality of 2D images, converting each image of the second plurality of 2D images into second tokens as input to the trained transformer encoder, and applying the trained transformer encoder to the second tokens to predict a state of disease in a subject.

Description

[0001] Docket No.2932719-000238-WO1 UNITED STATES PATENT APPLICATION FOR: SYSTEM AND METHOD FOR DETECTING A DISEASE STATE USING MRI DATA RELATED APPLICATIONS This application claims the benefit to US Application No.63 / 533,197, filed August 17, 2023. This application claims the benefit to US Application No.63 / 599,876, filed November 16, 2023. TECHNICAL FIELD The disclosure herein involves use of machine learning models to detect a disease state using subject MRI data. BACKGROUND Alzheimer’s disease (AD) is distinguished by the accumulation of aberrant protein deposits in the brain, known as plaques and tangles, which result in the death of nerve cells and the degeneration of brain tissue. Neural degeneration reduces cognitive function and causes mood and behavior changes. AD is typically categorized in three stages. The first stage is the preclinical stage, characterized by brain, blood, and cerebrospinal fluid (CSF) abnormalities without outward signs. It is believed that AD pathology begins at least 20 years before symptoms appear. The second stage of the disease is referred to as mild cognitive impairment (MCI), which involves cognitive impairment confined to a single cognitive domain, usually memory. Dementia, the final stage of the disease, is defined as a cognitive disturbance in more than one domain, often memory and executive function, with substantial interference with daily life activities. With the recent approval of new drugs for early AD intervention, early detection of AD and differentiation of MCI have become of primary importance for successful disease treatment and management. INCORPORATION BY REFERENCE Each patent, patent application, and / or publication mentioned in this specification is herein incorporated by reference in its entirety to the same extent as if each individual patent, patent application, and / or publication was specifically and individually indicated to be incorporated by reference. Docket No.2932719-000238-WO1 SUMMARY OF THE INVENTION In embodiments, a method is described herein comprising receiving first magnetic resonance imaging (MRI) data of a first plurality of subjects, wherein the first MRI image data comprises a first plurality of two-dimensional (2D) images, pre-processing the first MRI data for analysis, converting each two-dimensional image of the first plurality of 2D images into first tokens as input for a transformer encoder, wherein the transformer encoder comprises a time series classification transformer, training the transformer encoder using the first input tokens, receiving second MRI data of a subject, wherein the second MRI data comprises a second plurality of 2D images, converting each image of the second plurality of 2D images into second tokens as input to the trained transformer encoder, and applying the trained transformer encoder to the second tokens to predict a state of disease in a subject. In embodiments, the converting the first and second plurality of 2D images comprises converting each image into 224x224 pixel images. In embodiments, each 224x224 pixel image is divided into 14x14 patches, wherein each patch comprises a 16x16 pixel image. In embodiments, the converting the first and second plurality of 2D images comprises linearly projecting each patch into a vector. In embodiments, the converting comprises adding positional embeddings to the first and second tokens. In embodiments, the preprocessing comprises standardizing the first and second MRI data in Montreal Neurological Institute space. In embodiments, the preprocessing comprises stripping skull image data from the first and second MRI data using statistical parametric mapping 12. In embodiments, the first and second plurality of 2D images lie in the coronal plane. In embodiments, the first and second plurality of 2D images lie in the sagittal plane. In embodiments, the first and second plurality of 2D images lie in the axial plane. In embodiments, the transformer encoder comprises a multi-head attention layer. In embodiments, the transformer encoder comprises a feed forward neural network layer. In embodiments, output of the multi-head attention layer is fed into the feed forward neural network layer. Docket No.2932719-000238-WO1 In embodiments, the trained transformer encoder comprises a multilayer perceptron head for predicting the state of disease in the subject. In embodiments, the state of disease comprises a state of Alzheimer’s disease. In embodiments, a method is described herein comprising receiving first magnetic resonance imaging (MRI) data of a first plurality of subjects, wherein the first MRI image data comprises a first plurality of two-dimensional (2D) images, wherein each image corresponds to a time in a series, pre-processing the first MRI data for analysis.converting the two dimensional image data and corresponding time of each image of the first plurality of 2D images into first tokens as input for a transformer encoder, wherein the transformer encoder comprises a time series classification transformer, training the transformer encoder using the first input tokens, receiving second MRI data of a subject, wherein the second MRI data comprises a second plurality of 2D images, wherein each image corresponds to a time in a series, converting the two dimensional image data and corresponding time of each image of the second plurality of 2D images into second tokens as input to the trained transformer encoder, and applying the trained transformer encoder to the second tokens to predict a state of disease in a subject. In embodiments, the converting the first and second plurality of 2D images comprises linearly projecting the two dimensional image data and corresponding time of each image into a vector. In embodiments, the converting comprises adding positional embeddings to the first and second tokens. In embodiments, the preprocessing comprises standardizing the first and second MRI data in Montreal Neurological Institute space. In embodiments, the preprocessing comprises stripping skull image data from the first and second MRI data using statistical parametric mapping 12. In embodiments, the first and second plurality of 2D images comprise a series of parallel images in the coronal plane. In embodiments, the first and second plurality of 2D images comprise a series of parallel images in the sagittal plane. In embodiments, the first and second plurality of 2D images comprise a series of parallel images in the axial plane. In embodiments, the transformer encoder comprises a multi-head attention layer. In embodiments, the transformer encoder comprises a feed forward neural network layer. Docket No.2932719-000238-WO1 In embodiments, output of the multi-head attention layer is fed into the feed forward neural network layer. In embodiments, the trained transformer encoder comprises a multilayer perceptron head for predicting the state of disease in the subject. In embodiments, the state of disease comprises a state of Alzheimer’s disease. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 shows the proposed pipeline of the ViT-TST. MRI images were pre-processed using CAT12 (image registration to standardize the images and skull stripping to reduce biases by ensuring consistent voxel intensities). ViT was used from each plane to derive slice attributes. Finally, the time series transformer was used to classify the feature sequences. Figure 2 shows a sample of an MRI slice in three planes (coronal, sagittal, and axial). A, original MRI; B, segmented anatomical image with the skull removed; and C, segmented anatomical scan warped to MNI space. Figure 3 shows 14 ൈ 14 image patches. Patches of 16 ൈ 16 pixels were taken from the input images, with 14 ൈ 14 patches. Figure 4 shows a visual summary of the proposed method. ViTs consist of several transformer blocks. Each transformer block comprises two sub-layers, a feed-forward layer, and a multi-head self-attention layer. Figure 5 shows a summary of the proposed model. Features were extracted using ViT and the sequence of features was classified using the time series transformer model. Figure 6 shows confusion matrices for binary classification on ADNI1: Complete 3Yr 3T. Figure 7 shows confusion matrices of axial planes for multiclass disease patient classification (NC, MCI, and AD) on ADNI1: Complete 3Yr 3T. Figure 8 shows a general overview of transformers. Each encoder has two main sublayers, a multi-head attention layer and a fully connected FFN. Figure 9 shows layers of self-attention in the Encoder, showing residual skip connections around both layers and two Layer Norm layers. Figure 10 shows confusion matrices for binary classification on ADNI1: Complete 1Yr 1.5T. Figure 11 shows confusion matrices of sagittal planes for multiclass classification on ADNI1: Complete 1Yr 1.5T. Figure 12 shows a sample of MRI slices from subjects with NC, MCI, and dementia. Docket No.2932719-000238-WO1 Figure 13 shows tubelet embedding. Figures 14A and 14B shows a pipeline proposed by ViTranZheimer. Figure 15 presents the confusion matrices depicting the outcomes of a multiclass disease classification analysis. Figure 16 shows a pure-transformer architecture for video classification. Figure 17 shows uniform frame sampling. Figure 18 shows tubelet embedding. Figure 19 shows a factorized encoder model. Figure 20 shows a factorized self-attention model. Figure 21 shows a factorized dot-product attention model. DETAILED DESCRIPTION Example 1 Abstract In this study, we investigated the use of Vision Transformer (ViT) for Magnetic Resonance Image (MRI) processing in the context of Alzheimer’s disease (AD) diagnosis. We used ViT to extract features from MRIs and then mapped them to a feature sequence. In addition, we performed sequence modeling to maintain interdependencies among associated features and used a time series transformer to classify the features. We used Alzheimer’s Disease Neuroimaging Initiative (ADNI) T1-weighted MRIs to evaluate the performance of the proposed model for both binary classification of people with AD vs. non-AD patients and multiclass evaluation of patients with AD, non-AD, and mild cognitive impairment (MCI). Finally, we compared the performance of our proposed model with other deep learning models such as convolutional neural network (CNN), long short-term memory network (LSTM), Bidirectional LSTM (BiLSTM), CNN with BiLSTM, CNN with Transformer, and ViT with BiLSTM. The proposed method performs efficiently for AD diagnosis in terms of accuracy (99.048% and 99.014% for binary and multiclass disease classification, respectively), with better precision (accuracy of positive prediction), recall (positive instance), and F-score (balance between precision and recall). Our proposed model captures the long-range dependencies of the MRI sequences and models the global context of the MRI through a self- attention mechanism and reduces overfitting compared to traditional CNN models. 1 Introduction Alzheimer’s disease (AD) is distinguished by the accumulation of aberrant protein deposits in the brain, known as plaques and tangles, which result in the death of nerve cells and Docket No.2932719-000238-WO1 the degeneration of brain tissue. Neural degeneration reduces cognitive function and causes mood and behavior changes [1], [2]. AD is typically categorized in three stages [3], [4]. The first stage is the preclinical stage, characterized by brain, blood, and cerebrospinal fluid (CSF) abnormalities without outward signs [5]. It is believed that AD pathology begins at least 20 years before symptoms appear [6]. The second stage of the disease is referred to as mild cognitive impairment (MCI), which involves cognitive impairment confined to a single cognitive domain, usually memory. Dementia, the final stage of the disease, is defined as a cognitive disturbance in more than one domain, often memory and executive function, with substantial interference with daily life activities. With the recent approval of new drugs for early AD intervention, early detection of AD and differentiation of MCI have become of primary importance for successful disease treatment and management [7], [8], and to slow disease progression and improve the quality of life for those with AD [9],

[0010] . AD classification is one of the most challenging problems neurologists face [1]. Advances in computer-aided diagnosis (CAD) systems based on neuroimaging data tools have improved classification. CAD systems can be divided into conventional and deep learning-based techniques. Most traditional approaches to image analysis employ a four-stage pipeline of pre-processing, segmentation, feature extraction, and classification [9]. Deep learning (DL) algorithms have an advantage over conventionally based methods because they require little or no image pre-processing. They can automatically infer an optimal data representation from raw images without requiring prior feature selection, resulting in a more objective and less biased process

[0010] ,

[0011] ,

[0012] –

[0014] . CNN-based architectures are used extensively for medical image analysis. They have been applied to 2D and 3D ultrasound and MRI images

[0015] and are the deep models used most frequently to detect AD

[0015] –

[0018] . However, a 3D MRI brain image consists of stacked 2D data slices. 3D-CNN model to learn spatiotemporal features would be optimal, which is impossible with 2D CNN. However, because it requires many parameters and a high amount of computation, the 3D model cannot be used to construct deep models

[0015] . Although transformer architecture dominates natural language processing, its use in medical imaging has been limited

[0019] . However, Vision Transformer (ViT) has recently gained popularity due to its impressive results in various medical imaging tasks, including image classification, object detection, and semantic segmentation

[0020] . ViT took note of the scaling success of Transformers in NLP and applied a standard Transformer to images with minimal modifications. Transformer-based architectures have also been used in medical image analysis

[0021]

[0022]

[0025] . ViT has recently demonstrated superior performance in many computer- Docket No.2932719-000238-WO1 vision tasks, making it a viable alternative to CNN as a network architecture

[0023] . CNNs collect features gradually from local to global by adding more convolutional layers. ViT, on the other hand, uses a multi-headed self-attention mechanism to capture long-range dependencies. For this approach, the model equally weights all elements in the input sequence for superior performance. ViT extracts features across the entire image without degrading image resolution, preventing spatial loss from information skipping. Thus, ViT is ideal for brain imaging analysis. The self-attention strategy of ViT has the capacity to accurately capture the interdependencies between various dispersed networks of brain regions

[0024] . ViT is based on the concept of Transformers from natural language processing (NLP) applied to medical images. It uses a Transformer architecture to process MRI images instead of text. Other neural network models, which process the image sequences sequentially (RNN) or in parallel (CNN), require more time to train and infer the results, and fail to control for long-term dependencies among the image layers

[0025] . The joint transformer handles long-range dependencies, avoids recursion, and allows parallel computation to reduce training time and avoid performance drops due to long-range dependencies

[0025] . The goal of this article is twofold with respect to the proposed ViT: (1) evaluate the predictive performance of ViT combined with a transformer neural network; and (2) capture long-range dependencies and the global context of MRIs, allow parallel computation to reduce training time, and avoid performance drops due to long-range dependencies. We propose testing the hypothesis that ViT with a time series transformer performs better for AD patient classification based on MRI by using the self-attention mechanism to capture long-range dependencies and contextual relationships from MRI images. 2 Methods We used ViT to derive T1-weighted MRI slice attributes and a transformer neural network model for sequential feature classification, maintaining inter-association between the slices. The transformer neural network architecture and the ViT architecture for the sequential feature classification model are explained in Supplementary Sections 1 and 1.1. The summary of the ADNI dataset and steps of the proposed method are described in Sections 0-2.4, and the pipeline of the proposed method’s architecture is shown in Figure 1. Figure 1 shows the proposed pipeline of the ViT-TST. MRI images were pre-processed using CAT12 (image registration to standardize the images and skull stripping to reduce biases by ensuring consistent voxel intensities). ViT was used from each plane to derive slice attributes. Finally, the time series transformer was used to classify the feature sequences. 2.1 Dataset Docket No.2932719-000238-WO1 Data used in this study were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). The ADNI was launched in 2003 as a public- private partnership led by Principal Investigator Michael W. Weiner, MD. The primary goal of ADNI has been to test whether serial magnetic resonance imaging (MRI), positron emission tomography (PET), other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of mild cognitive impairment (MCI) and early Alzheimer’s disease (AD). We performed binary and multiclass classification using T1-weighted 3D MRI scans from ADNI

[0026] ,

[0027] . We trained and tested the models on subjects who had scans taken at screening and at 6- and 12-month visits (ADNI1: Complete 1Yr 1.5T data) and on subjects who had scans taken at screening, and at 6 months, 1 year, and 18 months (MCI only), and 2 and 3 years (normal and MCI only) (ADNI1: Complete 3Yr 3T data). We tested the model performance in three types of MRI scans (from the top down, axial plane; from front to back, coronal plane; and side to side, sagittal plane). We randomly split each data set into 60% training, 20% testing, and 20% validation sets. We performed extensive experiments on binary (NC / AD) and multiclassification tasks (NC / MCI / AD) to assess the proposed method. The details of model variants are listed in Supplementary Table S1. The configuration of training parameters is summarized in Supplementary Table S2. Both binary and multiclassifications were performed for all sagittal, coronal, and axial planes. We also implemented different baseline architectures to make comparisons with the proposed method. The details of baseline model variants are listed in Supplementary Table S3. The Complete 1Yr 1.5T data results are provided in Supplementary Section 3. The descriptive statistics of the ADNI data are provided in Error! Reference source not found.. Table 1. The details of data collections. # Image Scans NC MCI AD Male Female Age (years) ADNI1: Complete 3Yr 3T 351 129 145 77 194 157 75 േ 7.07 ADNI1: Complete 1Yr 1.5T 2294 705 1113 476 1341 953 75 േ 6.6 #NC, cognitively normal; MCI, mild cognitive impairment; AD, Alzheimer’s disease T1-weighted MRI scans were standardized in Montreal Neurological Institute (MNI) space. For comparison across subjects each skull was stripped using Statistical Parametric Mapping 12 (SPM12)

[0028] and Computational Anatomy Toolbox (CAT12; http: / / www.neuro.uni-jena.de / cat / ) in MATLAB (see Figure 2). Docket No.2932719-000238-WO1 Figure 2 shows a sample of an MRI slice in three planes (coronal, sagittal, and axial). A, original MRI; B, segmented anatomical image with the skull removed; and C, segmented anatomical scan warped to MNI space. 2.3 Handling 3D MRI using 2D ViT ViT models are pre-trained using a vast number of 2D data (mageNet21K

[0029] and 21,843 classes at a resolution of 224 ൈ 224 pixels. By taking advantage of transfer learning with a pre-trained 2D network model, we split the standardized 3D MRI into 2D slices. In the2D slices, the sizes of each slice in the axial, coronal, and sagittal planes were 113 ൈ 137,113 ൈ 113, and 137 ൈ 113 pixels, respectively.Each slice was then converted to an image of 224 ൈ 224 pixels, and each slice was divided into 14 ൈ 14 patches where each patch was 16 ൈ 16 pixels (see Figure 3). Figure 3 shows 14 ൈ 14 image patches. Patches of 16 ൈ 16 pixels were taken from the input images, with 14 ൈ 14 patches. These patches, which were flattened into a vector as a sequence of 196 patches, were considered to be an input token for the model, called the self-attention model. Later, we applied a multiple-layer self-attention model and feed-forward neural network to process the sequence of patched pixels and perform the feature selection. The proposed model comprised multiple transformer blocks, each applying the multi-head attention layer as a self-attention mechanism to the patch sequence. Finally, the output of the transformer encoder was processed via a classifier head to produce the final class probability output. Figure provides a visual summary of the proposed model. Figure 4 shows a visual summary of the proposed method. ViTs consist of several transformer blocks. Each transformer block comprises two sub-layers, a feed-forward layer, and a multi-head self-attention layer. 2.4 Sequence classification using a time series transformer After features extraction using ViT, a transformer model

[0025] was used as a tensor for time-series classification to maintain the relationships between the slices. Due to the self- attention mechanisms of transformer-based time series classifications, long-term dependencies between time steps (features of each slice) were captured more effectively. We then used feature embedding to map each token sequence to a meaningful numeric vector. As, ViT uses feature embedding to convert and map the features to a vector, we didn’t need another embedding for the time series transformer. As each slice provides dimensional numerical Docket No.2932719-000238-WO1 features, the MRI classification problem was changed to a multi-dimensional time series classification. The summary of the model is illustrated in Figure 5. Figure 5. Summary of the proposed model. Features were extracted using ViT and the sequence of features was classified using the time series transformer model. 3 Results The analysis was performed in two main steps: feature extraction and sequence modeling tasks on ADNI1: Complete 3Yr 3T and ADNI1: Complete 1Yr 1.5T MRI data. First, we compared our classification result with CNN alongside Bi-LSTM, CNN alongside Transformer, and ViT alongside Bi-LSTM. Then, we compared the classification performance, model accuracy, precision, F-score, and recall. The results for ADNI1: Complete 1Yr 1.5T are shown in the Supplement Section 2.1. 4 Experiments on ADNI1: Complete 3Yr 3T 4.1 Binary classification The results from the four architectures were similar, with CNN-TST and ViT-TST achieving the highest accuracy and precision scores of 98.81% and 0.99, respectively. Meanwhile, CNN-Bi-LSTM and ViT-Bi-LSTM had slightly lower scores but still performed well, with accuracy scores of 97.14% and 97.38%, respectively. To measure the ability of our proposed model to capture the positive instances, we calculated the recall score, and to measure the balance between the precision and the recall, we calculated the F-Score. The F-scores and recall scores for all architectures were very similar, ranging from 0.97 to 0.99, indicating good performance in these metrics (see Table 2). The architectures CNN-Bi-LSTM and ViT-TST had the highest accuracy, at 98.333%, while CNN-TST had the lowest accuracy, at 97.381%. Based on precision, F-score, and recall, the CNN-Bi-LSTM and ViT-TST architectures achieved the highest performance, with precision and recall scores of 0.98 and an F-score of 0.98. The CNN-TST also had a strong performance, with a precision and recall of 0.98 and an F-score of 0.97. The ViT-Bi-LSTM model had the lowest precision score, 0.97, but still achieved a strong F-score and recall score of 0.98 and 0.97, respectively. The table shows the performance of four different deep-learning architectures on a particular task. The ViT-TST architecture performed the best among all the models, achieving the highest accuracy of 99.048% and the highest scores for precision, F-score, and recall. The ViT-Bi-LSTM architecture also performed well, achieving an accuracy of 98.571% and competitive precision, F-score, and recall scores. The CNN-TST architecture attained an accuracy of 98.333% and high precision, F-score, and recall scores. Overall, the ViT-TST and Docket No.2932719-000238-WO1 ViT-Bi-LSTM architectures were better suited for the given task, followed by CNN-TST. The CNN-Bi-LSTM architecture was the least effective, although the differences were minor. The results on sagittal, coronal, and axial planes for binary classification (NC and AD) are listed in Table 2. Table 2. Results for binary disease classification on ADNI1: Complete 3Yr 3T. Architecture ACC Precision Recall F-score Sagittal CNN-Bi-LSTM 97.143% 0.97 0.97 0.97 (±3.658) CNN-TST 98.810% 0.99 0.99 0.99 (±2.195) ViT-Bi-LSTM 97.381% 0.97 0.97 0.97 (±2.704) ViT-TST 98.571% 0.99 0.98 0.98 (±2.857) Coronal CNN-Bi-LSTM 98.333% 0.98 0.98 0.98 the confusion matrix based on the proposed method and compared it with the other methods. The performance of the proposed method for binary disease classification using different MRI planes is shown in Figure 6. The diagonal cells indicate the prediction performance of the models when used to identify the true positive cases. ViT with time series transformer performed better than the other models for all the planes. Figure 6 shows confusion matrices for binary classification on ADNI1: Complete 3Yr 3T. Docket No.2932719-000238-WO1 4.2 Multiclass classification ViT-TST architecture achieved the highest accuracy and precision scores for multiclass classification based on sagittal planes. The CNN-Bi-LSTM model achieved an accuracy of 98.028%, with a precision of 0.98, an F-score of 0.98, and a recall of 0.98. The ViT-TST model achieved an accuracy of 98.310 %, with a precision of 0.98, an F-score of 0.98, and a recall of 0.98. All four architectures had the best precision score of 0.98, with the ViT-TST architecture achieving the highest accuracy and precision scores (see Table 3). For the coronal plane, CNN-Bi-LSTM achieved an accuracy of 96.479%, with a precision of 0.96, an F-score of 0.96, and a recall of 0.96. ViT-TST achieved the highest accuracy, 99.014%, with a precision of 0.99, an F-score of 0.99, and a recall of 0.99. Overall, the CNN-Bi-LSTM architecture had lower scores than the other architectures, especially for accuracy. For the axial plane, the ViT-Bi-LSTM and CNN-TST architectures also performed well, with accuracy, precision, F-score, and recall values above 0.98. The CNN-Bi-LSTM architecture had the lowest precision and F-score values but achieved high accuracy and recall value. These results suggest that the ViT-TST architecture is the most efficient for this task. The results on sagittal, coronal, and axial planes for multiclass classification (NC, MCI, and AD) are listed in Table 3. Table 3. Results for multiclass disease patient classification on ADNI1: Complete 3Yr 3T Architecture ACC Precision Recall F-score

[0002] Docket No.2932719-000238-WO1 ViT-Bi-LSTM 98.028% 0.98 0.98 0.98 (±1.804) ViT-TST 99.014% 0.99 0.99 0.99 (±1.268) * TST, time series transformer; ACC, accuracy; Precision, accuracy of positive prediction; Recall, accuracy of positive instances; F1-Score, balance between and recall positive (TP), true negative (TN), false positive (FP), and false negative (FN). It consistently demonstrated exceptional performance in accurately classifying instances belonging to various classes, with high TP and TN values across all classes. This demonstrated its ability to accurately identify positive instances and differentiate them from negatives. Additionally, the algorithm minimized false negatives (FN), reducing the risk of misclassifying positive occurrences as negatives, crucial in medical diagnostic applications such as AD classification. The ViT-TST also excelled in managing false positives (FP), ensuring that instances from other classes were correctly identified as non-relevant. Overall, the superiority of ViT-TST in TP, TN, FN, and FP further solidified its position as a highly effective and reliable choice for multiclass disease classification tasks. The performance of the proposed method based on a confusion matrix for multiclass classification on different planes as shown in Figure 7 Figure 7 shows confusion matrices of axial planes for multiclass disease patient classification (NC, MCI, and AD) on ADNI1: Complete 3Yr 3T. We also compared our proposed model with conventional deep models. Table and Table show numerical results from different deep learning model for the binary and multiclassification tasks on ADNI datasets and compared with our proposed model. For binary classification, the proposed method outperformed other traditional models. Moreover, as shown in Table, the proposed method ranked first among all competitors. Table 4. Performance compared with state-of-the-art methods for binary disease patient classification. work Input Image scans Method NC / AD classification(NC / MCI / AD) ACC (%) SEN (%) SPE (%)

[0030] Voxel based 429 / - / 858 3D CNN 90.3 82.4 96.5

[0031] Voxel based 119 / 233 / 97 3D DenseNet 88.9 86.6 90.8

[0032] Voxel based 330 / 299 / 299 3D CNN 93.2 95.0 89.8

[0033] Patch based 324 / 316 / 319 self-attention 98.0 97.7 98.2

[0034] Voxel based 457 / 808 / 346 3D ResNet 94.00 - -

[0035] Voxel based 407 / - / 418 3D CNN 99.20 98.90 99.50

[0036] ROI based 209 / 401 / 188 2.5D CNN 79.90 84.00 74.80 Proposed Method Sequence based 129 / 145 / 77 ViT-TST 99.048 99.5 - Docket No.2932719-000238-WO1 Proposed Method Sequence based 705 / 1113 / 476 ViT-TST 95.169 95.5 The results on sagittal, coronal, and axial planes for multiclass disease patient classification (NC, MCI, and AD) are shown in Table 5. Table 5. Performance compared with state-of-the-art methods for multiclass disease patient classification. work Input Image scans Method NC / MCI / AD classification (%) (NC / MCI / AD) ACC recall Precision SPE F1- Score

[0037] Slice based 300 / 300 / 300 VGGNet.16 91.85 - - - -

[0038] Voxel based 70 / 70 / 70 3D CNN 89.10 - - - -

[0039] Slice based 229 / 243 / 188 ResNet-18 56.80 - - - -

[0040] Slice based 169 / 234 / 101 2D CNN 96.00 96.0 - 98.0 -

[0041] Voxel based 574 / 808 / 346 3D ResNet 87.00 - - - -

[0042] Voxel based 207 / 215 / 193 3D VGGNet 91.13 - - - -

[0043] Slice based 50 / 50 / 50 VGGNet-16 95.73 96 96.33 - 95.66

[0044] Voxel based 351 / 297 / 221 3D DenseNets 97.52 97 97.13 - 97.02

[0045] Patch based 475 / 224 / 70 3D CNN 97.48 95.33 97.33 97.0

[0046] Slice based 25 / 13 / 25 ResNet18 & 98.21 98.14 - 98.14 - DenseNet121 Proposed Method Sequence-based 129 / 145 / 77 ViT-TST 99.01 99.0 99.0 99.0 Proposed Method Sequence-based 705 / 1113 / 476 ViT-TST 91.42 91.0 92.0 - 90.0 The ADNI1: Complete 1Yr 1.5T results are shown in Supplement Section 2.2.1. 5 Discussion In this study, we proposed use of a vision transformer with sequential transformer architecture for binary and multiclass MRI classification. We also compared the performance of the proposed model with other traditional image analysis architectures such as convolutional neural networks and long short-term memory networks on medium and large-sized ADNI datasets. Reducing false negatives (FN) is crucial in medical diagnosis and healthcare, since models can inaccurately categorize positive occurrences as negative, causing failure to identify or detect cases. In AD categorization, false negative outcomes may misclassify individuals as being in good health, preventing timely identification and administration of appropriate medical interventions. The ViT-TST algorithm appears to be the best choice for binary AD classification. It provides a balanced combination of accuracy and sensitivity, which is crucial for medical applications where both false positives and false negatives can have significant implications. Since minimizing FN is a top priority in AD classification, ViT-TST would be the most suitable choice. Docket No.2932719-000238-WO1 Our proposed method, the ViT-TST, outperformed the ViT-Bi-LSTM, CNN-TST, and CNN-Bi-LSTM models when classifying MCI patients. The ViT-TST achieved higher accuracy, sensitivity, or recall, resulting in an increase in the number of TP predictions and a decrease in the number of FN. This suggests that the ViT-TST accurately identified more MCI patients, reducing the chances that they would be misclassified as healthy or as belonging to other classes. The viability of the ViT-TST for multiclass disease classification tasks, particularly in the diagnosis of AD, is supported by its promising results.With the approval of new drugs for early intervention in AD [7], [8], early detection of AD and differentiation of MCI have become critical. Classifying MCI vs. NC or AD in a multiclassification framework is challenging because it is a heterogeneous condition with many subtypes and causes. Many studies have overlooked this problem and have experimented only with binary classification. However, our experiments were performed on binary (NC / AD) and multiclass disease classification tasks (NC / MCI / AD), and the performance of each model was evaluated using accuracy, precision, F-score, and recall. When averaging the accuracy rates over all planes for binary disease classification on the ADNI1: Complete 3Yr 3T data, ViT-TST achieved the highest average accuracy of 98.65% (see Table), followed closely by CNN-TST with 98.17%. ViT-Bi-LSTM and CNN-Bi-LSTM also performed well, with average accuracies of 97.85% and 97.22%, respectively (see Table). These results suggest that all four architectures are effective for binary disease patient classification on this dataset, with CNN-Bi-LSTM being the most effective. When averaging the accuracy rates over all planes for multiclass disease classification on the ADNI1: Complete 3Yr 3T, ViT-TST achieved the highest average accuracy of 98.77%, followed by ViT-Bi-LSTM with 97.74%. CNN-TST and CNN-Bi-LSTM also performed well, with average accuracies of 97.64% and 97.08%, respectively (see Table). These results suggest that all four architectures are effective for multiclass classification on this dataset, with ViT- TST being the most effective. Evaluation of all the results obtained revealed the ViT-TST architecture to be consistently among the top-performing architectures across all datasets and classification tasks. Therefore, the ViT-TST architecture may be a good choice when designing a classification model for similar datasets and tasks if computational resources and other practical considerations allow. Thus, we have devised a way to classify AD based on deep learning by combining pre- trained ViT and time series transformers. Many tasks have limited data, making training a model from scratch difficult. In small or unbalanced target datasets, transfer learning lets the Docket No.2932719-000238-WO1 model learn from a large dataset and adapt to the smaller target dataset. Transfer learning in ViT improves generalization, training speed, and adaptability to datasets and tasks; the proposed method generalizes well when trained on insufficient amounts of data. To take advantage of transfer learning with a pre-trained ViT, the 3D data from a plane were split into a 2D slice array. The problem with slice-based approaches such as CNN is that they fail to retain the dependencies of associated features between slices. To overcome this problem, the sequence classification task uses a time series classification with a transformer. Another alternative is to use 3D deep models; however, transfer learning with a pre-trained 3D is not currently available. Therefore, these approaches do not generalize well when trained on insufficient data. 6 Conclusions Overall, our results show that all four architectures achieved high levels of accuracy and performance on binary and multiclass disease classification tasks. Based on accuracy scores, all models performed well. However, we conclude that the Vi-TST and ViT-Bi-LSTM models perform better than the CNN-TST and CNN-Bi-LSTM models in terms of long-term dependencies among the spatial and temporal patterns in dynamic MRI sequences. In addition, they capture the global context using the self-attention mechanism to ensure that the relevant information from the entire or sequence of images is considered during classification, which reduces overfitting. Abbreviations AD: Alzheimer’s disease ADNI: Alzheimer’s Disease Neuroimaging Initiative ViT: vision transformer CNN: convolutional neural network RNN: recurrent neural networks LSTM: long short-term memory network Bi-LSTM: bidirectional long short-term memory network MRI: magnetic resonance imaging Supplemental to Example 1 1 Transformers The term ‘transformer’ first appeared in the field of machine translation. The Transformer Neural Network is an attempt to create an architecture that can efficiently deal Docket No.2932719-000238-WO1 with long-range dependencies, avoid recursion, and allow parallel computation to reduce training time and avoid performance drops due to long-range dependencies

[0030] . In natural language processing (NLP), a Transformer comprises two parts: Encoder and Decoder. The encoder turns the input (a sequence of words) into hidden layers, and the decoder turns the hidden layers back into sequences of words in natural language

[0031] . Self-attention-based architectures, in particular transformers, were built entirely on attention mechanisms. Neither reiteration nor rephrasing plays any role. NLP, computer vision, and audio processing are some of the areas in which transformers have revolutionized deep learning. The critical features of transformers consist of Non-sequential, Self-Attention, and Positional embeddings

[0030] . Transformers were invented for machine translation to translate sentences from one language to another (See Figure 8 (a)). The core of a transformer consists of a series of encoder and decoder layers. Encoder and decoder layers have their respective embedding layers for their corresponding inputs (Figure 8 (b and c)). The Transformer needs to know the meaning of each word and where it falls in the sequence. Word embedding represents words as real- valued vectors in a vector space such that similar words have similar encoding. The position encoding layer represents the position of a word in the sequence. The outputs of input Embedding are fed into the encoder. In the encoder, they first pass through a self-attention layer where they pay attention to themselves. In other words, the self-attention layer computes the relationship between tokens in an input sequence. Next, a position-wise feed-forward neural network (FFN) receives the output of the self-attention layer. Subsequently, a decoder with the same two layers as the encoder and an attention layer helps it focus on the most crucial parts of the input sequence. The output of the FFN is then sent upward to the subsequent Encoder (See Figure 8 (d)). The output of the final encoder is sent to each decoder in the decoder stack. It is worth mentioning that the structure of the Decoder is very similar to that of the Encoder, except that an additional multi-head attention layer, the Encoder-Decoder attention layer, is included in the decoder. The final Decoder in the stack feeds its output to the Output module, which transforms it into the output sentence. Figure 8 shows a general overview of transformers. Each encoder has two main sublayers, a multi-head attention layer and a fully connected FFN. This module consists of two consecutive layers, the linear and SoftMax layers. The Word Scores, which represent the likelihood of each word appearing in the target vocabulary, are projected from the Decoder vector by the Linear layer. The output sequence of the Transformer is created by the SoftMax layer, which converts these scores into probabilities and maps them to the appropriate word in the vocabulary. Docket No.2932719-000238-WO1 The self-attention mechanism is the foundation of a Transformer’s groundbreaking performance, which associates every word in the input sequence with every other word. Without self-attention, the relationships between words cannot be discovered in the NLP. In addition, in a time-series analysis, finding relationships across time is not possible without self- attention. A single-head attention layer can create a single context for all tokens. However, generating different contexts from one sequence is desired. Multi-head attention is a module for attention mechanisms that runs a different attention mechanism through a different head. This allows self-attention to work in parallel, which means that the same tokens can pay attention to one other from different points of view. 1.1 ViT The motivation behind ViT is the notable success of transformers in NLP. Therefore, in ViT, slight changes have been made to the standard transformer to enable the same performance with images. As input, the standard Transformer accepts a 1D sequence of token embeddings. The image is divided into patches handled in the same manner as visual tokens. To manage 2D images, ^^ ∈ ℝுൈ^ൈ^the image is flattened into a sequence of 2D patches ^^^∈ ℝேൈ^^మ∙^^, where (H; W) represents the size of the image, C represents the number of color channels, (P; P) the size of each image patch, and ^^= ^^ ^^ / ^^2is the number of patches. Next, each patch is turned into a vector and then projected linearly into tokens: ^^^ ൌ ^ ^^^^^^^,^^^^^, ^^ଶ^^, … , ^^ே^^൧, ^^ ∈ ℝ^^మൈ^, (1) where D is the is added so that the patches can maintain the information regarding their positions: ^^^^ൌ ^^^ ^ ^^^^^, ^^^^^∈ ℝ^ேା^^ൈ^(2) The tokens are then on top of one another. Each base block has a multi-head self-attention block and a multi-layer perceptron (MLP) block with Layer Norm (LNType equation here.), which is formulated as follows: ^^^ᇱ^ൌ MSA൫LN^ ^^ℓି^^൯^ ^^ℓି^, ℓ ൌ 1 … ^^ (3)(4) (5) Unlike the original transformer, the ViT does not have a decoder module. 1.2 Transformer model for time-series classification Transformer models have proven effective in numerous NLP applications, including translation and text classification. Due to its strong capability for capturing long-term Docket No.2932719-000238-WO1 dependencies, a Transformer is effective in various time-series classification tasks

[0032] . By treating the time series as a sequence of tokens, researchers have recently applied Transformers to the classification of time series. The vanilla transformer was first developed for machine translation. The aim was to translate a sentence from one language to another. It mainly consisted of a stack of encoder and decoder layers. The Encoder mapped the input (language sequence) into hidden layers, and the Decoder mapped the output (natural language sequences). The classification application has no sequence output, unlike the machine translation application. Because of this, the decoder is not used in the transformers. Viewed from a mathematical perspective, a time series comprises ^^ time steps and ^^ classes. Each time step ^^ in the time series has ^^ features. The time series can be represented as a matrix ^^ with dimensions ^^ ൈ ^^, where each row represents a time step, and each column represents a feature. It takes as input the time series matrix ^^ and outputs an attention matrix with the exact dimensions. There are several encoders in the encoder stack. Each encoder consists of a multi-head attention layer and a feed-forward layer. A series of feed-forward layers and self-attention layers make up the encoder of the transformer model. The self- attention layer models the temporal dependencies between various time steps in the time series. Embedding and Position Encoding feed the first encoder. The previous Encoder feeds the stack. Each encoder has two major sublayers, a multi-head attention layer and a position-wise fully connected FFN (Figure 8 (d)). Along with these two sublayers, it also has Residual skip connections around both layers and two Layer Norm layers (see Figure 9). Figure 9 shows the layers of self-attention in the Encoder, showing residual skip connections around both layers and two Layer Norm layers. In the self-attention layer, attention weights are computed between each time step and every other time step in the time series. These attention weights are between 0 and 1. A scaled dot-product attention function converts a query vector ( ^^) and a set of key-value ( ^^) pairs into an output vector ( ^^). The importance of each time step in representing the temporal patterns of the time series is reflected in these attention weights. The following equation is used to calculate the attention weights (output matrix): ℎ^^ ^^ ^^ ൌ attention Q, K, V ൌ softmax ൬^^^^ ^(6)In the attention mechanism, the query vector processed, the key (K) represents the context or memory that the model compares against, and the value (V) is the associated information that gets combined based on the attention scores. Docket No.2932719-000238-WO1 Each head in the attention layer independently processes different parts (or patches) of the input, enabling the model to capture various aspects of the input simultaneously. The attention weights are normalized by the SoftMax function so that their sum is 1. The square root of ^^ ^^ is a scaling factor that aids in stabilizing the gradients during training. Unlike the sequence-to-sequence transformer, which had the same Q, K, but a different V, in the time-series classification transformer, all the matrices are identical. A multi-head attention layer comprises parallel scaled dot-product attention layers, each referred to as a head. The outputs from the H heads are then added and projected onto another dense layer to create the final hidden representation. The multi-head attention function is asfollows:multihead ^Q, K, V^ ൌ concat ^head , head … head ^W^^ ଶ ୦ (7)head ൌ attention ^ ^ ^୧ ൫QW୧ , KW୧ , VW୧ ൯ (8)where QW^୧ ,KW୧^and VW୧^are weight matrices that map the input matrix ^^ to the query, key, and value spaces. Lastly, the feed-forward layer output is passed through a SoftMax function to generate a probability distribution over the R classes. Based on the class with the highest probability, the time series class label is predicted. 2 Experimental setting All the experiments were carried out using Python on an Intel (R) Core (TM) i5-6400 CPU @ 2.70GHz 2.71, 16 GB RAM, and 8GB NVIDIA GeForce RTX 2080 running Windows models were implemented using Transformers, Scikit-Learn libraries, and Keras on top of TensorFlow. The model was trained by Adam optimizer for 100 epochs with a batch size of 32 and a learning rate of 1e-4. Sparse Categorical Cross-entropy was used as a loss function. Also, in the final layers, SoftMax classifiers were used (Table S1).

[0003] Docket No.2932719-000238-WO1 In addition, as the call-back, the model checkpoint was set to save only the best solution found during training based on the validation’s loss function evaluation. This was accomplished by monitoring the metric on a validation set during training and saving the model checkpoint only when the metric improved. The model checkpoint was saved when the metric improved on a validation set during training (Table S2). The details of baseline model variants are listed in Table S3. Docket No.2932719-000238-WO1 2 2.1.1 Binary classification The results on sagittal planes for binary classification show that among the four models, CNN-Bi-LSTM performed the best with an accuracy of 96.314% (±1.039), followed closely by ViT-TST with an accuracy of 95.000% (±2.151). Both models achieved high precision, F- score, and recall, indicating they are well-suited for binary classification on sagittal planes. The other two models, CNN-TST and ViT-Bi-LSTM, also performed reasonably well, with accuracy scores of 94.492% (±2.169) and 94.364% (±2.085), respectively. However, their precision, F-score, and recall scores were slightly lower than those of the top-performing models. On coronal planes, ViT-Bi-LSTM achieved the highest average accuracy (94.958%), followed closely by CNN-TST (94.195%), CNN-Bi-LSTM (94.237%), and ViT-TST (94.153%). The precision, F-score, and recall of all algorithms were also quite high, achieving over 94% for each metric. However, CNN-TST achieved a similar accuracy (94.195%) with a lower standard deviation, indicating that it may be a more stable algorithm overall. Docket No.2932719-000238-WO1 The axial plane results clearly show that all four algorithms perform well, with accuracy ranging from 94.66% to 95.68%. The ViT-Bi-LSTM algorithm has the highest accuracy (95.68%) and F-score (0.96). The highest precision and recall values are 0.96. The ViT-TST algorithm has the second-highest accuracy of 95.17%, with precision, F-score, and recall of 0.95. CNN-TST outperforms CNN-Bi-LSTM with 94.78% accuracy. Both algorithms have 0.95 precision, F-score, and recall. The best axial plane algorithm is ViT-Bi-LSTM, followed by ViT-TST (see Table S3). The results for binary classification (NC and AD) on the ADNI data set are shown in Table S3. We also calculated each model’s classification error rate (confusion matrix). Figure 10 depicts the confusion matrices of all the planes for binary classification. Figure 10 shows confusion matrices for binary classification on ADNI1: Complete 1Yr 1.5T. 2.1.2 Multiclass classification The table presents the performance of four different architectures for multiclass classification of brain images in sagittal planes, using various evaluation metrics. The CNN- Bi-LSTM, CNN-TST, ViT-Bi-LSTM, and ViT-TST architectures achieved accuracy scores of 90.24%, 87.211%, 88.824%, and 89.237%, respectively. In terms of precision, the CNN-Bi- LSTM architecture achieved the highest score of 0.91, followed by both ViT architectures with a precision score of 0.89, and finally, the CNN-TST architecture with a precision score of 0.87. In recall, the CNN-Bi-LSTM architecture achieved the highest score of 0.90, followed by the ViT-TST architecture with a recall score of 0.89, the ViT-Bi-LSTM architecture with a recall Docket No.2932719-000238-WO1 score of 0.88, and finally, the CNN-TST architecture with an F-score of 0.86. The standard deviations for all four architectures range from 1.336 to 2.157, indicating relatively small differences in performance. Comparing the results of the four architectures, the ViT-TST architecture achieved the highest accuracy, precision, and recall scores, with an accuracy of 91.416%, precision of 0.92, and recall of 0.91. This suggests that the ViT-TST architecture is the most effective at classifying the three classes (NC, MCI, and AD) on the Complete 1Yr 1.5T. In addition, the CNN-Bi-LSTM and CNN-TST architectures achieved similar scores in all metrics, with the CNN-Bi-LSTM architecture having a slightly higher accuracy of 90.261% (±2.220) compared to the CNN-Transformer architecture with 90.174% (±2.084). This suggests that both architectures are similarly effective at classifying the three classes on the Complete 1Yr 1.5Tt. Conversely, the ViT-Bi-LSTM architecture achieved the lowest scores in all metrics, with an accuracy of 89.847% (±2.117), precision of 0.90, recall of 0.89, and F-score of 0.89. This suggests that the ViT-Bi-LSTM architecture is the least effective at classifying the three classes on the Complete 1Yr 1.5T. Looking at the performance of the four architectures on the Complete 1Yr 1.5T dataset for multiclass classification (NC, MCI, and AD), the ViT-TST and CNN-TST architectures achieved the highest accuracy scores, both performing better than the other two architectures. The ViT-Bi-LSTM architecture also performed well with a slightly lower accuracy score than the previously mentioned architectures. The CNN-Bi-LSTM architecture had the lowest accuracy score of the four, although it still achieved a respectable score of 88.976%. The precision, F-score, and recall scores were consistent with the accuracy scores, with the ViT- TST and CNN-TST architectures achieving the highest scores and the CNN-Bi-LSTM architecture achieving the lowest scores. The results are reliable and consistent, with relatively small standard deviations for all architectures. Table 4 shows the results of multiclass classification (NC, MCI, and AD) in the sagittal, coronal, and axial planes, respectively.

[0004] Docket No.2932719-000238-WO1 Figure 11 shows confusion matrices of sagittal planes for multiclass classification on ADNI1: Complete 1Yr 1.5T. In comparison to the original Vision Transformer (ViT), the embodiment described above uses time series classification. This model leverages the transformer architecture specifically designed to handle sequential time series data, enabling effective classification tasks that the original ViT, which was focused on image data, does not address. With respect to the embodiment described above, note that the linear projection of patch features in models like Vision Transformer (ViT) does involve embedding other than the position embedding. Specifically, the linear projection transforms each image patch into a vector (embedding) that represents the patch's features in a lower-dimensional space. This patch embedding captures the content of the patch itself, independent of its position in the image. The position embedding is then added to these patch embeddings to encode the positional information, helping the model understand the order and spatial arrangement of the patches. References [1] R. A. Hazarika, D. Kandar, and A. K. Maji, “An experimental analysis of different Deep Learning based Models for Alzheimer’s Disease classification using Brain Magnetic Resonance Images,” Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 10, pp. 8576–8598, Nov. 2022, doi: 10.1016 / J.JKSUCI.2021.09.003. [2] R. Jain, N. Jain, A. Aggarwal, and D. J. Hemanth, “Convolutional neural network based Alzheimer’s disease classification from magnetic resonance brain images,” Cogn Syst Res, vol.57, pp.147–159, Oct.2019, doi: 10.1016 / J.COGSYS.2018.12.015. Docket No.2932719-000238-WO1 [3] K. Blennow, … H. Z.-C. S., and U. 2012, “Fluid biomarkers in Alzheimer disease,” perspectivesinmedicine.cshlp.org, 2012, doi: 10.1101 / cshperspect.a006221. [4] M. Khojaste-Sarakhsi, S. S. Haghighi, S. M. T. F. Ghomi, and E. Marchiori, “Deep learning for Alzheimer’s disease diagnosis: A survey,” Artif Intell Med, vol. 130, p. 102332, Aug.2022, doi: 10.1016 / J.ARTMED.2022.102332. [5] A. A.-A. & dementia and U.2019, “2019 Alzheimer’s disease facts and figures,” Wiley Online Library, 2012. [6] A. Alberdi, A. Aztiria, and A. Basarab, “On the early diagnosis of Alzheimer’s Disease from multimodal signals: A survey,” Artif Intell Med, vol.71, pp. 1–29, Jul. 2016, doi: 10.1016 / J.ARTMED.2016.06.003. [7] E. McDade et al., “Lecanemab in patients with early Alzheimer’s disease: detailed results on biomarker, cognitive, and clinical effects from the randomized and open-label extension of the phase 2 proof-of-concept study,” Alzheimers Res Ther, vol. 14, no. 1, Dec.2022, doi: 10.1186 / S13195-022-01124-2. [8] J. R. Sims et al., “Donanemab in Early Symptomatic Alzheimer Disease: The TRAILBLAZER-ALZ 2 Randomized Clinical Trial,” JAMA, Jul. 2023, doi: 10.1001 / JAMA.2023.13239. [9] A. Loddo, S. Buttau, and C. Di Ruberto, “Deep learning based pipelines for Alzheimer’s disease diagnosis: A comparative study and a novel deep-ensemble method,” Comput Biol Med, vol.141, p.105032, Feb.2022, doi: 10.1016 / J.COMPBIOMED.2021.105032.

[0010] B. Zhao, H. Lu, S. Chen, J. Liu, and D. Wu, “Convolutional neural networks for time series classification,” Journal of Systems Engineering and Electronics, vol. 28, no. 1, pp.162–169, Feb.2017, doi: 10.21629 / JSEE.2017.01.18.

[0011] Q. Wen et al., “Transformers in Time Series: A Survey,” Feb. 2022, doi: 10.48550 / arxiv.2202.07125.

[0012] L. Yue, X. Gong, J. Li, H. Ji, M. Li, and A. K. Nandi, “Hierarchical feature extraction for early Alzheimer’s disease diagnosis,” IEEE Access, vol.7, pp.93752–93760, 2019, doi: 10.1109 / ACCESS.2019.2926288.

[0013] I. R. R. Silva, G. S. L. Silva, R. G. De Souza, W. P. Dos Santos, and R. A. A. De Fagundes, “Model Based on Deep Feature Extraction for Diagnosis of Alzheimer’s Disease,” Proceedings of the International Joint Conference on Neural Networks, vol. 2019-July, Jul.2019, doi: 10.1109 / IJCNN.2019.8852138.

[0014] F. Zhang, Z. Li, B. Zhang, H. Du, B. Wang, and X. Zhang, “Multi-modal deep learning model for auxiliary diagnosis of Alzheimer’s disease,” Neurocomputing, vol. 361, pp. 185–195, Oct.2019, doi: 10.1016 / J.NEUCOM.2019.04.093.

[0015] J. Jang and D. Hwang, “M3T: Three-Dimensional Medical Image Classifier Using Multi-Plane and Multi-Slice Transformer.” pp.20718–20729, 2022.

[0016] K. A. N. N. P. Gunawardena, R. N. Rajapakse, and N. D. Kodikara, “Applying convolutional neural networks for pre-detection of Alzheimer’s disease from structural MRI data,” 201724th International Conference on Mechatronics and Machine Vision in Practice, M2VIP 2017, vol. 2017-December, pp. 1–7, Dec. 2017, doi: 10.1109 / M2VIP.2017.8211486.

[0017] H. Choi and K. H. Jin, “Predicting cognitive decline with deep learning of brain metabolism and amyloid imaging,” Behavioural Brain Research, vol.344, pp.103–109, May 2018, doi: 10.1016 / J.BBR.2018.02.017.

[0018] S. Esmaeilzadeh, D. I. Belivanis, K. M. Pohl, and E. Adeli, “End-to-end alzheimer’s disease diagnosis and biomarker identification,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol.11046 LNCS, pp.337–345, 2018, doi: 10.1007 / 978-3-030-00919- 9_39 / COVER. Docket No.2932719-000238-WO1

[0019] A. Dosovitskiy et al., “AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE”, Accessed: Mar. 28, 2023. [Online]. Available: https: / / github.com /

[0020] J. Kim, K. Shim, J. Kim, and B. Shim, “Vision Transformer-based Feature Extraction for Generalized Zero-Shot Learning,” Feb.2023, doi: 10.48550 / arxiv.2302.00875.

[0021] Y. Lyu, X. Yu, D. Zhu, and L. Zhang, “Classification of Alzheimer’s Disease via Vision Transformer: Classification of Alzheimer’s Disease via Vision Transformer,” ACM International Conference Proceeding Series, pp. 463–468, Jun. 2022, doi: 10.1145 / 3529190.3534754.

[0022] R. Kushol, A. Masoumzadeh, D. Huo, S. Kalra, and Y. H. Yang, “Addformer: Alzheimer’s Disease Detection from Structural Mri Using Fusion Transformer,” Proceedings - International Symposium on Biomedical Imaging, vol.2022-March 2022, doi: 10.1109 / ISBI52829.2022.9761421.

[0023] J. Li, J. Chen, Y. Tang, C. Wang, B. A. Landman, and S. K. Zhou, “Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives,” Med Image Anal, vol. 85, p. 102762, Apr. 2023, doi: 10.1016 / J.MEDIA.2023.102762.

[0024] Y. Lyu, X. Yu, D. Zhu, and L. Zhang, “Classification of Alzheimer’s Disease via Vision Transformer: Classification of Alzheimer’s Disease via Vision Transformer,” ACM International Conference Proceeding Series, pp. 463–468, Jun. 2022, doi: 10.1145 / 3529190.3534754.

[0025] A. Vaswani et al., “Attention Is All You Need,” Adv Neural Inf Process Syst, vol.2017- December, pp.5999–6009, Jun.2017, doi: 10.48550 / arxiv.1706.03762.

[0026] “ADNI | Alzheimer’s Disease Neuroimaging Initiative.” https: / / adni.loni.usc.edu / (accessed Apr.03, 2023).

[0027] C. R. Jack et al., “The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods,” Journal of Magnetic Resonance Imaging, vol. 27, no. 4, pp. 685–691, Apr. 2008, doi: 10.1002 / JMRI.21049.

[0028] J. Ashburner and K. J. Friston, “Unified segmentation,” Neuroimage, vol.26, no.3, pp. 839–851, Jul.2005, doi: 10.1016 / J.NEUROIMAGE.2005.02.018.

[0029] T. Ridnik, E. Ben-Baruch, A. Noy, and L. Zelnik-Manor, “ImageNet-21K Pretraining for the Masses,” Apr. 2021, Accessed: Apr. 05, 2023. [Online]. Available: https: / / arxiv.org / abs / 2104.10972v4

[0030] C. Lian, M. Liu, J. Zhang, and D. Shen, “Hierarchical fully convolutional network for joint atrophy localization and Alzheimer’s disease diagnosis using structural MRI,” IEEE Trans Pattern Anal Mach Intell, vol. 42, no. 4, pp. 880–893, Apr. 2020, doi: 10.1109 / TPAMI.2018.2889096.

[0031] M. Liu et al., “A multi-model deep convolutional neural network for automatic hippocampus segmentation and classification in Alzheimer’s disease,” Neuroimage, vol. 208, p.116459, Mar.2020, doi: 10.1016 / J.NEUROIMAGE.2019.116459.

[0032] J. Li, Y. Wei, C. Wang, Q. Hu, Y. Liu, and L. Xu, “3-D CNN-Based Multichannel Contrastive Learning for Alzheimer’s Disease Automatic Diagnosis,” IEEE Trans Instrum Meas, vol.71, 2022, doi: 10.1109 / TIM.2022.3162265.

[0033] J. Zhu et al., “Efficient self-attention mechanism and structural distilling model for Alzheimer’s disease diagnosis,” Comput Biol Med, vol.147, p.105737, Aug.2022, doi: 10.1016 / J.COMPBIOMED.2022.105737.

[0034] H. Karasawa, C. L. Liu, and H. Ohwada, “Deep 3D Convolutional Neural Network Architectures for Alzheimer’s Disease Diagnosis,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Docket No.2932719-000238-WO1 Bioinformatics), vol.10751 LNAI, pp.287–296, 2018, doi: 10.1007 / 978-3-319-75417- 8_27.

[0035] S. Basaia, F. Agosta, L. Wagner, E. Canu, G. M.-N. Clinical, and undefined 2019, “Automated classification of Alzheimer’s disease and mild cognitive impairment using a single MRI and deep neural networks,” Elsevier, Accessed: Mar. 29, 2023. [Online]. Available: https: / / www.sciencedirect.com / science / article / pii / S2213158218303930

[0036] W. Lin et al., “Convolutional neural networks-based MRI image analysis for the Alzheimer’s disease prediction from mild cognitive impairment,” Front Neurosci, vol. 12, no. NOV, p.777, Nov.2018, doi: 10.3389 / FNINS.2018.00777 / BIBTEX.

[0037] C. D. Billones, O. J. L. D. Demetria, D. E. D. Hostallero, and P. C. Naval, “DemNet: A Convolutional Neural Network for the detection of Alzheimer’s Disease and Mild Cognitive Impairment,” IEEE Region 10 Annual International Conference, Proceedings / TENCON, pp. 3724–3727, Feb. 2017, doi: 10.1109 / TENCON.2016.7848755.

[0038] E. Hosseini-Asl, … R. K.-2016 I. international, and undefined 2016, “Alzheimer’s disease diagnostics by adaptation of 3D convolutional network,” ieeexplore.ieee.org, Accessed: Mar. 29, 2023. [Online]. Available: https: / / ieeexplore.ieee.org / abstract / document / 7532332 / ?casa_token=Neb5n7ikTZMA AAAA:PkEGLJT7qw9U49OS9KRibb0AFV1ImpMxt_SViSMvquUaRK5BjceVLgbe 3YznJAG0Tw20np3KSMNT0Pk

[0039] A. Valliani, A. S.-P. of the 8th A. international conference, and undefined 2017, “Deep residual nets for improved Alzheimer’s diagnosis,” dl.acm.org, p.615, Aug. 2017, doi: 10.1145 / 3107411.3108224.

[0040] K. Gunawardena, … R. R.-… on M. and, and undefined 2017, “Applying convolutional neural networks for pre-detection of alzheimer’s disease from structural MRI data,” ieeexplore.ieee.org, Accessed: Mar. 29, 2023. [Online]. Available: https: / / ieeexplore.ieee.org / abstract / document / 8211486 / ?casa_token=0Vm5OBjwvlYA AAAA:PKVbNMAMIsDz- HLaNNWN_khu_UcFL6wO3FEtTxOxVE5tuCet49yXSNB- smS9lU5C5knfNN1GTZT9RWs

[0041] H. Karasawa, C. L. Liu, and H. Ohwada, “Deep 3D Convolutional Neural Network Architectures for Alzheimer’s Disease Diagnosis,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol.10751 LNAI, pp.287–296, 2018, doi: 10.1007 / 978-3-319-75417- 8_27.

[0042] T. D. Vu, N. H. Ho, H. J. Yang, J. Kim, and H. C. Song, “Non-white matter tissue extraction and deep convolutional neural network for Alzheimer’s disease detection,” Soft comput, vol.22, no.20, pp.6825–6833, Oct.2018, doi: 10.1007 / S00500-018-3421- 5.

[0043] R. Jain, N. Jain, A. Aggarwal, D. H.-C. S. Research, and undefined 2019, “Convolutional neural network based Alzheimer’s disease classification from magnetic resonance brain images,” Elsevier, Accessed: Mar. 29, 2023. [Online]. Available: https: / / www.sciencedirect.com / science / article / pii / S1389041718309562

[0044] H. Wang et al., “Ensemble of 3D densely connected convolutional network for diagnosis of mild cognitive impairment and Alzheimer’s disease,” Neurocomputing, vol.333, pp. 145–156, Mar.2019, doi: 10.1016 / J.NEUCOM.2018.12.018.

[0045] N. Goenka and S. Tiwari, “AlzVNet: A volumetric convolutional neural network for multiclass classification of Alzheimer’s disease through multiple neuroimaging computational approaches,” Biomed Signal Process Control, vol. 74, p. 103500, Apr. 2022, doi: 10.1016 / J.BSPC.2022.103500. Docket No.2932719-000238-WO1

[0046] M. Odusami, R. Maskeliūnas, and R. Damaševičius, “An Intelligent System for Early Recognition of Alzheimer’s Disease Using Neuroimaging,” Sensors 2022, Vol. 22, Page 740, vol.22, no.3, p.740, Jan.2022, doi: 10.3390 / S22030740. Example 2 Abstract Alzheimer's disease (AD) is a neurodegenerative disorder affecting millions worldwide, necessitating early and accurate diagnosis for optimal patient management. In recent years, advancements in deep learning have shown remarkable potential in medical image analysis. In this study, we present ‘ViTranZheimer,’ an AD diagnosis approach by leveraging video vision transformers to analyze 3D brain MRI data. By treating the 3D MRI volumes as videos, we exploit the temporal dependencies between slices to capture intricate structural relationships. The video vision transformer's self-attention mechanisms enable the model to learn long-range dependencies and identify subtle patterns that may indicate AD progression. Our proposed deep learning framework seeks to enhance the accuracy and sensitivity of AD diagnosis, empowering clinicians with a powerful tool for early detection and personalized treatment planning. We thoroughly validate the performance of the video vision transformer on ADNI dataset and conduct comparative analyses with other relevant models. The proposed model is compared with two hybrid models, CNN-BiLSTM and ViT-BiLSTM. CNN-LSTM is the combination of a convolutional neural network and a bidirectional long-short-term memory network, while ViT-BiLSTM is the combination of a vision transformer with BiLSTM. The accuracy level achieved in the ViTranZheimer, CNN-BiLSTM, and ViT-BiLSTM models is 96.479%, 97.465%, and 98.6%, respectively. This research advances the understanding of applying deep learning techniques in neuroimaging and Alzheimer's disease research, paving the way for more effective and efficient clinical diagnosis and interventions. 1. Introduction Until now, Alzheimer's disease (AD) has persisted as one of the most debilitating chronic neurological disorders, predominantly impacting individuals aged 65 and above [1]. From a clinical standpoint, AD is marked by a wide range of dementia symptoms, such as memory problems, aphasia, apraxia, agnosia, problems with spatial skills, executive dysfunction, and changes in personality and behavior [2,3]. With the increasing aging population, AD is anticipated to emerge as a significant global burden in the forthcoming decades. According to the World Health Organization (WHO) [4], the prevalence of AD was estimated to be 50 million individuals in 2015, with projections indicating it will triple by the Docket No.2932719-000238-WO1 year 2050 [1,2]. Typically, the advancement of Alzheimer's disease (AD) can be categorized into three stages: normal cognition (NC), mild cognitive impairment (MCI), and AD [5]. The accurate diagnosis of AD and its early stage, mild cognitive impairment (MCI), is of utmost importance due to the rapid rise in its prevalence. This is essential for timely treatment and potentially delaying the progression of AD [6]. At the moment, there is no cure for AD, and treatments can only slow the disease's progress [4]. Early diagnosis is very important because the most effective treatments for Alzheimer's disease are those that are used early on [7]. AD can be diagnosed using various techniques, including cognitive and neuropsychological tests, neurological examinations, biomarkers, genetic testing, and Neuroimaging. Neuroimaging with structural magnetic resonance imaging (sMRI) helps find brain damage (such as atrophy, tumors, and lesions) and rule out causes of dementia other than AD. sMRI is a painless, non- invasive process used in clinical settings for diagnosing AD without injections, surgery, or radiation exposure, making it safe and well-tolerated by patients. In recent years, advances in deep learning have indeed revolutionized medical image analysis [8]. The use of this technique has demonstrated remarkable potential in various facets of medical imaging, bringing substantial advantages to healthcare and diagnosis. Like other areas of medical image analysis, brain MRI has benefited greatly from the application of deep learning [9,10]. Deep learning methods in image processing utilize advanced neural network architectures to automatically learn and extract relevant features from images. Key deep learning methods include Convolutional Neural Networks (CNNs)

[0011] , Recurrent Neural Networks (RNNs) [12–14], and Vision Transformers (ViTs)

[0015] . CNNs use convolutional layers to scan images with learnable filters, detecting patterns and features at different scales. RNNs are designed to handle sequential data but can also be adapted for image processing tasks, such as image captioning. ViTs use self-attention mechanisms to process images as sequences of tokens, capturing spatial relationships and dependencies among image patches. These methods have shown impressive results in image classification, object detection, and other vision tasks. CNNs have proven to be highly effective in computer vision tasks like image classification

[0016] , object detection / tracking [17,18], and semantic segmentation

[0019] . They are also widely used in medical image analysis, particularly in 2D and 3D ultrasound and MRI images

[0020] , and are the deep models most commonly used to detect AD [20–22]. CNNs are based on the brain's visual cortex. They can take 2D or 3D images as input data and use several convolutional layers to pull out features

[0023] . Docket No.2932719-000238-WO1 Since an MR brain scan is 3D data made of stacked 2D slices, a model that can describe the whole brain is needed. A 3DCNN model to learn spatiotemporal features is the most straightforward approach, as 2DCNN cannot do so. However, 3DCNN has many parameters and computation is high. Slice-based methods split 3D neuroimages into 2D slices for MRI analysis. However, slice-based methods analyze 3D neuroimages into 2D slices in MRI scans. When classifying 3D-MRI scans using slide-based methods, spatial features in each slice and their relationships must be considered. Pre-trained 2D-CNNs can extract slice-specific features. Deep sequence modeling models like RNN-based models can then model slice dependency. In contrast to CNN-based models, ViT-based models lack an inductive bias; therefore, it is necessary to train these models with massive amounts of information. Since ViT models are pre-trained on a large number of two-dimensional data, their use on two-dimensional data becomes efficient. Since the MRI is 3D and the available data are insufficient to train the 3D ViT, it is unreasonable to expect the 3D ViT to produce similarly effective results as the standard ViT. The common method involves splitting 3D data into 2D slice array from plane to take advantage of transfer learning with a pre-trained 2D network model. ViT extracts slice features separately, utilizing deep sequence models like RNN to model slice dependency, ensuring relationships between slices are maintained. A method is introduced herein for diagnosing AD by applying video vision transformers to 3D brain MRI scans. 2. Materials and method 2.1 Structural MRI data The ADNI [24,25] dataset, created in 2003, is a comprehensive set of data on Alzheimer's disease (AD) and related disorders. It includes data from various sources, including cognitive tests, genetic analysis, PET, and MRI. The ADNI MRI Core has created standardized analysis sets for analysis, allowing for more careful comparisons and meaningful comparisons of algorithms. This study used two medium and large-sized shared standard data collections, ADNI1: Complete 3Yr 3T and ADNI1: Complete 1Yr 1.5T, to compare different brain structures and algorithms. The medium dataset included 351 image scans, while the large dataset included 2294 scans. Both datasets were randomly split into 60% training, 20% testing, and 20% validation sets. The study aimed to ensure meaningful comparisons and reduce the possibility of differences in algorithm performance due to different input data. Table 1 shows details of the data collection. Table 1. The details of data collections Docket No.2932719-000238-WO1 # Image Scans NC MCI AD Male Female Age ADNI1: Complete 3Yr 3T 351 129 145 77 194 157 75 േ 7.07 2.2 MRI pre-processing The processing and analysis of sMRI data consists of several steps. These steps include aligning the images, registering them to a standard template such as MNI space, segmenting the images into different tissue types such as gray matter, white matter, and cerebrospinal fluid, and conducting voxel-based analyses (See Figure 12 and Table 2). Image registration is crucial for ensuring the spatial alignment of MRI scans, as it helps standardize them with a fixed-size template. Figure 12 shows a sample of MRI slices from subjects with NC, MCI, and dementia. The first columns show the original MRI, the second, third, and fourth columns show colored,segmented, and normalized MRI, respectively. Voxel resolution (original > intern; vox):1.00 ൈ 1.01 ൈ 1.20 ^^ ^^ଷ ^ 1.00 ൈ 1.01 ൈ 1.20 ^^ ^^ଷ; 1.50ଷ ^^ ^^ଷThe Montreal Neurological Institute's MNI space is a common coordinate system for neuroimaging based on a template brain from hundreds of healthy MRI scans. This space provides a more precise and uniform reference frame for analyzing brain activity across various imaging studies and research teams. To perform skull stripping, normalization, and image registration, we wrapped MRI scans in the MNI-152 space using the Statistical Parametric Mapping 12

[0026] tool and the Computational Anatomy Toolbox in MATLAB. Moreover, out of all the frames, only the central 32 sections containing significant data are selected, while the remaining sections from the beginning and end are excluded due to their predominantly consisting of background information unrelated to the brain tissue. Table 2. The details of MRIs for subjects with NC, MCI, and dementia Image processing quality: Volumes CSF GM WM Resolution 83.47% Absolute volume 301 593 462 ^^ ^^ଷC Bias 98.01% Relative volume 22.2 43.7 34.1% N weighted image quality 82.46% TIV 1356 ^^ ^^ଷ(IQR) Resolution 83.58% Absolute volume 396 708 617 ^^ ^^ଷIC Bias 98.16% Relative volume 23.0 41.1 35.9% M weighted image quality 83.34% TIV 1721 ^^ ^^ ^^ଷ(IQR) Resolution 83.51% Absolute volume 691 699 601 ^^ ^^ଷDBias 97.13% Relative volume 34.7 35.1 30.2% A weighted image quality 85.08% TIV 1991 ^^ ^^ଷ(IQR) Docket No.2932719-000238-WO1 TIV (Third module: estimate total intracranial volume); CSF (Cerebrospinal fluid); GM (Gray matter); WM (White matter) 2.3 Classification This section provides a comprehensive presentation of the proposed framework. We employed 3D-MRI in our approach for the classification of AD. In sMRI, the images are typically represented as volumes or 3D stacks in a 3D format. Each volume consists of a matrix of voxels, each of which represents a small 3D unit of brain tissue. These voxels are similar to pixels in a 2D image, but they also possess a third dimension (depth). The typical representation of an sMRI volume's dimensions is (X, Y, Z), where X denotes the volume's number of voxels along the left-to-right (sagittal) axis, Y denotes its number of voxels along the anterior-posterior (coronal) axis, and Z denotes its number of voxels along the superior-inferior (axial) axis. By thinking of the 3D MRI volumes as frames in a video that come one after the other, we are able to use the temporal relationships between slices to show complex structural connections and possible changes caused by diseases. In this study, the recently introduced ViViT

[0027] , a transformer-based video encoder, is employed for video classification tasks. Four Vision Transformer variants suggested: spatio-temporal, factorized encoder, factorized self- attention, and factorized dot-product attention. We used spatio-temporal attention because it lets a model capture both spatial features (information in a single frame) and temporal dynamics (changes between frames). It helps the model learn and represent complex patterns and relationships in the data that span both space and time. ViViT employs a Tubelet embedding which refers to the process of extracting non- overlapping, spatio-temporal "tubes" from the input volume see Figure 13. This approach expands ViT's embedding to 3D, which aligns with a 3D convolution (Tubelet embedding). This technique corresponds to a 3D convolution and is an extension of ViT's embedding. Initially, volumes from the video are extracted. These volumes consist of patches of the frame along with temporal information. The volumes are flattened in order to create video tokens. These tokens are extracted from the temporal, height, and width dimensions of a tubelet with dimensions ^^ ൈ ℎ ൈ ^^. Reduced dimensions of tubelets consequently lead to a higher number of tokens, thereby augmenting computational requirements. This method incorporates spatio- temporal information into the tokenization process in an intuitive manner. Figure 13 show tubelet embedding: It involves the extraction and linear embedding of nonoverlapping tubelets that cover the entire spatio-temporal input volume. Docket No.2932719-000238-WO1 Figures 14A and 14B shows a pipeline proposed by ViTranZheimer. CAT12 was utilized to pre-process MRI images (image registration to standardize the images and skull stripping to reduce biases by ensuring consistent voxel intensities). ViViT was applied to whole frames to determine their characteristics and interdependencies. 3. Results The proposed framework is constructed using the Keras framework and the TensorFlow backend. On a workstation with an NVIDIA GeForce GTX 4080 GPU and 12 GB of RAM, all experiments were conducted. To test the proposed method, we conducted experiments on multi-classification tasks (CN, MCI, and AD). Error! Reference source not found. contains a list of the models' specifications. Adam Optimizer was utilized to train the ViViT model for a total of 1500 epochs with a batch size of 128 and a learning rate of 1e-4. The model has a total of 466,115 trainable parameters. The configuration of training parameters is summarized in Table 3. As the callback, the model checkpoint was configured to save only the optimal solution discovered during training, based on the loss function evaluation performed during validation. During training, the metric on a validation set is monitored, and the model checkpoint is only saved when the metric improves. The model checkpoint is saved whenever a metric improves on a validation set during training. Table 3 Details of model variants Parameter name Values Hyper parameters of ViViT Optimizer Adam Batch size 128 Epoch 1500 Input Shape (32, 64, 64, 1) Layer norm 1e-6 Learning rate 1e-4 Number of heads 16 Number of Layers 16 Patch size (32, 16, 16) Projection dim 32 The classification task was done on coronal plane. We implemented different baseline architectures to make comparisons with the proposed method. We compared the accuracy of our classification using CNN and Bi-LSTM, as well as ViT and Transformer. The CNN and ViT were utilized in order to derive the attributes of the T1-weighted MRI slices, and the Bi- LSTM model was utilized in order to classify the sequential features while maintaining the inter-association between the slices. The details of baseline model variants are listed in Error! Docket No.2932719-000238-WO1 Reference source not found.. Afterwards, we compared metrics like classification accuracy, model precision, recall, and F-score. Repeated 10-fold stratified cross-validation and testing were adopted for the evaluation of our model on the ADNI dataset. The cross-validation and test were done 10 times by moving the start subset of the cross-validation and test settings to the next subset. This way, each sample of all the datasets was only used for the test once. This approach offers several advantages, including the ability to reduce variance in performance estimates, utilize more training data, prevent overfitting, and ensure consistent, fair model evaluation across different machine learning models. The primary objective of this study is to conduct a thorough examination of the proposed methodology in relation to existing research. To this end, various models from the literature were compared with the proposed method. Table 4. Details of baseline model variants Parameter name Values CNN Model ResNet-101 Trained with ImageNet-1k (contains 1,200,000 images) at resolution 224x224. Params 44.7M Depth 209 Size of last layer (Feature extractor) 1 ൈ 2048 Bi-LSTM Layers 6 Hidden Units 64 Epoch 100 Batch Size 25 learning rate 1e-4 ^^ 0.9 Adam Optimizer^^^ଶ0.999 1e-07 Loss Function Sparse Categorical Cross entropy Dropout 0.15 Activation tanh Input Dimension 32 ൈ 2048 Output Dimension #classes (2 / 3) Params 538,515 Our findings indicate that our model demonstrated the highest level of accuracy in classifying cognitive status for individuals with NC, MCI, and AD in various clinical diagnosis tasks (see Table 5). context of multiclass disease classification on the ADNI1: Complete 3Yr 3T dataset, three distinct neural network architectures exhibited impressive performance. The CNN-Bi-LSTM model achieved an accuracy of 96.479%, with strong precision, recall, and F- score values of 0.96, indicating its proficiency in correctly classifying disease categories. ViT- Bi-LSTM surpassed this performance, achieving an accuracy of 97.465% and maintaining high Docket No.2932719-000238-WO1 precision, recall, and F-score values at 0.97. Notably, ViTranZheimer outperformed both models, with an exceptional accuracy of 98.6%, accompanied by precision, recall, and F-score metrics also at 0.97. These results show that deep learning models can accurately categorize diseases in the ADNI1 dataset, with ViTranZheimer having the highest accuracy. Table 5. Results for multiclass disease classification on ADNI1: Complete 3Yr 3T. Architecture ACC Precision Recall F-score CNN-Bi-LSTM 96.479% (±2.205) 0.96 0.96 0.96 ViT-Bi-LSTM 97.465% (±2.164) 0.97 0.97 0.97 ViTranZheimer 98.6% (+ / -1.4) 0.97 0.97 0.97 In Figure 15, we present the confusion matrices depicting the outcomes of our multiclass disease classification analysis. In particular, Figure 15 shows confusion matrices for multiclass disease classification (NC, MCI, and AD) on ADNI1: Complete 3Yr 3T. Note that the results of Figure 15 incorporate information from coronal, sagittal, and axial planes combined. In the AD class, ViTranZheimer attains a remarkable 97% true positive rate, correctly identifying 97% of AD cases, while maintaining a perfect 100% true negative rate for non-AD cases, showcasing its ability to effectively identify AD cases. ViT-Bi-LSTM also performs well for AD, achieving a true positive rate of 93%. CNN-Bi-LSTM, while decent, lags slightly behind with a truly positive rate of 93%. In the case of NC, ViTranZheimer demonstrates perfect sensitivity and specificity, correctly classifying all NC cases without any false positives or false negatives. ViT-Bi-LSTM and CNN-Bi-LSTM both exhibit excellent accuracy for NC and follow the ViTranZheimer with true positive rates of 99% and 97%, respectively. Regarding Mild Cognitive Impairment (MCI), it boasts a 98% true positive rate and a 99% true negative rate, with only 1% false positives and 1% false negatives. All three methods demonstrate strong performance, with ViTranZheimer and ViT-Bi-LSTM achieving a commendable 98% true positive rate, closely followed by CNN-Bi-LSTM with a slightly higher 99% true positive rate. Overall, ViTranZheimer excels in differentiating between AD, NC, and MCI, offering a promising diagnostic approach for Alzheimer's Disease utilizing video vision transformers and 3D brain MRI data." Docket No.2932719-000238-WO1 Additionally, we present a comparative analysis of our findings in relation to existing studies that have been reported in the literature. Table 6 presents the numerical outcomes pertaining to the multi-classification tasks conducted on the ADNI datasets. Table 6. Performance compared with state-of-the-art methods for multiclass disease classification. work Input Image scans Method NC / MCI / AD classification (%) (NC / MCI / AD ACC recall Precision SPE F1- ) Score

[0028] Slice based 300 / 300 / 300 VGGNet.16 91.85 - - - -

[0029] Voxel based 70 / 70 / 70 3D CNN 89.10 - - - -

[0030] Slice based 229 / 243 / 188 ResNet-18 56.80 - - - -

[0031] Slice based 169 / 234 / 101 2D CNN 96.0096.0 - 98.0 -

[0032] Voxel based 574 / 808 / 346 3D ResNet 87.00 - - - -

[0033] Voxel based 207 / 215 / 193 3D VGGNet 91.13 - - - -

[0034] Slice based 50 / 50 / 50 VGGNet-16 95.7396 96.33 - 95.66

[0035] Voxel based 351 / 297 / 221 3D DenseNets 97.5297 97.13 - 97.02

[0036] Patch based 475 / 224 / 70 3D CNN 97.4895.33 97.33 97.0

[0037] Slice based 25 / 13 / 25 ResNet18 & 98.2198.14 - 98.14 - DenseNet121 ProposedSequence-based 129 / 145 / 77 ViViT98.6 99.0 97.0 - 97.0Method Comparing our study to others was difficult due to the lack of precise data sets and training procedures. ADNI is useful for Alzheimer's disease research, but most researchers choose data based on personal preference and accordance. Most of them have unequal sample sizes and no data overlap. Additionally, most studies have used binary (CN / AD) classification. The biggest challenge is multi-classifying data samples to distinguish MCI from AD and NC. All Table 6 data were chosen for personal preference and accordance. The third column of these tables shows that not only the number of samples is not equal, but the data may also not overlap. To this end, this study used a medium-sized standard data collection named ADNI1: Complete 3Yr 3T that is publicly available.

[0005] Docket No.2932719-000238-WO1 4. Discussion CNN-BiLSTM models are a type of neural network that combines the strengths of CNNs and BiLSTM networks. ViT-BiLSTM, on the other hand, is a hybrid deep learning architecture that combines the strengths of ViTs and LSTM networks. CNN / ViT-BiLSTM models typically work by first using a CNN / ViT network to extract spatial features from each frame of a video. The extracted features are then fed into an LSTM network to learn the temporal dependencies between the frames. CNNs are good at capturing local spatial patterns in images, while LSTMs are good at capturing long-term temporal dependencies in videos. ViViT models, on the other hand, are a type of transformer architecture that was specifically designed for video processing. CNNs are famous for their ability to capture local patterns and features in images via convolutional layers, whereas ViTs have demonstrated promise in capturing global context and relationships in images via self-attention mechanisms. Combining these attributes may result in enhanced performance in computer vision tasks. Also, using 3CNN can handle the long sequences of frames (slices) encountered in video (3D-MRI). This model is end-to-end in the sense that it can take raw video data as input and produce the desired output directly, without the need for any intermediate steps or preprocessing. This is in contrast to traditional video processing pipelines, which typically involve multiple separate stages, such as feature extraction, tracking, and classification. CNN / ViT-BiLSTM models are computationally expensive to train and deploy due to the need for two separate neural networks, while ViViT models are more efficient due to their single neural network. They are also difficult to optimize due to the joint training and optimization of the two networks. However, CNN / ViT-BiLSTM models may be less accurate on some video processing tasks due to their specificity for video processing, while ViViT models are designed for general-purpose architecture. The CNN / ViT models were trained for 100 epochs, with each individual slice undergoing 100 epochs of training. When considering the selection of 32 slices from the middle of each MRI, this amounted to a cumulative 3200 epochs for each MRI. In contrast, the ViViT model underwent a training process spanning 1500 epochs for an MRI. ViTranZheimer outperformed the CNN-Bi-LSTM and ViT-Bi-LSTM models in Alzheimer's Disease classification accuracy, with a significant improvement of 98.6%. This improvement surpasses the CNN-Bi-LSTM model by 2.121 percentage points and the ViT-Bi- Docket No.2932719-000238-WO1 LSTM model by 1.135 percentage points, indicating a substantial enhancement in the accuracy of the ViTranZheimer model. To ensure that the comparison of methods is fair, it is important to use a common dataset. Most researchers choose data based on personal preference and availability. In this study, we used a standard data collection named ADNI1: Complete 3Yr 3T, which is publicly available. This allows other researchers to fairly compare their methods with ours using the same data collection. It is worth emphasizing that AD primarily affects the hippocampus, making it a crucial focus in diagnosis and monitoring. The hippocampus is one of the earliest and most vulnerable brain regions affected by AD pathology, making it a key biomarker for detection and progression tracking. The accuracy and reliability of hippocampus detection depends on successful segmentation, which defines hippocampal region boundaries in MRI images, identifying the region of interest for further analysis. 5. Conclusions This study introduces 'ViTranZheimer,' a novel deep learning approach for AD classification that makes use of video vision transformers and 3D brain MRI data. To capture complex structural relationships, we model the 3D MRI volumes as videos and take advantage of the temporal dependencies between slices. Our research demonstrates the potential of deep learning models for AD diagnosis, with an emphasis on multi-class classification tasks involving NC, MCI, and AD cases. We investigated and compared various neural network architectures, such as CNN-BiLSTM, ViT-BiLSTM, and ViTranZheimer, revealing significant improvements in AD classification accuracy. The proposed approaches are thoroughly validated on ADNI dataset and compared to other relevant models. ViTranZheimer emerged as the top performer, achieving a remarkable 98.6% accuracy rate, demonstrating its diagnostic potential. The findings demonstrated that the proposed model exhibited superior performance and accuracy compared to alternative models, with a statistically significant advantage in terms of accuracy. Future endeavors focus on MRI harmonization and ViTranZheimer to enhance AD diagnosis, enhancing accuracy and sensitivity, and fostering a comprehensive neuroimaging framework. In comparison to a Video Vision Transformer (ViT), the embodiment described above uses time series classification. This model leverages the transformer architecture specifically designed to handle sequential time series data, enabling effective classification tasks that the original ViT, which was focused on image data, does not address. Docket No.2932719-000238-WO1 References [1] N. Rahim, S. El-Sappagh, S. Ali, K. Muhammad, J. Del Ser, T. Abuhmed, Prediction of Alzheimer’s progression based on multimodal Deep-Learning-based fusion and visual Explainability of time-series data, Information Fusion.92 (2023) 363–388. https: / / doi.org / 10.1016 / J.INFFUS.2022.11.028. [2] Y. Zhang, S. Wang, K. Xia, Y. Jiang, P. Qian, Alzheimer’s disease multiclass diagnosis via multimodal neuroimaging embedding feature selection and fusion, Information Fusion. 66 (2021) 170–183. https: / / doi.org / 10.1016 / J.INFFUS.2020.09.002. [3] D. Castillo-Barnes, L. Su, J. Ramírez, D. Salas-Gonzalez, F.J. Martinez-Murcia, I.A. Illan, F. Segovia, A. Ortiz, C. Cruchaga, M.R. Farlow, C. Xiong, N.R. Graff-Radford, P.R. Schofield, C.L. Masters, S. Salloway, M. Jucker, H. Mori, J. Levin, J.M. Gorriz, D.I.A.N. (DIAN), Autosomal dominantly inherited alzheimer disease: Analysis of genetic subgroups by machine learning, Information Fusion. 58 (2020) 153–167. https: / / doi.org / 10.1016 / J.INFFUS.2020.01.001. [4] J. Albright, Forecasting the progression of Alzheimer’s disease using neural networks and a novel preprocessing algorithm, Alzheimer’s & Dementia: Translational Research & Clinical Interventions. 5 (2019) 483–491. https: / / doi.org / 10.1016 / J.TRCI.2019.07.001. [5] A. Loddo, S. Buttau, C. Di Ruberto, Deep learning based pipelines for Alzheimer’s disease diagnosis: A comparative study and a novel deep-ensemble method, Comput Biol Med. 141 (2022) 105032. https: / / doi.org / 10.1016 / J.COMPBIOMED.2021.105032. [6] F.J. Martinez-Murcia, J.M. Górriz, J. Ramírez, A. Ortiz, A Structural Parametrization of the Brain Using Hidden Markov Models-Based Paths in Alzheimer’s Disease, Https: / / Doi.Org / 10.1142 / S0129065716500246.26 (2016). https: / / doi.org / 10.1142 / S0129065716500246. [7] A. Abrol, M. Bhattarai, A. Fedorov, Y. Du, S. Plis, V. Calhoun, Deep residual learning for neuroimaging: An application to predict progression to Alzheimer’s disease, J Neurosci Methods. 339 (2020) 108701. https: / / doi.org / 10.1016 / J.JNEUMETH.2020.108701. [8] P.K. Mall, P.K. Singh, S. Srivastav, V. Narayan, M. Paprzycki, T. Jaworska, M. Ganzha, A comprehensive review of deep neural networks for medical image processing: Recent developments and future opportunities, Healthcare Analytics.4 (2023) 100216. https: / / doi.org / 10.1016 / J.HEALTH.2023.100216. [9] B.A. Jonsson, G. Bjornsdottir, T.E. Thorgeirsson, L.M. Ellingsen, G.B. Walters, D.F. Gudbjartsson, H. Stefansson, K. Stefansson, M.O. Ulfarsson, Brain age prediction using deep learning uncovers associated sequence variants, Nature Communications 201910:1.10 (2019) 1–10. https: / / doi.org / 10.1038 / s41467- 019-13163-9.

[0010] A. Boutet, R. Madhavan, G.J.B. Elias, S.E. Joel, R. Gramer, M. Ranjan, V. Paramanandam, D. Xu, J. Germann, A. Loh, S.K. Kalia, M. Hodaie, B. Li, S. Prasad, A. Coblentz, R.P. Munhoz, J. Ashe, W. Kucharczyk, A. Fasano, A.M. Lozano, Predicting optimal deep brain stimulation parameters for Parkinson’s disease using functional MRI and machine learning, Nature Communications 202112:1.12 (2021) 1–13. https: / / doi.org / 10.1038 / s41467-021-23311-9. Docket No.2932719-000238-WO1

[0011] B. Zhao, H. Lu, S. Chen, J. Liu, D. Wu, Convolutional neural networks for time series classification, Journal of Systems Engineering and Electronics. 28 (2017) 162–169. https: / / doi.org / 10.21629 / JSEE.2017.01.18.

[0012] L. Yue, X. Gong, J. Li, H. Ji, M. Li, A.K. Nandi, Hierarchical feature extraction for early Alzheimer’s disease diagnosis, IEEE Access.7 (2019) 93752–93760. https: / / doi.org / 10.1109 / ACCESS.2019.2926288.

[0013] I.R.R. Silva, G.S.L. Silva, R.G. De Souza, W.P. Dos Santos, R.A.A. De Fagundes, Model Based on Deep Feature Extraction for Diagnosis of Alzheimer’s Disease, Proceedings of the International Joint Conference on Neural Networks.2019-July (2019). https: / / doi.org / 10.1109 / IJCNN.2019.8852138.

[0014] F. Zhang, Z. Li, B. Zhang, H. Du, B. Wang, X. Zhang, Multi-modal deep learning model for auxiliary diagnosis of Alzheimer’s disease, Neurocomputing. 361 (2019) 185–195. https: / / doi.org / 10.1016 / J.NEUCOM.2019.04.093.

[0015] Q. Wen, T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan, L. Sun, Transformers in Time Series: A Survey, (2022). https: / / doi.org / 10.48550 / arxiv.2202.07125.

[0016] A. Krizhevsky, I. Sutskever, G.E. Hinton, ImageNet classification with deep convolutional neural networks, Commun ACM.60 (2017) 84–90. https: / / doi.org / 10.1145 / 3065386.

[0017] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You Only Look Once: Unified, Real-Time Object Detection, (2016) 779–788.

[0018] N. Wojke, A. Bewley, D. Paulus, Simple online and realtime tracking with a deep association metric, Proceedings - International Conference on Image Processing, ICIP.2017-September (2018) 3645–3649. https: / / doi.org / 10.1109 / ICIP.2017.8296962.

[0019] O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for biomedical image segmentation, Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 9351 (2015) 234–241. https: / / doi.org / 10.1007 / 978-3-319-24574- 4_28 / COVER.

[0020] J. Jang, D. Hwang, M3T: Three-Dimensional Medical Image Classifier Using Multi-Plane and Multi- Slice Transformer, (2022) 20718–20729.

[0021] K.A.N.N.P. Gunawardena, R.N. Rajapakse, N.D. Kodikara, Applying convolutional neural networks for pre-detection of Alzheimer’s disease from structural MRI data, 201724th International Conference on Mechatronics and Machine Vision in Practice, M2VIP 2017. 2017-December (2017) 1–7. https: / / doi.org / 10.1109 / M2VIP.2017.8211486.

[0022] H. Choi, K.H. Jin, Predicting cognitive decline with deep learning of brain metabolism and amyloid imaging, Behavioural Brain Research.344 (2018) 103–109. https: / / doi.org / 10.1016 / J.BBR.2018.02.017.

[0023] A. Ebrahimi, S. Luo, R. Chiong, Deep sequence modelling for Alzheimer’s disease detection using MRI, Comput Biol Med.134 (2021) 104537. https: / / doi.org / 10.1016 / J.COMPBIOMED.2021.104537.

[0024] ADNI | Alzheimer’s Disease Neuroimaging Initiative, (n.d.). https: / / adni.loni.usc.edu / (accessed April 3, 2023). Docket No.2932719-000238-WO1

[0025] C.R. Jack, M.A. Bernstein, N.C. Fox, P. Thompson, G. Alexander, D. Harvey, B. Borowski, P.J. Britson, J.L. Whitwell, C. Ward, A.M. Dale, J.P. Felmlee, J.L. Gunter, D.L.G. Hill, R. Killiany, N. Schuff, S. Fox- Bosetti, C. Lin, C. Studholme, C.S. DeCarli, G. Krueger, H.A. Ward, G.J. Metzger, K.T. Scott, R. Mallozzi, D. Blezek, J. Levy, J.P. Debbins, A.S. Fleisher, M. Albert, R. Green, G. Bartzokis, G. Glover, J. Mugler, M.W. Weiner, The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods, Journal of Magnetic Resonance Imaging.27 (2008) 685–691. https: / / doi.org / 10.1002 / JMRI.21049.

[0026] J. Ashburner, K.J. Friston, Unified segmentation, Neuroimage. 26 (2005) 839–851. https: / / doi.org / 10.1016 / J.NEUROIMAGE.2005.02.018.

[0027] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M.L. Luči´c, C. Schmid, ViViT: A Video Vision Transformer, (n.d.).

[0028] C.D. Billones, O.J.L.D. Demetria, D.E.D. Hostallero, P.C. Naval, DemNet: A Convolutional Neural Network for the detection of Alzheimer’s Disease and Mild Cognitive Impairment, IEEE Region 10 Annual International Conference, Proceedings / TENCON. (2017) 3724–3727. https: / / doi.org / 10.1109 / TENCON.2016.7848755.

[0029] E. Hosseini-Asl, … R.K.-2016 I. international, undefined 2016, Alzheimer’s disease diagnostics by adaptation of 3D convolutional network, Ieeexplore.Ieee.Org. (n.d.). https: / / ieeexplore.ieee.org / abstract / document / 7532332 / ?casa_token=Neb5n7ikTZMAAAAA:PkEGLJT 7qw9U49OS9KRibb0AFV1ImpMxt_SViSMvquUaRK5BjceVLgbe3YznJAG0Tw20np3KSMNT0Pk (accessed March 29, 2023).

[0030] A. Valliani, A.S.-P. of the 8th A. international conference, undefined 2017, Deep residual nets for improved Alzheimer’s diagnosis, Dl.Acm.Org. (2017) 615. https: / / doi.org / 10.1145 / 3107411.3108224.

[0031] K. Gunawardena, … R.R.-… on M. and, undefined 2017, Applying convolutional neural networks for pre-detection of alzheimer’s disease from structural MRI data, Ieeexplore.Ieee.Org. (n.d.). https: / / ieeexplore.ieee.org / abstract / document / 8211486 / ?casa_token=0Vm5OBjwvlYAAAAA:PKVbNM AMIsDz-HLaNNWN_khu_UcFL6wO3FEtTxOxVE5tuCet49yXSNB-smS9lU5C5knfNN1GTZT9RWs (accessed March 29, 2023).

[0032] H. Karasawa, C.L. Liu, H. Ohwada, Deep 3D Convolutional Neural Network Architectures for Alzheimer’s Disease Diagnosis, Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 10751 LNAI (2018) 287–296. https: / / doi.org / 10.1007 / 978-3-319-75417-8_27.

[0033] T.D. Vu, N.H. Ho, H.J. Yang, J. Kim, H.C. Song, Non-white matter tissue extraction and deep convolutional neural network for Alzheimer’s disease detection, Soft Comput. 22 (2018) 6825–6833. https: / / doi.org / 10.1007 / S00500-018-3421-5.

[0034] R. Jain, N. Jain, A. Aggarwal, D.H.-C.S. Research, undefined 2019, Convolutional neural network based Alzheimer’s disease classification from magnetic resonance brain images, Elsevier. (n.d.). https: / / www.sciencedirect.com / science / article / pii / S1389041718309562 (accessed March 29, 2023). Docket No.2932719-000238-WO1

[0035] H. Wang, Y. Shen, S. Wang, T. Xiao, L. Deng, X. Wang, X. Zhao, Ensemble of 3D densely connected convolutional network for diagnosis of mild cognitive impairment and Alzheimer’s disease, Neurocomputing.333 (2019) 145–156. https: / / doi.org / 10.1016 / J.NEUCOM.2018.12.018.

[0036] N. Goenka, S. Tiwari, AlzVNet: A volumetric convolutional neural network for multiclass classification of Alzheimer’s disease through multiple neuroimaging computational approaches, Biomed Signal Process Control.74 (2022) 103500. https: / / doi.org / 10.1016 / J.BSPC.2022.103500.

[0037] M. Odusami, R. Maskeliūnas, R. Damaševičius, An Intelligent System for Early Recognition of Alzheimer’s Disease Using Neuroimaging, Sensors 2022, Vol. 22, Page 740. 22 (2022) 740. https: / / doi.org / 10.3390 / S22030740. Supplemental to Example 2 1. Introduction Approaches based on deep convolutional neural networks have advanced the state-of- the-art across many standard datasets for vision problems since AlexNet

[0038] . At the same time, the most prominent architecture of choice in sequence-to-sequence modelling (e.g. in natural language processing) is the transformer

[0068] , which does not use convolutions, but is based on multi-headed self-attention. This operation is particularly effective at modelling long-range dependencies and allows the model to attend over all elements in the input sequence. This is in stark contrast to convolutions where the corresponding “receptive field” is limited, and grows linearly with the depth of the network. The success of attention-based models in NLP has recently inspired approaches in computer vision to integrate transformers into CNNs [75, 7], as well as some attempts to replace convolutions completely [49, 3, 53]. However, it is only very recently with the Vision Transformer (ViT)

[0018] , that a pure-transformer based architecture has outperformed its convolutional counterparts in image classification. Dosovitskiy et al.

[0018] closely followed the original transformer architecture of

[0068] , and noticed that its main benefits were observed at large scale – as transformers lack some of the inductive biases of convolutions (such as translational equivariance), they seem to require more data

[0018] or stronger regularization

[0064] . Inspired by ViT, and the fact that attention-based architectures are an intuitive choice for modelling long range contextual relationships in video, we develop several transformer- based models for video classification. Currently, the most performant models are based on deep 3D convolutional architectures [8, 20, 21] which were a natural extension of image classification CNNs [27, 60]. Recently, these models were augmented by incorporating self attention into their later layers to better capture long-range dependencies [75, 23, 79, 1]. Docket No.2932719-000238-WO1 As shown in Figure 16, we propose pure-transformer models for video classification. The main operation performed in this architecture is self-attention, and it is computed on a sequence of spatio-temporal tokens that we extract from the input video. To effectively process the large number of spatio-temporal tokens that may be encountered in video, we present several methods of factorizing our model along spatial and temporal dimensions to increase efficiency and scalability. Furthermore, to train our model effectively on smaller datasets, we show how to regularize our model during training and leverage pretrained image models. We also note that convolutional models have been developed by the community for several years, and there are thus many “best practices” associated with such models. As pure- transformer models present different characteristics, we need to determine the best design choices for such architectures. We conduct a thorough ablation analysis of tokenization strategies, model architecture and regularization methods. Informed by this analysis, we achieve state of- the-art results on multiple standard video classification benchmarks, including Kinetics 400 and 600

[0035] , Epic Kitchens 100

[0013] , Something-Something v2

[0026] and Moments in Time

[0045] . Figure 16 proposes a pure-transformer architecture for video classification, inspired by the recent success of such models for images

[0018] . To effectively process a large number of spatio-temporal tokens, we develop several model variants which factorize different components of the transformer encoder over the spatial- and temporal-dimensions. As shown on the right, these factorizations correspond to different attention patterns over space and time. 2. Related Work Architectures for video understanding have mirrored advances in image recognition. Early video research used hand-crafted features to encode appearance and motion information [41, 69]. The success of AlexNet on ImageNet [38, 16] initially led to the repurposing of 2D image convolutional networks (CNNs) for video as “two stream” networks [34, 56, 47]. These models processed RGB frames and optical flow images independently before fusing them at the end. Availability of larger video classification datasets such as Kinetics

[0035] subsequently facilitated the training of spatio-temporal 3D CNNs [8, 22, 65] which have significantly more parameters and thus require larger training datasets. As 3D convolutional networks require significantly more computation than their image counterparts, many architectures factorize convolutions across spatial and temporal dimensions and / or use grouped convolutions [59, 66, 67, 81, 20]. We also leverage factorization of the spatial and temporal dimensions of videos to increase efficiency, but in the context of transformer-based models. Docket No.2932719-000238-WO1 Concurrently, in natural language processing (NLP), Vaswani et al.

[0068] achieved state- of-the-art results by replacing convolutions and recurrent networks with the transformer network that consisted only of self-attention, layer normalization and multilayer perceptron (MLP) operations. Current state-of-the-art architectures in NLP [17, 52] remain transformer- based, and have been scaled to web-scale datasets [5]. Many variants of the transformer have also been proposed to reduce the computational cost of self attention when processing longer sequences [10, 11, 37, 62, 63, 73] and to improve parameter efficiency [40, 14]. Although self- attention has been employed extensively in computer vision, it has, in contrast, been typically incorporated as a layer at the end or in the later stages of the network [75, 7, 32, 77, 83] or to augment residual blocks [30, 6, 9, 57] within a ResNet architecture

[0027] . Although previous works attempted to replace convolutions in vision architectures [49, 53, 55], it is only very recently that Dosovitisky et al.

[0018] showed with their ViT architecture that pure-transformer networks, similar to those employed in NLP, can achieve state-of-the-art results for image classification too. The authors showed that such models are only effective at large scale, as transformers lack some of inductive biases of convolutional networks (such as translational equivariance), and thus require datasets larger than the common ImageNet ILSRVC dataset

[0016] to train. ViT has inspired a large amount of follow-up work in the community, and we note that there are a number of concurrent approaches on extending it to other tasks in computer vision [71, 74, 84, 85] and improving its data efficiency [64, 48]. In particular, [4, 46] have also proposed transformer-based models for video. In this paper, we develop pure-transformer architectures for video classification. We propose several variants of our model, including those that are more efficient by factorizing the spatial and temporal dimensions of the input video. We also show how additional regularization and pretrained models can be used to combat the fact that video datasets are not as large as their image counterparts that ViT was originally trained on. Furthermore, we outperform the state of- the-art across five popular datasets. 3. Video Vision Transformers We start by summarizing the recently proposed Vision Transformer

[0018] in Sec. 3.1, and then discuss two approaches for extracting tokens from video in Sec. 3.2. Finally, we develop several transformer-based architectures for video classification in Sec.3.3 and 3.4. 3.1. Overview of Vision Transformers (ViT) Vision Transformer (ViT)

[0018] adapts the transformer architecture of

[0068] to process 2D images with minimal changes. In particular, ViT extracts ^^ non-overlapping image patches, Docket No.2932719-000238-WO1 ^^^∈ ℝ^ൈ௪, performs a linear projection and then rasterize them into 1D tokens ^^^∈ ℝௗ. Thesequence of tokens input to the following transformer encoder is^^ ൌ ^ ^^^^^; ^^ ^^^, ^^ ^^ଶ, … , ^^ ^^ே^ ^ p; (1)where the projection by E is equivalent to a 2D convolution. As shown in Figure 16, an optional learned classification token ^^^^^is prepended to this sequence, and its representation at the final layer of the encoder serves as the final representation used by the classification layer

[0017] . In addition, a learned positional embedding, p ∈ ℝேൈௗ, is added to the tokens to retain positional information, as the subsequent self-attention operations in the transformer are permutation invariant. The tokens are then passed through an encoder consisting of a sequence of ^^ transformer layers. Each layer ℓ comprises of Multi-Headed Self-Attention

[0068] , layer normalisation (LN) [2], and MLP blocks as follows: ^^ℓൌ MSA^LN^zℓ^^ ^ zℓ(2) ^^ℓା^^ ^^ℓ(3) The MLP consists of two the token-dimensionality, ^^, a to classify the encoded input based on ^^^^^^∈ ℝௗ, if it was prepended to the input, or a global average pooling of all the tokens, ^^^, As the transformer

[0068] , which forms the basis of ViT

[0018] , is a flexible architecture that can operate on any sequence of input tokens ^^ ∈ ℝேൈௗ, we describe strategies for tokenizing videos next. 3.2. Embedding video clips We consider two simple methods for mapping a video ^^ ∈ ℝ்ൈுൈ^ൈ^to a sequence of tokens ^^^ ∈ ℝ^^ൈ^^ൈ^^ൈௗ. We then add the positional embedding and reshape into ℝேൈௗto obtain ^^, the input to the transformer. Uniform frame sampling As illustrated in Figure 17, a straightforward method of tokenizing the input video is to uniformly sample ^^௧frames from the input video clip, embed each 2D frame independently using the same method as ViT

[0018] , and concatenate all these tokens together. Concretely, if ^^^∙ ^^௪non-overlapping image patches are extracted from each frame, as in

[0018] , then a total of ^^௧∙ ^^^∙ ^^௪tokens will be forwarded through the transformer encoder. Intuitively, this process may be seen as simply constructing a large 2D image to be tokenized following ViT. We note that this is the input embedding method employed by the concurrent work of [4]. Docket No.2932719-000238-WO1 Figure 17 shows uniform frame sampling: We simply sample ^^௧frames, and embed each 2D frame independently following ViT

[0018] . Figure 18 shows Tubelet nonoverlapping tubelets that span the spatio- Tubelet embedding An alternate method, as shown in Figure 18, is to extract non- overlapping, spatio-temporal “tubes” from the input volume, and to linearly project this to ℝௗ. This method is an extension of ViT’s embedding to 3D, and corresponds to a 3D convolution.For a tubelet of dimension ^^ ൈ ℎ ൈ ^^, ^^ ൌ^்^, ^^ ு ^௧௧ ^ൌ^^^and ^^௪ൌ^௪^, tokens are extractedfrom the temporal, height, and width dimensions thus result in more tokens which increases the computation. Intuitively, this method fuses spatio- temporal information during tokenization, in contrast to “Uniform frame sampling” where temporal information from different frames is fused by the transformer. 3.3. Transformer Models for Video As illustrated in Figure 16, we propose multiple transformer-based architectures. We begin with a straightforward extension of ViT

[0018] that models pairwise interactions between all spatio-temporal tokens, and then develop more efficient variants which factorize the spatial and temporal dimensions of the input video at various levels of the transformer architecture. Model 1: Spatio-temporal attention This model simply forwards all spatio-temporal tokens extracted from the video, ^^^, through the transformer encoder. We note that this has also been explored concurrently by [4] in their “Joint Space-Time” model. In contrast to CNN architectures, where the receptive field grows linearly with the number of layers, each transformer layer models all pair wise interactions between all spatio-temporal tokens, and it thus models long-range interactions across the video from the first layer. However, as it models all pairwise interactions, Multi-Headed Self Attention (MSA)

[0068] has quadratic complexity with respect to the number of tokens. This complexity is pertinent for video, as the number of tokens increases linearly with the number of input frames, and motivates the development of more efficient architectures next. Docket No.2932719-000238-WO1 Figure 19 model consists of two transformer encoders in tokens extracted from the same temporal index to produce a latent representation per time-index. The second transformer models interactions between time steps. It thus corresponds to a “late fusion” of spatial- and temporal information. Model 2: Factorized encoder As shown in Figure 19, this model consists of two separate transformer encoders. The first, spatial encoder, only models interactions betweentokens extracted from the same temporal index. A representation for each temporal index, ℎ^ ∈ℝௗ, is obtained after ^^ layers: This is the encoded c ^ೞ^ lassification token, ^^^^^ if it wasprepended to the input (Eq.1), or a global average pooling from the tokens output by the spatialencoder, ^^^ೞ, otherwise. The frame-level representations, ℎ^, are concatenated into ^^ ∈ℝ^^ଡ଼ௗ, and then forwarded through a temporal encoder consisting of ^^௧ transformer layers tomodel interactions between tokens from different temporal indices. The output token of this encoder is then finally classified. This architecture corresponds to a “late fusion” [34, 56, 72, 46] of temporal information, and the initial spatial encoder is identical to the one used for image classification. It is thus analogous to CNN architectures such as [24, 34, 72, 86] which first extract per-frame features, and then aggregate them into a final representation before classifying them. Although this model has more transformer layers than Model 1 (and thus more parameters), it requires fewer floating point operations (FLOPs), as the two separate transformer blocks have a complexity of ^^^^ ^^^∙ ^^^^^2^ ^^2^^^ compared to ^^^^ ^^௧∙ ^^ℎ∙ ^^^^^2^ of Model 1. Docket No.2932719-000238-WO1 Figure 20 shows factorized self-attention (Model 3). Within each transformer block, the multi- headed self-attention operation is factorized into two operations (indicated by striped boxes) that first only compute self-attention spatially, and then temporally. Model 3: Factorized self-attention This model, in contrast, contains the same number of transformer layers as Model 1. However, instead of computing multi-headed self-attention across all pairs of tokens, zℓ, at layer Ɩ, we factorize the operation to first only compute self- attention spatially (among all tokens extracted from the same temporal index), and then temporally (among all tokens extracted from the same spatial index) as shown in Figure 20. Each self-attention block in the transformer thus models spatio-temporal interactions, but does so more efficiently than Model 1 by factorizing the operation over two smaller sets of elements, thus achieving the same computational complexity as Model 2. We note that factorizing attention over input dimensions has also been explored in [29, 78], and concurrently in the context of video by [4] in their “Divided Space-Time” model. This operation can be performed efficiently by reshaping the tokens z from ℝ^ൈ^^∙^^∙^^∙ௗto ℝ^^ൈ^^∙^^∙ௗ(denoted by z^) to compute spatial self-attention. Similarly, the input to z is reshap^^∙^^ൈ^^∙ௗ ௧ed to ℝ . Here we assume the leading dimension is the “batch dimension”. Our factorized self-attention is defined as ^^^ℓൌ MSA^LN^z^ℓ^^ ^ z^ℓ(4) (5) (6) We observed that spatial self-attention does not make a difference, provided that the model parameters are initialized as described in Sec. 3.4. Note that the number of parameters, however, increases compared to Model 1, as there is an additional self-attention layer (cf. Eq.7). We do not use a classification token in this model, to avoid ambiguities when reshaping the input tokens between spatial and temporal dimensions. Figure 21 shows . For half of the heads, we compute dot-product attention over only the spatial axes, and for the other half, over only the temporal axis. Docket No.2932719-000238-WO1 Model 4: Factorized dot-product attention Finally, we develop a model which has the same computational complexity as Models 2 and 3, while retaining the same number of parameters as the unfactorized Model 1. The factorization of spatial- and temporal dimensions is similar in spirit to Model 3, but we factorize the multi-head dot-product attention operation instead (Figure 21). Concretely, we compute attention weights for each token separately over the spatial and temporal-dimensions using different heads. First, we note that the attentionoperation for each head is defined astention ^^, ^^, ^^ ൌ Softmax ൬ ^^ ^ ^At ^ ^ ^^ௗೖ^V. (7)In self-attention, the queries ^^ ൌ ^^ ^^^, keys ^^ ൌ ^^ ேൈௗ projections of the input X with X, Q, K, V ∈ ℝ . Note case 1), the spatial and temporal dimensions are merged as ^^ ൌ ^^௧∙ ^^^∙ ^^௪. The main idea here is to modify the keys and values for each query to only attend over tokensfrom the same spatial and temporal index by constructing K ^^, V^ ∈ ℝ ^∙^^ൈௗ and K௧ , V௧ ∈ℝ^^ൈௗ, namely the keys and values corresponding to these dimensions. of the attention heads, we attend over tokens from the spatial dimension by computing Y^ ൌAttention^Q, K^, V^^, and for the rest we attend over the temporal dimension by computingY௧ ൌ Attention^Q, K௧ , V௧^. Given that we are only changing the attention neighborhood foreach query, the attention operation has the same dimension as in the unfactorized case, namelyY^, Y௧ ∈ ℝேൈௗ. We then combine the outputs of multiple heads by concatenating them andusing a linear projection

[0068] , ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^ ^^^, ^^௧^ ^^^. 3.4. Initialization by leveraging pretrained models ViT

[0018] has been shown to only be effective when trained on large-scale datasets, as transformers lack some of the inductive biases of convolutional networks

[0018] . However, even the largest video datasets such as Kinetics

[0035] , have several orders of magnitude less labelled examples when compared to their image counterparts [16, 39, 58]. As a result, training large models from scratch to high accuracy is extremely challenging. To sidestep this issue, and enable more efficient training we initialize our video models from pretrained image models. However, this raises several practical questions, specifically on how to initialize parameters not present or incompatible with image models. We now discuss several effective strategies to initialize these largescale video classification models. Positional embeddings A positional embedding p is added to each input token (Eq.1). However, our video models have ^^௧times more tokens than the pretrained image model. As a result, we initialize the positional embeddings by “repeating” them temporally from ℝ^^∙^^ൈௗ Docket No.2932719-000238-WO1 to ℝ^^∙^^∙^^ൈௗ. Therefore, at initialisation, all tokens with the same spatial index have the same embedding which is then fine-tuned. weights, E When using the “tubelet embedding” tokenization method (Sec. 3.2), the embedding filter E is a 3D tensor, compared to the 2D tensor in the pretrained model, ^^୧୫ୟ^^A common approach for initialising 3D convolutional filters from 2D filters for video classification is to “inflate” them by replicating the filters along the temporal dimension and averaging them [8, 22] asEൌ ^௧ ^E୧୫ୟ^^, … , E୧୫ୟ^^, … , E୧୫ୟ^^൧. (8)We consider an additional strategy, which we denote as “central frame initialization”, where E is initialized with zeroes along all temporal positions, except at the center ^௧ଶ^,Eൌ ^O, … , E୧୫ୟ^^, … , O൧. (9)Therefore, the 3D convolutional filter effectively behaves like “Uniform frame sampling” (Sec. 3.2) at initialization, while also enabling the model to learn to aggregate temporal information from multiple frames as training progresses. Transformer weights for Model 3 The transformer block in Model 3 (Figure 20) differs from the pretrained ViT model

[0018] , in that it contains two multi-headed self attention (MSA) modules. In this case, we initialize the spatial MSA module from the pretrained module, and initialize all weights of the temporal MSA with zeroes, such that Eq.5 behaves as a residual connection

[0027] at initialization.

Claims

Docket No.2932719-000238-WO1 CLAIMS 1. A method comprising, receiving first magnetic resonance imaging (MRI) data of a first plurality of subjects, wherein the first MRI image data comprises a first plurality of two-dimensional (2D) images; pre-processing the first MRI data for analysis; converting each two-dimensional image of the first plurality of 2D images into first tokens as input for a transformer encoder, wherein the transformer encoder comprises a time series classification transformer; training the transformer encoder using the first input tokens; receiving second MRI data of a subject, wherein the second MRI data comprises a second plurality of 2D images; pre-processing the second MRI data for analysis; converting each image of the second plurality of 2D images into second tokens as input to the trained transformer encoder; applying the trained transformer encoder to the second tokens to predict a state of disease in a subject.

2. The method of claim 1, wherein the converting the first and second plurality of 2D images comprises converting each image into 224x224 pixel images.

3. The method of claim 2, wherein each 224x224 pixel image is divided into 14x14 patches, wherein each patch comprises a 16x16 pixel image.

4. The method of claim 3, wherein the converting the first and second plurality of 2D images comprises linearly projecting each patch into a vector.

5. The method of claim 4, wherein the converting comprises adding positional embeddings to the first and second tokens.

6. The method of claim 1, wherein the preprocessing comprises standardizing the first and second MRI data in Montreal Neurological Institute space.

7. The method of claim 6, wherein the preprocessing comprises stripping skull image data from the first and second MRI data using statistical parametric mapping 12.Docket No.2932719-000238-WO1 8. The method of claim 1, wherein the first and second plurality of 2D images lie in the coronal plane.

9. The method of claim 1, wherein the first and second plurality of 2D images lie in the sagittal plane.

10. The method of claim 1, wherein the first and second plurality of 2D images lie in the axial plane.

11. The method of claim 1, wherein the transformer encoder comprises a multi-head attention layer.

12. The method of claim 11, wherein the transformer encoder comprises a feed forward neural network layer.

13. The method of claim 12, wherein output of the multi-head attention layer is fed into the feed forward neural network layer.

14. The method of claim 13, wherein the trained transformer encoder comprises a multilayer perceptron head for predicting the state of disease in the subject.

15. The method of claim 14, wherein the state of disease comprises a state of Alzheimer’s disease.

16. A method comprising, receiving first magnetic resonance imaging (MRI) data of a first plurality of subjects, wherein the first MRI image data comprises a first plurality of two-dimensional (2D) images, wherein each image corresponds to a time in a series; pre-processing the first MRI data for analysis; converting the two dimensional image data and corresponding time of each image of the first plurality of 2D images into first tokens as input for a transformer encoder, wherein the transformer encoder comprises a time series classification transformer; training the transformer encoder using the first input tokens;Docket No.2932719-000238-WO1 receiving second MRI data of a subject, wherein the second MRI data comprises a second plurality of 2D images, wherein each image corresponds to a time in a series; pre-processing the second MRI data for analysis; converting the two dimensional image data and corresponding time of each image of the second plurality of 2D images into second tokens as input to the trained transformer encoder; applying the trained transformer encoder to the second tokens to predict a state of disease in a subject.

17. The method of claim 16, wherein the converting the first and second plurality of 2D images comprises linearly projecting the two dimensional image data and corresponding time of each image into a vector.

18. The method of claim 17, wherein the converting comprises adding positional embeddings to the first and second tokens.

19. The method of claim 1, wherein the preprocessing comprises standardizing the first and second MRI data in Montreal Neurological Institute space.

20. The method of claim 19, wherein the preprocessing comprises stripping skull image data from the first and second MRI data using statistical parametric mapping 12.

21. The method of claim 16, wherein the first and second plurality of 2D images comprise a series of parallel images in the coronal plane.

22. The method of claim 16, wherein the first and second plurality of 2D images comprise a series of parallel images in the sagittal plane. 23 The method of claim 16, wherein the first and second plurality of 2D images comprise a series of parallel images in the axial plane.

24. The method of claim 16, wherein the transformer encoder comprises a multi-head attention layer.Docket No.2932719-000238-WO1 25. The method of claim 24, wherein the transformer encoder comprises a feed forward neural network layer.

26. The method of claim 25 wherein output of the multi-head attention layer is fed into the feed forward neural network layer.

27. The method of claim 26, wherein the trained transformer encoder comprises a multilayer perceptron head for predicting the state of disease in the subject.

28. The method of claim 27, wherein the state of disease comprises a state of Alzheimer’s disease.

Images