Method and system for generating a brain dynamics foundation model

Abstract

Disclosed is a computer-implemented method for generating a brain dynamics foundation model. The method involves receiving functional neuroimaging data, such as fMRI time series, and identifying regions of interest (ROIs) corresponding to known functional brain areas. A positional embedding is computed using functional connectivity gradients derived from the neuroimaging data. The data is patchified into spatial and temporal patches, from which an observation block is selected and encoded to produce a latent representation. A set of target regions is then sampled from Cross-ROI, Cross-Time, and Double-Cross regions, each comprising patches not present in the observation block. The observation encoder is trained to predict representations of the target regions based on the observation block and their positional encodings. The trained model is adapted to downstream tasks using further neuroimaging data by applying a linear head and fine-tuning the encoder, enabling the model to generalize and make predictions on new task-specific functional neuroimaging inputs.

Description

[0001] Method and System for Generating a Brain Dynamics Foundation Model

[0002] Technical Field

[0003] The present invention relates, in general terms, to methods and systems for modelling brain dynamics using self- supervised learning on neuroimaging data such as functional magnetic resonance imaging (fMRI) or textual information such as clinical records. The invention further relates to the generation of foundation models for brain analysis, enabling prediction, diagnosis, and interpretation of cognitive traits and neurological disorders.

[0004] Background

[0005] The analysis of brain activity using neuroimaging modalities such as fMRI, Electroencephalography (EEG), and Magnetoencephalography (MEG) has become an essential tool in neuroscience and clinical diagnostics. With the increasing availability of large-scale neuroimaging datasets, machine learning models, particularly deep learning approaches, have been developed to extract complex patterns from brain signals to support tasks such as demographic prediction, cognitive trait estimation, disease diagnosis, and prognosis.

[0006] Most technologies seeking to develop models for brain analysis are based on task-specific models that suffer from limited generalizability and adaptability to other tasks. In addition, they fail to leverage the vast amounts of unlabelled fMRI data available.

[0007] One notable prior attempt at brain modelling is Brain Language Model (BrainLM) developed by Ortega Caro et al. (2023), which employs a masking auto-encoder (MAE) architecture for self-supervised pretraining of fMRI data. While BrainLM pioneered the use of foundation models in this domain, its effectiveness is hindered by several drawbacks. First, the direct reconstruction of masked patches may not be well suited for inherently noisy and information-sparse data such as fMRI, leading to difficulty in distinguishing signal from noise. Second, BrainLM uses anatomical locations as positional encodings, which do not effectively capture the functional organization of the brain. Third, its masking strategy is random and not tailored to the specific characteristics of neuroimaging data, reducing its efficiency in learning meaningful representations. Finally, the generative architecture of BrainLM underperforms in linear probing evaluations, a key indicator of the quality of pretrained representations, and its generalizability has not been shown beyond predominantly Caucasian cohorts.

[0008] Furthermore, most existing approaches require extensive fine-tuning to adapt to downstream tasks, limiting their scalability and real-world applicability, particularly in clinical settings where labelled data and computational resources may be scarce.

[0009] Accordingly, there exists a need for an improved foundation model that better captures the functional structure of the brain, tailored to the spatiotemporal complexity of neuroimaging data, enables effective off-the-shelf use via linear probing, and generalizes well across diverse populations and data sources.

[0010] Summary

[0011] According to a first aspect of the present invention, there is provided a computer-implemented method for generating a brain dynamics foundation model, comprising: receiving functional neuroimaging data comprising a time series of brain activity of a brain; identifying a plurality of regions of interest (ROIs) in the functional neuroimaging data, corresponding to known functional regions of the brain; generating a positional embedding based on functional connectivity gradients derived from the functional neuroimaging data corresponding to the ROIs; patchifying the time series into patches, comprising temporal patches and spatial patches; selecting an observation block comprising one or more of the patches; producing a patch-level latent representation of the observation block by encoding the observation block using an observation encoder; sampling a plurality of target regions, each target region being a said patch that is not in the observation block, the plurality of target regions including ones of said patches from each of Cross-ROI regions, Cross-Time regions and Double-Cross regions; generating the model, by training the observation encoder to predict latent representations of the sampled target regions based on the patch-level representation of the observation block and a positional encoding, of each corresponding patch, derived from the positional embedding; receiving a further functional neuroimaging data, corresponding to a task; patchifying the further functional neuroimaging data; applying a linear head to the observation encoder, to condition an output of the observation encoder on the task; updating the model over the further functional neuroimaging data; and applying the updated model to new functional neuroimaging data to predict an output, for the new functional neuroimaging data, corresponding to the task.

[0012] According to a second aspect of the present invention, the method further comprises: receiving structural neuroimaging data corresponding to the functional neuroimaging data; and aligning the structural neuroimaging data and the functional neuroimaging data by incorporating positional harmonics derived from the structural neuroimaging data to the positional embedding.

[0013] Additionally, generating the positional embedding comprises: encoding the structural neuroimaging data using a structural encoder to generate a structural latent representation; training a plurality of tokens by providing the structural latent representation and the functional latent representation as inputs to a multimodal fusion module comprising a transformer for masking portions of the structural and functional latent representations; training the plurality of tokens by reconstructing the structural neuroimaging data and functional neuroimaging data from the respective latent representations; and decoding the tokens to reconstruct original structural and functional latent representations, wherein the multimodal fusion module is trained to minimize a fusion loss defined as a sum of mean squared errors between the reconstructed and original structural and functional latent representations.

[0014] Preferably, the tokens may be updated using attention-based interactions with the structural and functional latent representations.

[0015] Furthermore, the method further comprises discarding the structural and functional decoders after pretraining and retaining only the trained tokens as a unified representation of brain structure and function for downstream tasks.

[0016] Additionally, patchifying the time series into patches comprises determining a patch length based on a fixed temporal duration divided by the repetition time (TR) of the input scan, resizing a set of base embedding weights to match the determined patch length using a pseudoinverse linear transformation, and embedding the patches using the resized embedding weights, such that the method accommodates input data with varying TRs. Furthermore, observation of the Cross-ROI regions and Cross-Time regions is spatially generalized across different ROIs or temporally generalized across timesteps, and observation of the Double-Cross regions is generalized to unseen ROIs at unencountered timesteps.

[0017] Additionally, the method comprises shuffling the ROIs prior to producing the patch-level representation, training the model comprises minimizing a training loss defined as an averaged L2 distance between the predicted latent representations and the corresponding target latent representations.

[0018] According to a third aspect of the invention, there is provided a one-dimensional brain dynamics foundation model token produced by the method.

[0019] According to a fourth aspect, there is provided a system comprising at least one processor and memory storing instructions for executing the method described above.

[0020] Brief description of the drawings

[0021] Embodiments of the present invention will now be described, by way of nonlimiting example, with reference to the drawings in which :

[0022] Figure 1 is a schematic overview of a method for generating a brain dynamics foundation model (BDFM) according to an embodiment of the invention.

[0023] Figure 2 is a flow diagram of data processing steps according to an embodiment of the invention.

[0024] Figure 2a is a flow diagram of computer-implemented method for generating a brain dynamics foundation model according to an embodiment of the invention.

[0025] Figure 3 illustrates the effect of model scale on BDFM performance, comparing variants with ViT-Small, ViT-Base, and ViT-Large backbones, according to an embodiment of the invention.

[0026] Figure 4 depicts a set of comparison graphs showing the performance of BDFM and BrainLM across different tasks using fine-tuning and linear probing, according to an embodiment of the invention.

[0027] Figure 5 depicts a set of bar graphs comparing the performance of BDFM with ablated versions that use sine / cosine functions and anatomical locations for spatial positional embedding, according to an embodiment of the invention.

[0028] Figure 6 depicts line graphs comparing the performance of BDFM using the proposed spatiotemporal masking technique versus a baseline model using vanilla multi-block sampling across different pretraining epochs, according to an embodiment of the invention.

[0029] Figure 7 depicts bar charts of attention weights across major brain networks for NC / MCI classification, comparing results from Caucasian and Asian cohorts, according to an embodiment of the invention.

[0030] Figure 8 depicts a schematic diagram illustrating a two-stage framework for integrating brain structure and function. Part A shows the Unimodal Encoding (UE) stage, where structural MRI (sMRI) and fMRI data are processed separately using reconstruction and patch embedding techniques. Part B depicts the Multimodal Fusion (MF) stage, where latent representations are fused via a central Harmonizer model into unified brain hub tokens capable of reconstructing both structural and functional inputs.

[0031] Figure 9 illustrates the Temporal Adaptive Patch Embedding (TAPE) method, which enables consistent token representation across fMRI scans with heterogenous repetition times (TRs), according to an embodiment of the invention.

[0032] Figure 10 depicts performance scaling of BDFM with varying numbers of ID brain hub tokens during both fine-tuning and linear probing across ADHD- 200 and ABIDE-II datasets, according to an embodiment of the invention.

[0033] Figure 11 shows an ablation study evaluating the effects of geometric pre-alignment, data augmentation, and multimodal fusion on model performance across ADHD-200 and ABIDE-II datasets, according to an embodiment of the invention.

[0034] Detailed description

[0035] Described herein are methods and systems for modelling brain dynamics using self-supervised learning on neuroimaging data such as fMRI, EEG, and magnetoencephalography (MEG), or textual information such as clinical records and experimental results. Some embodiments use multiple types of brain data, including data relating to functional activity, such as fMRI data, and data relating to structural detail of the brain, such as brain shape and volume from Tl- weighted magnetic resonance imaging (MRI) scans, gryi and sulci size and others. The resulting brain models can be used for brain analysis, enabling prediction, diagnosis, and interpretation of cognitive traits and neurological disorders.

[0036] Understanding large-scale brain activity data is crucial for deciphering the complex mechanisms underlying cognitive processes and human behaviour. fMRI captures blood-oxygen-level dependent (BOLD) signals that reflect regional brain activity.

[0037] Rather than directly modelling the original brain activity time series, which are inherently noisy and characterized by sparse information density, the present methods leverage a latent space representation of the fMRI time series, or fMRI time series data. This latent representation is extracted using an encoder, such as a Vision Transformer (ViT) - a deep learning model for computer vision tasks that adapts a transformer architecture, originally designed for natural language processing (NLP), by treating an image as a sequence of fixed-size patches. The use of a latent space offers an improved signal-to-noise ratio (SNR) through a form of feature compression, facilitating the capture of subtle yet semantically meaningful patterns in brain activity. Furthermore, the present invention makes use of a non-generative framework for self-supervised learning from image data - e.g., the recently proposed Image-based Joint-Embedding Predictive Architecture (I-JEPA) model. Unlike conventional masked autoencoder (MAE) approaches that focus on reconstructing the masked input, the present models predict latent representations of target blocks during pretraining. By operating in the latent space, such an approach improves the semantic richness of the learned representations while enhancing scalability and computational efficiency. The present invention incorporates this predictive paradigm to enable effective modelling of spatiotemporal brain dynamics from fMRI data.

[0038] Figure 2 illustrates a computer-implemented method 200 for generating a brain dynamics foundation model. Broadly, the method involves receiving functional neuroimaging data. Regions of interest (ROIs) are then identified in that data, based on known functional regions (including known networks) of the brain. The functional regions can include lobes (frontal, parietal, temporal and occipital) the cerebellum, subcortical structures and others. Functional regions are those for which the functions are generally known and are located relatively consistently across different individuals - e.g., regions governing motor control, sensory perception, emotion and executive functions. These functional regions can also include networks such as the default mode network, salience network, central executive network and others. Once the ROIs are identified, functional connectivity gradients are derived from the functional neuroimaging data corresponding to the ROIs - functional connectivity gradients can be calculated using pre-existing software packages such as the BrainSpace Python package. These are used to generate a brain functional coordinate system (herein interchangeably referred to as a "positional embedding"). The functional neuroimaging data is then patchified, spatiotemporal ly. This patchification means enables masks to be placed over patched in the fMRI data, and the patches that are masked can be changed temporally (in some embodiments, the masked patches may remain the same through the time series functional neuroimaging data). An observation block is selected and a patch-level latent representation of the observation block is produced. This latent representation enables the model to be trained by sampling a plurality of target regions (being a region to model is being trained to predict) and training the model to predict the latent representations of the target regions.

[0039] The step of receiving the functional neuroimaging data (step A) comprises receiving a time series of brain activity of a brain. This step can also include preprocessing, to adapt the time series for learning. Preprocessing may involve normalising the time series data to a common size, noise removal and other preprocessing steps. Preprocessing may be unnecessary for sufficiently clean input data.

[0040] Step B of Figure 2 itself involves a number of steps. These include identifying ROIs in the input data, from which a brain functional coordinate system can be generated and the time series is then patchified. The step of identifying ROIs is based on known functional regions of the brain - as mentioned above, the term "functional regions" is intended to include brain networks as well as specific regions of the brain. This is a parcellation technique whereby the functional neuroimaging data is registered (aligned with) a standard space (e.g., MNI152). Atlases of known regions can be loaded in the same space, with atlas labels being applied to the space to extract average signals within each parcel (ROI) from, for example, 4D fMRI data. The known functional regions can include cortical and subcortical regions, and others. Known cortical regions can be identified using the Schaefer-400 atlas and subcortical regions may be identified using the Tian-Scale III atlases. Alternative, or additional, brain atlases could be used such as Allen Institute for Brain Science atlas, Harvard -Oxford Atlas (HOA), Yale Brain Atlas (YBA), SENSAAS (particularly for fMRI data), BrainMap database, Desikan-Killiany, Destrieux, Glasser, Gordon, Yeo, and Julich-Brain atlases for cortical and subcortical regions (some of which also cover brain networks), and NeuroSynth, BALSA, the Masonic Institute for the Developing Brain (MIDB) Precision Brain, and the Human Brainnetome atlases for brain networks. The regions themselves may be identified by overlaying the atlas over the time series data, and normalising the data to align with the atlas. Alternatively, the regions themselves may be identified by segmenting the time series data with respect to the specific atlas - i.e., identifying regions of the brain based on shape and location derived from the atlas.

[0041] Once the ROIs are identified, they can be used to generate the brain functional coordinate system. Since the ROIs are known, and their functional connectivity is known (i.e., that two regions of the brain have related functions and are thus functionally connected), brain gradient positioning can be used to provide a functional coordinate framework for the brain, based on gradients derived from functional connectivity. Brain gradient positioning is used to develop positional embeddings for the transformer-based architecture used in subsequent model training, as these embeddings encode the positional context of each token within a sequence - e.g., within the time series data. Such embeddings may be implemented using fixed sine and cosine functions across multiple frequencies, or as learnable parameters optimized during training. fMRI and other functional neuroimaging data inherently contains complex spatiotemporal information, necessitating distinct treatment of the temporal and spatial dimensions. With respect to the temporal dimension, corresponding to discrete time steps recorded during a scanning session (i.e., during capture of the functional neuroimaging data and, in some embodiments, structural neuroimaging data), the use of conventional sine and cosine positional embeddings is appropriate - as each time point in a region of interest (ROI) follows a sequential temporal order. However, this approach is unsuitable for the spatial dimension, where ROIs span across three-dimensional brain volumes without any intrinsic or natural ordering. The absence of such ordering renders conventional sinusoidal embeddings ineffective for capturing spatial relationships among ROIs. While anatomical locations of ROIs have been explored as a basis for spatial positional encoding, they are insufficient to reflect the brain's functional organization. Spatially adjacent ROIs may exhibit distinct activation patterns, illustrating the lack of spatial coherence in functional brain data. Accordingly, anatomical location-based embeddings fail to capture functional heterogeneity. To address this, functional connectivity gradients are used to derive spatial positional information that more accurately reflects functional parcellation of the brain.

[0042] The functional connectivity gradient is a continuous representation of the functional relationships among different ROIs within the brain. The location of the regions and their functional connectivity gradients determine brain gradient positioning - i.e., the location of brain gradients, which automatically embeds their functional connectivity as this was determined prior. Each dimension or attribute within the gradient corresponds to an axis in a latent space that characterizes the organization of brain regions and functional networks. These attributes include any against which future data can be categorised - for example, age, cognition (measured using standard neurophysiological assessments) and probability or likelihood of brain disease. The relative positioning of ROIs along these axes reflects the similarity of their connectivity profiles, wherein a shorter distance between two ROIs implies a higher degree of functional similarity.

[0043] The concept of a spatial gradient in brain organization is based on the notion of a synaptic hierarchy underlying cognitive function. Functional brain networks in humans and non-human primates exhibit linear distributions along such gradient axes. Empirical findings suggest that these gradients encode functional variations associated with individual differences in age, cognition, and neurological disorders (brain disease).

[0044] This gradient-based representation is used to derive spatial positional information. The spatial positional information reflects functional relationships between brain regions, rather than anatomical relationships between brain regions (anatomical relationships may be captured using structural imaging information). Spatial positional information enables spatial context to be embedded within the latent space used by transformer-based models.

[0045] To generate the functional connectivity gradients, a non-negative affinity matrix A(ij') is computed. The non-negative affinity matrix is constructed as a graph Laplacian derived from pairwise connectivity strengths between ROIs: where c, and cj represents the features (functional connectivity) across the ROIs / and j, respectively. Gradients are subsequently derived using a diffusion map technique, which is a nonlinear dimensionality reduction method that captures the underlying manifold structure of the data. From the affinity matrix A, a diffusion matrix Lsand a corresponding diffusion operator Msare computed as follows: (2) where D is the degree matrix of A. Here <5 is set to a predetermined number (e.g., 0.5) to maintain the global relations between ROIs in the embedding space.

[0046] The method further comprises computing the eigenvectors and eigenvalues of the diffusion operator Ms. The graph Laplacian yields the set of eigenvalues ^ (k indicating the eigenvalue) and corresponding eigenvectors from largest to smallest, which represent the functional gradients of the brain. The parameter t denotes the diffusion time, which controls the granularity or scale of the diffusion process. Each eigenvalue may be adjusted by computing Afc / (1 - Ak). This adjustment improves robustness against noise in the lower spectrum. In particular, the adjustment is calculated to reduce the influence of unstable components of the functional (and / or structural) neuroimaging data.

[0047] In some embodiments, a diffusion map is also generated, to ascertain how functions diffuse through the brain over the time series. The diffusion map may be generated by stacking column vectors Ote R.nxm, where n represents the total number of regions of interest (ROIs), and m corresponds to the number of gradient dimensions, at time t. The columns vectors are the eigenvectors of Ms.

[0048] After calculating the eigenvalues and eigenvectors, a gradient matrix G is generated to represent the spatial embedding of the ROIs along the derived functional gradients:

[0049] G may be of the same dimensionality is A.

[0050] In accordance with the present invention, the resulting gradient matrix G e j^nxd / 2 ,where n is the number of ROIs and m is the number of gradient dimensions, is used to define the spatial positioning of the ROIs. In particular, the methodology comprises generating a positional embedding P from G. To achieve this, G is transformed into using a trainable linear layer. In particular, G is transformed to G e R’1XC!- / 2where d represents the embedding dimension of the ViT backbone. A temporal position encoding is then generated. The temporal positioning encoder may be generated using predefined sine and cosine functions applied across the time series dimension. The temporal positioning encoder may be determined according to the formula, T e Rnxd / 2. The final positional embedding is obtained by concatenating the temporal and spatial components. The final positional embedding can be formulated as P = [T, G] e Rnxd.

[0051] To facilitate training, per Step B, the time series data is patchified into spatiotemporal patches (e.g., for the brain as reflected in the functional neuroimaging data, spatial patches include respectively different portions of the brain, temporal patches include the same portion of the brain, at different timesteps and, in some embodiments, a two patches may cover both different spatially distributed portions of the brain at different timesteps). These positional embedding of the brain gradients (i.e., functional connectivity embeddings) can then be added to the spatiotemporal patches, to facilitate informed prediction. In particular, prediction is based not only on learned relationships between data, but also on functional connectivity between different spatially distributed regions of the brain, and the temporal interaction between those regions - e.g., that activation of one region occurs concurrently, succeeds or precedes activation of another region.

[0052] The brain dynamics model predicts latent representations of brain dynamics. It therefore operates in the latent space to predict, based on an observation block (or, in some embodiments, multiple observation blocks) representations of multiple target blocks. Where multiple observation blocks are selected, a prediction of each target block may be averaged. To this end, Step C of Figure 2 involves selecting an observation block and producing a patent-level representation of the observation block.

[0053] To illustrate with respect to fMRI, though other functional neuroimaging techniques may be used, for a given input fMRI time series, the temporal signal for each brain parcel (ROI) is divided into patches - i.e., patchified. Patchification can be based on regular temporal samples of a signal and / or regularly spaced spatial regions of a signal. Prior to, or after, patchifying, the ROIs may be shuffled. Shuffling randomizes spatial ordering. This removes the influence of spatial proximity from the training phase. Each patch may comprise p time points, as illustrated by the dashed boxes in Figure 1.

[0054] The observation block x is then selected. Selection may be purely random. Alternatively, selection may be from within a defined spatial range (which ROIs or patches)and temporal range (number of time steps from the start of the time series). The spatial and temporal range may be denoted by 0^,^} where specifies the range ratio along the ROI dimension, and rfi defines the index range for the temporal patches (e.g., 10 time patches in total). Regarding notation, "o" is subscript label referring to an observation, R refers to ROI, and T refers to a time step. So, these range ratios refer to the relative proportion of ROIs or time steps used to sample a block.

[0055] The selected observation block x is then passed through an observation encoder fethat generates a patch-level latent representation of the observation block. In particular, fggenerates a set of patch-level representations sxfor the observation block, defined as: (4) where Bxrepresents the set of patch indices associated with the observation block x, and sXjdenotes the encoded representation of the jthpatch within the block. There may be a single patch index where the observation block comprises a single patch, or multiple indices where the observation block comprises multiple patches.

[0056] As shown in Figure 1 - schematically showing a method 100 for generating a Brain Dynamics Foundation Model (BDFM) according to an embodiment of the present invention - the framework begins by receiving functional neuroimaging data 101, such as resting-state fMRI time series. This data is preprocessed and segmented into a series of spatiotemporal patches (102), where each patch corresponds to a combination of a specific region of interest (ROI) and a discrete time step.

[0057] A subset of these patches is designated as the observation block, which is passed through a ViT encoder as described above to generate a latent representation of the observed brain activity. Per step D, the remaining input data, excluding the observation block, is sampled. In particular, the remaining input data is partitioned into multiple types of target regions. These types of target regions may be non-overlapping, such that no patch has its influence inadvertently weighted more heavily than the influences of other patches. Thus, the types of target regions are distinct.

[0058] The types of target regions include: (i) Cross-ROI regions (cr), in which target patches are sampled from ROIs that are spatially distinct from the observation block, enabling the model to generalize across different anatomical areas; (ii) Cross-Time regions ( / ?), in which target patches selected from time points that are temporally separate from those in the observation block, allowing the model to capture temporal dynamics in brain activity; and (iii) Double-Cross regions (y), which involve target patches that differ from the observation block in both spatial (ROI) and temporal (time step) dimensions.

[0059] The remaining input data is masked so that it can be used at Step E to train the model to predict the latent representations of unobserved brain states - i.e., the latent representations of the target regions. Using the fMRI time series example, the model will be trained to predict representations of other, masked parts of the fMRI time series within a latent space, given a single observation block as input.

[0060] Unlike masked autoencoders, that use randomised masking, the present framework introduces a more structured masking strategy to improve generalizability. Patches in fMRI data exhibit variation not only spatially, based on their position within the brain's functional organization, but also temporally, depending on dynamic brain states and external task conditions. Furthermore, the nonlinear interactions among distributed brain networks introduce additional complexity to the relationships between different spatiotemporal regions.

[0061] To ensure robust training across these dimensions, the model samples a fixed number K of target blocks from each of the three region types. This structured sampling mechanism introduces a strong inductive bias that encourages the model to learn diverse and generalized representations, rather than relying on low-complexity heuristics. The mask corresponding of the region is denoted by

[0062] In some embodiments, an overlapped sampling strategy is employed to enable a flexible observation-to-input ratio during training or pretraining. The basis for the overlapped strategy is that a sufficiently wide dynamic range of masking ratios can improve the quality of learned representations during self-supervised learning. To support this, the ratio of observed to masked inputs is not fixed. The ratio of observation patches to masked patches is instead dynamically adjusted using overlapping sampling regions.

[0063] Specifically, when sampling a target block Sy from a given region r, where r = a or / ?, the target is sampled from the union of the observation mask 'Bxand the region-specific mask BJ. In contrast, for region r = y , the target block is sampled directly from the corresponding region-specific mask By, without overlap with the observation block. Formally, the overlapped sampling strategy is defined as follows: where r is a specific region, y is the target and x is the observation. Following the overlapped sampling procedure, portions of the observation block may intersect with the a and / ? target regions. To prevent information leakage and maintain prediction integrity, any ROI in the observation block that overlaps with the a target region is removed. Similarly, all time steps corresponding to ROIs that overlap with the / ? target region are eliminated from the observation block. Refer to Table 6 for details regarding illustrative block sizes used in this configuration.

[0064] Per Step E, once the observation encoder fgproduces the encoded output sx, a predictor module g^, is trained to predict the latent representations of the target blocks Sy, conditioned on the positional embedding P (as illustrated in Figure 1, being a brain functional coordinate system). For present purposes, the training objective is defined as minimising the mean squared error ( L2distance) between the predicted target representations and the ground truth target representations Sy . The overall training loss £ is computed as:

[0065] The result of Steps A to E is a pretrained encoder network fsthat is extracted (Step F) for fine-tuning on a target task - thus, the original decoder, trained with the observation encoder during pretraining, may be discarded during fine- tuning. Fine tuning, reflected in the right side of Figure 2, involves data input steps G to I, that are the same as steps A to C - though Step H may make us of the functional coordinate system generated at Step B. Step J a linear head is applied to the encoder fg, for fine-tuning the encoder using a supervised loss function (Step K). The supervised loss function may be designed or calculated for a specific downstream (i.e., new) task or downstream tasks. The fine-tuned encoder is then tested, per Step L, in a standard manner, to ensure it is sufficiently accurate to perform the downstream task or tasks. The pretrained target encoder together with a linear head can be fine-tuned for different tasks. For classification tasks like determination of sex, NC / MCI, the output from the model will be passed through a softmax function to generate a prediction, since the output of the encoder is continuous but the task is discrete. For regression tasks such as age determination, the output directly from the model is used directly, since the output is continuous.

[0066] Though the model described above is "trained", in some embodiments, the above steps are considered "pretraining". In these embodiments, the model developed according to Steps A to E is generalized over particular neuroimaging data. That model can then be "trained" on a task - e.g., prediction of Normal Cognitive (NC) and Mild Cognitive Impairment (MCI) classification. As discussed above, Figure 2a is a flow diagram of computer-implemented method for generating a brain dynamics foundation model or BDFM according to an embodiment of the present invention. At step 202, the method begins by receiving functional neuroimaging data, such as functional magnetic resonance imaging (fMRI), comprising a time series of brain activity. At step 204, a plurality of regions of interest (ROIs) is identified within the fMRI data, each corresponding to known functional regions of the brain. In step 206, a positional embedding is generated based on functional connectivity gradients derived from the neuroimaging data associated with the identified ROIs.

[0067] Following this, at step 208, the time series is patchified into a plurality of spatial and temporal patches. An observation block comprising one or more of the patches is then selected in step 210. At step 212, the selected observation block is encoded using an observation encoder, such as a vision transformer (ViT), to produce a corresponding patch-level latent representation. In step 214, a plurality of target patches is sampled from the remaining (non-observation) regions of the data. The sampled target regions include patches from Cross-ROI regions (spatial variation), Cross-Time regions (temporal variation), and Double-Cross regions (both spatial and temporal variation).

[0068] In step 216, the observation encoder is trained to predict the latent representations of the target regions based on the observation block and positional encodings derived from the positional embedding. Subsequently, in step 218, further functional neuroimaging data corresponding to a downstream task is received. This data is also patchified in step 220. At step 222, a linear head is applied to the observation encoder to condition its output based on the downstream task. Then, at step 224, the model is updated using the further functional neuroimaging data. Finally, in step 226, the updated model is applied to new functional neuroimaging data to generate an output corresponding to the task.

[0069] Figure 3 illustrates the performance of the framework of Figures 1 and 2 across different model configurations, utilizing ViT-Small (ViT-S), ViT-Base (ViT-B), and ViT-Large (ViT-L) as backbone architectures. ViT configurations can include any number of parameters, such as ViT-S with 22 million parameters, ViT-B with 86 million parameters, and ViT-L with 307 million parameters. The results indicate a consistent improvement in performance with increasing model size. Specifically, a clear trend is observed in which both accuracy and correlation metrics improve as the model scale increases, with the ViT-L-based implementation of the consistently achieving the highest performance across evaluation benchmarks.

[0070] Datasets

[0071] During experiments, the brain foundation model was pretrained (i.e., Steps A to E) using a large-scale publicly available dataset, specifically the UK Biobank dataset, which includes resting-state fMRI recordings from approximately 40,162 individuals aged between 44 and 83 years. Approximately 80% of the dataset is allocated for self-supervised training, while the remaining 20% is reserved for internal evaluation tasks, such as age and sex prediction.

[0072] For external evaluation and validation of generalizability of a system implementing the framework, additional datasets were employed. These included:

[0073] (i) the HCP-Aging dataset from the Human Connectome Project, used to evaluate the prediction of cognitive traits and demographic attributes;

[0074] (ii) the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, used for diagnosis and prognosis of neurodegenerative diseases; and

[0075] (iii) a region-specific dataset comprising an Asian memory clinic cohort, used to assess the system's generalizability across different ethnic populations, particularly for the early detection of cognitive impairments.

[0076] The input fMRI data is parcellated into 450 regions of interest (ROIs), utilizing a combination of the Schaefer-400 atlas for cortical regions and the Tian-Scale III atlases for subcortical regions. Normalization of the time-series data is performed using robust scaling techniques to account for individual variability and signal noise.

[0077] In the default configuration, the model receives input sequences comprising 160 timepoints per ROI, resulting in a total input shape of 450 x 160. These sequences are extracted from longer recordings of 490 timepoints using a stride of three. For fine-tuning and downstream task evaluation (e.g., linear probing), the datasets are partitioned into training, validation, and testing subsets according to a 60:20 :20 split ratio.

[0078] Rather than reconstructing masked patches of the fMRI time series directly, the present invention operates in the latent space, as illustrated in Figure 1. In one embodiment, the present invention employs a Vision Transformer (ViT) as the observation encoder. The ViT is a type of deep neural network architecture that applies the transformer model (originally developed for natural language processing) to image or grid-like data, including spatiotemporal signals like functional MRI (fMRI). fMRI input data is first segmented into temporal-spatial patches. The ViT receives spatiotemporal patches derived from the fMRI time series and encodes a selected observation block into a latent representation. The observation encoder leverages self-attention mechanisms to capture spatial and temporal dependencies within the input data. By operating in the latent space rather than directly on the original brain activity time series, which are inherently noisy and information-sparse, the ViT-based encoder enables the present framework to extract abstract and semantically meaningful representations suitable for predicting representations of one or more target blocks.

[0079] A predictor module or decoder is used to decode the latent encoding of the input, generated by the observation encoder, to produce the output. To this end, the decoder is trained concurrently with the encoder - e.g., through backpropagation of mean squared error loss. The decoder may comprise a softmax layer, for discrete tasks such as classification. It may also include a regression layer for tasks with continuous output such as age determination, is designed as a lightweight (narrow) ViT variant, maintaining the same architectural configuration as the observation encoder, but with reduced depth and embedding dimensions. For instance, when ViT-S or ViT-B is used as the observation encoder, the corresponding predictor comprises 6 transformer layers with embedding dimensions of 192 and 384, respectively. When ViT-L is used, the predictor includes 12 transformer layers and an embedding dimension of 384. During pretraining, the model is configured without the use of a [cis] token. For evaluation, the target encoder is employed to generate a global fMRI representation by applying average pooling across the latent patch outputs. The parameters of the target encoder are incrementally updated through an Exponential Moving Average (EMA) of the observation encoder's parameters. In some embodiments, all the latent patches are averaged to formulate the global representation.

[0080] Main Results

[0081] Tables 1, 2, and 3 present a comparative evaluation of the present framework (BDFM) against both task-specific deep learning models and a prior foundation model, BrainLM, for functional neuroimaging analysis. The selected baseline models represent previously established state-of-the-art approaches and encompass diverse architectural paradigms: convolutional neural network (CNN)-based BrainNetCNN, graph neural network (GNN)-based BrainGNN, and transformer-based BNT.

[0082] To ensure a fair comparison, both BDFM and BrainLM were implemented using an identical ViT-Base (ViT-B) backbone architecture and were fine-tuned on each downstream task. BrainLM adopted a classification ([cis]) token during downstream evaluation.

[0083] The results demonstrate that the present framework consistently achieves state-of-the-art performance across a variety of downstream tasks, including those conducted on unseen subjects from the same pretraining cohort and on independent external datasets.

[0084] Table 1 : Internal tasks of age and sex prediction on UKB 20% held-out. The mean (standard deviation) of Mean Squared Error (MSE), Pearson Correlation ( p ), and / or Accuracy (ACC), Fl score across 5 independent runs is reported. T : the higher, the better; 1 : the lower, the better. The best results are in bold, with * denoting significant improvement over previous approaches ( p < 0.05 ).

[0085] +Trained-From-Scratch. Table 2: External tasks of demographics and trait prediction on HCP-Aging.

[0086] Table 3 : External tasks of brain disease diagnosis and prognosis on ADNI and an Asian disease cohort.

[0087] Linear Probing

[0088] BrainLM demonstrates its performance gains primarily through fine-tuning with an attached multilayer perceptron (MLP). However, to more directly evaluate the quality of the representations learned during pretraining, off-the-shelf assessments such as linear probing provide a more informative metric. As illustrated in Figure 4, the present framework consistently outperforms BrainLM under linear probing evaluation and shows a smaller performance drop between fine-tuning and linear probing. Linear probing serves as a diagnostic tool to benchmark the intrinsic quality of learned representations from the present framework and BrainLM, revealing that the present framework achieves better performance with less task-specific adaptation. This indicates that the present framework learns more robust and abstract representations.

[0089] Ablation Studv

[0090] Figure 5 illustrates a comparative evaluation of different positional embedding strategies used to encode spatial information of brain regions of interest (ROIs), in accordance with an embodiment of the present invention. Specifically, the figure compares the performance of the present framework employing Brain Gradient Positioning against two ablated variants using baseline spatial encodings: (i) fixed sine and cosine functions, and (ii) anatomical location-based encodings.

[0091] In the first baseline, sine and cosine positional embeddings are applied along the ROI dimension, following conventional transformer methods that encode position using periodic functions. In the second baseline, each ROI is assigned a positional embedding based on its anatomical coordinates in brain space. Both baselines are limited in their ability to capture the brain's functional organization— sine / cosine embeddings assume a sequential or grid-like structure, and anatomical locations may place functionally distinct regions close together, despite weak connectivity. In contrast, the present framework employs Brain Gradient Positioning, which derives continuous-valued spatial embeddings from functional connectivity gradients computed using diffusion map techniques. This method reflects population-level functional parcellations and preserves the nonlinear relationships among brain networks.

[0092] As shown in Figure 5, the use of Brain Gradient Positioning results in significantly improved performance on downstream tasks compared to both ablated variants. This demonstrates that the proposed approach enables more accurate modelling of brain dynamics by embedding ROIs in a functionally meaningful coordinate space.

[0093] Figure 6 illustrates the effectiveness of the Spatiotemporal Masking strategy of the present framework, compared to a baseline that employs standard multiblock target sampling. In accordance with one embodiment, the present framework divides the fMRI input, excluding the observation block, into structured target regions: Cross-ROI, Cross-Time, and Double-Cross. This structured masking introduces a stronger inductive bias that guides the model to learn spatial, temporal, and spatiotemporal dependencies in brain dynamics. As shown in Figure 6, the present framework consistently outperforms the baseline across varying pretraining epochs. The performance gap demonstrates that Spatiotemporal Masking enhances representation learning efficiency and robustness, even in early stages of training.

[0094] Figure 7 shows the network-level attention distribution produced by the present framework for classifying normal cognition (NC) versus mild cognitive impairment (MCI). The brain is divided into seven functional subnetworks based on the Schaefer atlas: control (CN), default mode (DMN), dorsal attention (DAN), limbic (LN), salience / ventral attention (SAN), somatomotor (SMN), and visual (VN).

[0095] To assess whether the present framework captures functional brain organization, the model's self-attention values are analysed. For each region of interest (ROI), attention is averaged across its 10 temporal patches. These values are then averaged within each subnetwork and normalized to obtain the network-level attention distribution. As shown in Figure 7, the present framework produces consistent attention patterns across both Caucasian and Asian groups. This suggests that the model learns generalizable and functionally meaningful brain representations.

[0096] Multimodal Brain Dynamics Foundation Model

[0097] In an extended embodiment, the present framework incorporates structural brain imaging data, such as Tl-weighted MRI, in addition to functional neuroimaging data. The structural information may include volumetric, morphological, or cortical thickness features and others, extracted from anatomical scans. The present framework further introduces a multimodal brain foundation model (referred to hereinafter as BrainHarmonix) that integrates morphological structure (e.g., derived from Tl-weighted MRI scans) with functional brain dynamics (e.g., from fMRI) into a compact, information-rich latent representation using continuous-valued one-dimensional (ID) tokens.

[0098] To incorporate structural embeddings into the latent representation, structural brain imaging data and functional neuroimaging data are aligned (herein interchangeably referred to as "pre-aligning"). In some embodiments, geometric harmonics are used to pre-align individual brain morphologies, e.g., cortical morphologies, to a population-level functional coordinate system such as the functional coordinate system derived for positional embeddings in Step B of Figure 2. This can be achieved by combining latent representations of the functional and structural data and applying the functional brain gradients and positional embeddings to the combined latent representations, since those representations have been aligned (i.e., applying the positional embeddings derived from the functional brain gradients, into the latent representation of the functional neuroimaging data, enables those positional embeddings to be similarly applied to the structural imaging data with which the functional neuroimaging data has been aligned). This can be achieved using a single encoder to encode the combined data set, or separate functional and structural encoders for the functional and structural imaging data, respectively. This alignment imposes a physics-informed inductive bias that captures biologically meaningful structural constraints and enhances both cross-subject and crossdataset generalization. By embedding such structural priors directly into the latent space, the framework improves the interpretability and robustness of the resulting functional representations.

[0099] The present invention further comprises a multimodal fusion module, herein referred to as a Harmonize / -, which fuses latent representations from the structural and functional encoders into unified ID tokens. These tokens serve as holistic, modality-agnostic representations of the brain. The tokens are compact, continuous-valued embeddings, that encapsulate the information from both structural morphology and functional dynamics. In some embodiments, 128 tokens can be averaged to be one token, with a linear head attached to generate prediction. Optionally, separate decoders may be used to reconstruct the original modality-specific latents, allowing for disentanglement of structure and function if required.

[0100] To support scalable pretraining across heterogeneous datasets with varying temporal resolutions (TRs), the present invention also introduces a Temporal Adaptive Patch Embedding (TAPE) module. TAPE enables the model to accommodate variable-length time series data without loss of temporal resolution or context. Additionally, a novel data augmentation technique is provided for fMRI, involving hierarchical downsampling of time series data to simulate and harmonize different TR levels across datasets.

[0101] As shown in Figure 8, the pretraining of the multimodal brain foundation model 800 includes two sequential stages: a unimodal encoding stage 800a and a multimodal fusion stage 800b. This design enables flexible integration of diverse neuroimaging modalities, including unpaired structural and functional data, while producing compact and generalizable brain representations.

[0102] In the first stage, referred to as Unimodal Encoding (UE) 800a, the system separately trains two modality-specific encoders: one for structural neuroimaging data (BrainHarmonix-S) 802, such as Tl-weighted MRI, and another for functional data (BrainHarmonix-F) 804, such as resting-state fMRI. This separation allows flexible use of unpaired structural and functional data.

[0103] For the structural encoder, BrainHarmonix-S 802, the framework of Figure 8 utilizes a masked autoencoder to learn latent representations of brain morphology. Since the structural imaging data will typically be 3-dimensoinal, the framework may use a three-dimensional Masked Autoencoder (3D MAE). This encoder 802 is applied to Tl-weighted MRI data and is configured to capture structural information relevant to cortical morphology, in a latent representation. The 3D MAE is pretrained using a masking strategy consistent with conventional masked autoencoding techniques, enabling the model to effectively leverage large-scale T1 imaging datasets for representation learning. For the functional encoder, BrainHarmonix-F 804, the framework of Figure 8 introduces two key innovations. First, a geometric harmonics-based alignment method that pre-aligns (i.e., before generation of embeddings) brain dynamics with structural geometry. This alignment harmonizes individual functional dynamics with population-level anatomical geometry, embedding biologically meaningful constraints into the learned representations. Second, a Temporal Adaptive Patch Embedding (TAPE) layer is proposed. The TAPE layer which enables the encoder to flexibly handle fMRI sequences with variable TRs. The TAPE layer dynamically adapts the patch segmentation process to match the temporal structure of each input sequence. As a result, the TAPE layer ensures compatibility across heterogeneous datasets.

[0104] For BrainHarmonix-F 804, the structural-functional coherence of brain dynamics modelling is enhanced by pre-aligning functional signals with anatomical morphology. Specifically, brain ROIs are positioned within the transformer architecture using geometric harmonics derived from a population-level cortical surface mesh. A cortical surface mesh will typically be a series of vertices, connected by edges, that computationally reproduce the surface of a region of the brain. A population-level mesh is the mesh, averaged across a population. There may be multiple populations, depending on the task being performed, predetermined demographics (e.g., age and sex) and other factors. Moreover, the averaging may be made with regard to known regions of the brain discussed with reference to Step B.

[0105] Geometric harmonics are the natural, orthogonal vibration patterns of the brain's folded surface. Given a mesh representation M of a population-averaged cortical surface derived from T1 imaging, the Laplace-Beltrami operator (LBO) AMis constructed to capture local vertex-to- vertex spatial relationships and cortical curvature. The corresponding eigenvalue problem can then be solved as follows: sthe sequence of geometric harmonics with the corresponding eigenvalues } ordered regarding spatial frequency. Each \pte RVxlis further downsampled through averaging within one ROI to formulate e R'Vxl, where V represents the number of vertices in the mesh and N denotes the number of ROIs in a brain parcellation. First J downsampled harmonics xpj e RWx / will be selected for learning positional embedding. A learnable linear layer is then incorporated to transform geometric harmonics i / J;into positional embeddings , where d denotes the embedding dimension of the transformer - BrainHarmonix-F.

[0106] By explicitly embedding geometric constraints into the functional neuroimaging latent representations, the framework pre-aligns functional brain organization with cortical structure, thereby facilitating more effective integration during subsequent multimodal fusion. Moreover, embedding this physics-informed inductive bias, derived from population level observations, can further enhance cross-subject and cross-dataset alignment.

[0107] Functional neuroimaging data are acquired across a wide range of scanners, sites, and pulse sequences, resulting in BOLD signals sampled at TRs ranging from sub-seconds to several seconds. However, existing brain dynamics foundation models typically operate under a fixed TR setting for both pretraining and downstream tasks, or downsample higher-resolution datasets to match the lowest TR available. This limitation hinders their capacity to leverage diverse datasets during pretraining and to adapt to downstream tasks that involve varying temporal resolutions. Moreover, downsampling leads to a loss of high- frequency temporal information, thereby reducing the richness and fidelity of the learned representations. The root of this limitation lies in the use of a fixed, uniform patch size that is incompatible with heterogeneous TRs.

[0108] To overcome this critical limitation, Temporal Adaptive Patch Embedding (TAPE) is introduced in BrainHarmonix-F, that dynamically accommodates varying TRs across functional neuroimaging data as shown in Figure 9. TAPE involves defining a consistent temporal duration r for any token (embedding from one fMRI patch) and a base embedding weight to’ e U«k' corresponding to the patch size k*. Given an arbitrary functional neuroimaging time series with repetition time TR = s, the corresponding patch size k and resized embedding weights to e where to is obtained by pseudoinverse resize (Pl-resize), with the linear transformation matrix Bjf e Rfcxr. Since different functional neuroimaging scans may vary in total duration, patchifying with temporally consistent tokens could result in a varying number of tokens across time series. If the maximum number of tokens per time series across the dataset is m, any time series producing fewer tokens ( n < m ) will be zero-padded followed by an attention mask, ensuring tokens derived from padding are excluded from attention computation shown in Figure 9.

[0109] In other words, the patch length k is dynamically determined for each functional neuroimaging scan based on its repetition time (TR). As a result, each patch represents a consistent duration r in seconds, regardless of the TR of the scan. For instance, if the desired temporal duration T is fixed at 3 seconds, and the TR of a given scan is 0.75 seconds, then the patch would contain k = round(3 / 0. 75) = 4 time points. This approach enables the model to process scans acquired with different temporal resolutions while ensuring that each token consistently represents the same real-world duration across datasets. This consistency improves cross-subject and cross-dataset alignment and allows for more effective pretraining using heterogeneous neuroimaging data.

[0110] The method further includes resizing patches based on a reference patch length. As a result, varying patch lengths across different TRs can be accommodated. The base embedding weights corresponding to a reference patch size k* are resized using a pseudoinverse linear transformation. Specifically, a transformation matrix is used to project the fixed-size embeddings into the appropriate length k , enabling seamless alignment of positional encodings across dynamic input sizes.

[0111] TAPE can be used in both the framework set out in Figure 1 and that set out in Figure 8.

[0112] At the second stage, i.e. Multimodal Fusion (MF) stage 800b, shown in Figure 8, a set of learnable one-dimensional (ID) brain hub tokens 814 is introduced to act as a representational bottleneck between modalities. The ID tokens are a shared medium through which information from both structural and functional modalities is compressed and integrated. The ID tokens capture the common, fused representation (capturing the relationships of, and between, the functional and structural information), enabling the model to learn joint patterns while reducing redundancy between modalities. The ID tokens are thus abstract latent representations, being learned embeddings trained to reconstruct both structural and functional latents, capturing shared, compressed representations of those latents. These hub tokens are optimized through an attention-based transformer model referred to as the Harmonizer ( ) 808 to reconstruct both structural and functional latent representations, thereby capturing shared information across modalities, as illustrated in Figure 8. The Harmonizer ( )

[0113] 808 fuses structural 810 and functional latents 812 into ID tokens 814, then decoder Ds816 and DF818 reconstruct modality-specific latents. The use of ID brain hub tokens 814 enables the construction of a unified and compact latent space that encodes both brain morphology and dynamics holistically.

[0114] Let be the modalitity-specific latents of aligned Tl-fMRI produced by BrainHarmonix-S and BrainHarmonix-F, respectively, where d is the common embedding dimension and NS,NFare the numbers of tokens in each modality. A set of NHlearnable continuous-valued ID brain hub tokens Hoe Rw«xd|Sintroduced, shared by all pairs and optimized jointly with the network in MF. At each forward pass, the hub tokens are concatenated with the structural and functional modality sequences or tokens, to form a combined input stream, which is then processed by the Harmonizer transformer (.W): where Zois the concatenated input to and H is the hub tokens updated Self-attention within 808 allows the ID tokens to gather information from both structural and functional tokens, while also enabling cross-modal interactions between Zsand ZF.

[0115] Two lightweight decoders ) project H back into each modality's latent space as shown in Figure 8. Formally, training in MF is defined as: where represents the parameters in H,Ds, and ©F, respectively. || • ||^ denotes the Mean Square Error. The Multimodal Fusion stage is trained using 69,360 matched Tl-fMRI scan pairs. Following training, the modality-specific decoders and embeddings are discarded during downstream adaptation, and only the fused ID brain hub representations are retained for use in downstream tasks. The tokens generated from the pertaining data (data used in Steps A to C of Figure 2) may not be reused - i.e., they may be discarded. The observation encoder is then retrained (fine-tuned) for generating ID brain hub tokens conditioned on a new task or tasks. The ID tokens generated during tine-tuning are then fed into a projection head for the new task or tasks.

[0116] Datasets

[0117] In accordance with one embodiment, the model was pretrained using two of the largest publicly available neuroimaging datasets: the UK Biobank (UKB) and the Adolescent Brain Cognitive Development (ABCD) study. From the UK Biobank, a total of 43,112 participants (ages 44-83) were included, comprising 46,455 Tl-weighted structural MRI scans and 40,162 resting-state fMRI time series acquired at a repetition time (TR) of 0.735 seconds. From the ABCD dataset, 11,221 participants (ages 8-11 at baseline) were selected, contributing 18,139 Tl-weighted scans and 30,771 resting-state fMRI time series acquired at a TR of 0.8 seconds. During the Unimodal Encoding (UE) stage 800a, a total of 64,594 Tl-weighted MRI scans from both datasets were used to pretrain the structural encoder (BrainHarmonix-S) 802. For functional data, a temporal downsampling strategy was applied to simulate diverse TR distributions. Specifically, UKB fMRI data were downsampled by factors of 1, 2, and 3, resulting in effective TRs of 0.735 s, 1.47 s, 2.205 s, and 2.94 s. Similarly, ABCD fMRI data were downsampled by factors of 1 and 2, yielding TRs of 0.8 s, 1.6 s, and 2.4 s. This resulted in a total of 252,961 augmented samples for functional pretraining (UKB: 40,162 x 4; ABCD: 30,771 x 3) used for training BrainHarmonix-F with support for heterogeneous TRs.

[0118] BrainHarmonix-F uses geometric-harmonic positional embeddings to make each fMRI token respect the brain's geometry, but its structural MRI and fMRI streams are still produced by two independent unimodal encoders. The subsequent Multimodal Fusion (MF) stage introduces shared ID tokens (herein also referred to as ID "brain-hub" tokens) that are trained to reconstruct both modality-specific latents, forcing the model to learn genuinely joint, cross-modal representations beyond this pre-alignment and giving it a compact embedding where structure-function dependencies are explicitly integrated. During the Multimodal Fusion (MF) stage 800b, a total of 69,360 matched Tl-fMRI pairs were extracted across both datasets. Each Tl-weighted image could correspond to multiple fMRI runs from the same session. All fMRI data were parcellated into N=400N = 400N=400 regions of interest (ROIs) using the Schaefer-400 functional atlas.

[0119] In one embodiment, the framework of Figure 8 was evaluated on a collection of six benchmark neuroimaging datasets comprising both neurodevelopmental and neurodegenerative conditions, in order to assess the generalization performance of the pretrained multimodal foundation model.

[0120] Three datasets were used to evaluate neurodevelopmental disorder classification performance: • The Autism Brain Imaging Data Exchange I (ABIDE-I) and ABIDE-II datasets, for binary classification between individuals diagnosed with autism spectrum disorder (ASD) and neurotypical controls;

[0121] • The ADHD-200 dataset, for binary classification between individuals diagnosed with attention-deficit hyperactivity disorder (ADHD) and matched controls.

[0122] Three additional datasets were used to assess performance on neurodegenerative disorders and cognitive function tasks:

[0123] • The Parkinson's Progression Markers Initiative (PPMI) dataset, with a repetition time (TR) of 2.5 seconds, used for multi-class classification across four categories: healthy controls, scans without evidence of dopaminergic deficit (SWEDD), prodromal Parkinson's cases, and confirmed Parkinson's disease (PD) cases;

[0124] • The Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, with a TR of 3.0 seconds, used for binary classification between mild cognitive impairment (MCI) and healthy controls;

[0125] • The Human Connectome Project - Aging (HCP-A) dataset, with a TR of 0.8 seconds, used for regression prediction of cognitive performance, specifically executive function as measured by Flanker task scores.

[0126] Each dataset was partitioned into training, validation, and testing subsets using a 60:20: 20 ratio. Model performance was averaged over three independent evaluation runs using distinct random splits. For neurodevelopmental datasets, stratified sampling was applied in accordance with the data stratification approach described in reference. Details

[0127] In accordance with one embodiment, the Vision Transformer- Base (ViT-B) architecture is adopted as the encoder backbone for both the structural and functional unimodal encoders, referred to as ES805 (BrainHarmonix-S) and sF 806 (BrainHarmonix-F) respectively, as shown in Figure 8. During the Unimodal Encoding (UE) stage, ES805 is pretrained using a Masked Autoencoder (MAE) framework, wherein input Tl-weighted MRI volumes are randomly masked and reconstructed. 805 therefore learns to predict latent representations from which the original structural imaging data can be reconstructed. For the functional encoder eF806, pretraining is conducted using the BDFM framework, which predicts latent representations of masked targets based on a single observed region.

[0128] In BrainHarmonix-F, the neuroimaging time series data are segmented into onedimensional temporal patches, with the patch length dynamically determined based on the repetition time (TR) of the input scan. Spatial positioning of each brain region of interest (ROI) is derived by linearly projecting both geometric harmonics and functional connectivity gradients, followed by averaging the two projections to produce the final positional embeddings. Functional pretraining utilizes a Spatiotemporal Masking strategy, which divides masked targets into structured categories (e.g., Cross-ROI, Cross-Time, and Double-Cross), in accordance with the method described with reference to Figures 1 and 2.

[0129] In the Multimodal Fusion (MF) stage, the present invention employs a transformer-based Harmonizer module ( ) 808, also based on a ViT-B encoder architecture. The Harmonizer operates on a fused input stream composed of structural latents 810, functional latents 812, and a set of learnable onedimensional brain hub tokens 814. During fusion, these hub tokens are refined via self-attention and then decoded using lightweight MAE-style decoders Ds816 and 2) 818, each configured to match the embedding dimension and depth of the encoder.

[0130] Throughout both the Multimodal Fusion stage and any downstream fine-tuning tasks, the modality-specific encoders ES805 and EF806 remain frozen. These encoders serve solely to generate structural and functional latent representations, which are input into the fusion module or task-specific heads. Given BrainHarmonix's flexible ID token-based architecture, it can be extended to integrate and leverage diverse textual information, including but not limited to clinical records, demographics, genomics, blood markers, and digital tracking data, thus significantly enriching its interpretive and predictive capabilities.

[0131] Main Results

[0132] The framework of Figure 8 demonstrates strong generalization capabilities across a broad range of downstream neuroimaging tasks, including the diagnosis of neurodevelopmental and neurodegenerative disorders, as well as cognitive function prediction, as summarized in Tables 4 and 5. As the first multimodal brain foundation model, the present methodology was benchmarked against leading structure-based and function-based neuroimaging models.

[0133] For structure-based comparisons, the present methodology was evaluated against BrainMVP (Rui et al., 2024), a state-of-the-art structural foundation model originally developed for multi-parametric MRI. Due to BrainMVP's incompatibility with Tl-weighted MRI-only pretraining, its pretrained weights were adopted and fine-tuned on each downstream dataset to enable comparison. For function-based comparisons, the present invention was tested against both task-specific models— including Brain NetCNN (Kawahara et al., 2017), BrainGNN (Li et al., 2021), and BrainNetTF (Kan et al., 2022), and foundation-level functional models such as BrainMass (Yang et al., 2024), BrainLM (Ortega Caro et al., 2024), and Brain Dynamics Foundation Model (BDFM) as described above.

[0134] Notably, prior brain dynamics foundation models such as BrainLM and Brain- JEPA are not equipped to handle fMRI data with heterogeneous TRs. As a result, these models were evaluated only on datasets with uniform TRs, specifically PPMI, ADNI, and HCP-A, following their original downsampling procedures (as reflected in Table 5). BrainMass was pretrained using the same pretraining datasets as the present framework, in accordance with its original configuration.

[0135] The results demonstrate that the present framework consistently outperforms both structure-based and function-based baselines. Among the ablation models, the functional-only variant (BrainHarmonix-F) achieved superior performance relative to all tested fMRI models, underscoring the effectiveness of modelling heterogeneous temporal dynamics across large-scale functional datasets. The structural-only variant (BrainHarmonix-S) also achieved performance that is superior to, or on par with, BrainMVP, despite relying exclusively on Tl-weighted MRI. This improvement is attributed to pretraining on a significantly larger T1 dataset, enabling the encoder to capture more robust morphological representations of the brain.

[0136] Further performance gains observed following the integration of structural and functional modalities highlight the advantages of multimodal representation learning. The results support the conclusion that the present invention— by fusing structural and dynamic information— provides a more comprehensive and transferable foundation for human brain modelling.

[0137] For downstream fine-tuning, the brain hub tokens are average-pooled to generate a global multimodal representation followed by a linear projection head. The main results were all based on NH= 128 ID tokens. FlashAttention and Flastattention-2 described by Dao et al. and Dao (2022 and 2024) is used in self-attention implementation to improve computational efficiency and reduce memory usage. Each pre-training process utilized 8 NVIDIA H100 GPUs (80GB). The pretraining of Jf with 128 ID tokens took around 10 hours.

[0138] Table 1 : Comparison on neurodevelopmental disorder diagnosis. Results are averaged over three random splits. The best results are highlighted in bold (* indicates statistical significance, p < 0.05 ), and second-best results are underlined. Task details in dataset section.1UniFormer (Li et al., 2023) as backbone;2UNET3D (Ronneberger et al., 2015) as backbone only.

[0139] Table 2: Comparison on neurodegenerative disease diagnosis and cognition prediction. Task details in dataset section.1UniFormer (Li et aL, 2023) as backbone;2UNET3D (Ronneberger et al., 2015) as backbone.

[0140] The effect of varying the number of one-dimensional (ID) brain hub tokens on downstream performance is analysed under both fine-tuning and linear probing configurations. Accuracy increases consistently as the number of tokens scales from 32 to 256, before reaching a performance plateau. This trend is observed in both training regimes: full fine-tuning, involving approximately 85.25 million trainable parameters, and linear probing, which uses only 0.0015 million trainable parameters with frozen encoders.

[0141] This finding, depicted in Figure 10, highlights the scalability and efficiency of the token-based representation. Even with minimal trainable parameters in linear probing, the present invention achieves performance exceeding previous state- of-the-art models. These results emphasize the generalizability of the learned representations and the effectiveness of the ID brain hub token mechanism introduced in the present invention.

[0142] Ablation experiments were conducted to evaluate the contributions of key components of the present invention, as summarized in Figure 11. The comparison between the top line ("with data augmentation") and the bottom line ("without data augmentation") demonstrates that augmenting fMRI time series via multi-TR downsampling consistently improves model performance. This result confirms the effectiveness of the proposed Temporal Adaptive Patch Embedding (TAPE) module in enriching the model's temporal representation space.

[0143] Further comparison between the central bar ("BrainHarmonix-F") and the leftmost bar ("BrainHarmonix-F without pre-alignment") highlights the value of geometric pre-alignment. Specifically, pre-aligning fMRI signals to cortical geometry using geometric harmonics— applied in addition to brain gradient positioning, yields significant performance gains. In contrast, the ablated version relies solely on brain gradient positioning, as implemented in Brain- JEPA.

[0144] Finally, the rightmost bar ("BrainHarmonix") demonstrates the benefit of multimodal fusion, where integrating both structural and functional modalities results in the highest accuracy among all evaluated configurations. This underscores the complementary nature of structural and dynamic brain representations and the value of their joint modeling in the present invention.

[0145] It will be appreciated that many further modifications and permutations of various aspects of the described embodiments are possible. Accordingly, the described aspects are intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.

[0146] Throughout this specification and the claims which follow, unless the context requires otherwise, the word "comprise", and variations such as "comprises" and "comprising", will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps. The reference in this specification to any prior publication (or information derived from it), or to any matter which is known, is not, and should not be taken as an acknowledgment or admission or any form of suggestion that that prior publication (or information derived from it) or known matter forms part of the common general knowledge in the field of endeavour to which this specification relates.

Claims

Claims1. A computer-implemented method for generating a brain dynamics foundation model, comprising: receiving functional neuroimaging data comprising a time series of brain activity of a brain; identifying a plurality of regions of interest (ROIs) in the functional neuroimaging data, corresponding to known functional regions of the brain; generating a positional embedding based on functional connectivity gradients derived from the functional neuroimaging data corresponding to the ROIs; patchifying the time series into patches, comprising temporal patches and spatial patches; selecting an observation block comprising one or more of the patches; producing a patch-level latent representation of the observation block by encoding the observation block using an observation encoder; sampling a plurality of target regions, each target region being a said patch that is not in the observation block, the plurality of target regions including ones of said patches from each of Cross-ROI regions, Cross-Time regions and Double-Cross regions; generating the model, by training the observation encoder to predict latent representations of the sampled target regions based on the patch-level representation of the observation block and a positional encoding, of each corresponding patch, derived from the positional embedding; receiving a further functional neuroimaging data, corresponding to a task; patchifying the further functional neuroimaging data; applying a linear head to the observation encoder, to condition an output of the observation encoder on the task;updating the model over the further functional neuroimaging data; and applying the updated model to new functional neuroimaging data to predict an output, for the new functional neuroimaging data, corresponding to the task.

2. The method according to claim 1, further comprising: receiving structural neuroimaging data corresponding to the functional neuroimaging data; and aligning the structural neuroimaging data and the functional neuroimaging data by incorporating positional harmonics derived from the structural neuroimaging data to the positional embedding.

3. The method of claim 2, wherein generating the positional embedding comprises: encoding the structural neuroimaging data using a structural encoder to generate a structural latent representation; training a plurality of tokens by providing the structural latent representation and the functional latent representation as inputs to a multimodal fusion module comprising a transformer for masking portions of the structural and functional latent representations; training the plurality of tokens by reconstructing the structural neuroimaging data and functional neuroimaging data from the respective latent representations; and decoding the tokens to reconstruct original structural and functional latent representations, wherein the multimodal fusion module is trained to minimize a fusion loss defined as a sum of mean squared errors between the reconstructed and original structural and functional latent representations.

4. The method of claim 3, wherein the tokens are updated using attentionbased interactions with the structural and functional latent representations.

5. The method of claim 3 or 4, wherein the tokens are one-dimensional.

6. The method of any one of claims 3 to 5, further comprising: discarding the structural and functional decoders after pretraining; and retaining only the trained tokens as a unified representation of brain structure and function for downstream tasks.

7. The method of any one of claims 1 to 6, wherein patchifying the time series into patches comprises: determining a patch length based on a fixed temporal duration divided by the repetition time (TR) of the input scan; resizing a set of base embedding weights to match the determined patch length using a pseudoinverse linear transformation; and embedding the patches using the resized embedding weights, such that the method accommodates input data with varying TRs.

8. The method according to any one of claims 1 to 7, wherein the functional connectivity gradient comprises a plurality of attributes, each attribute representing a latent space of one or both of brain regions and networks in the functional neuroimaging data.

9. The method according to any one of claims 3 to 6, wherein the functional connectivity gradient is derived by: calculating a non-negative affinity matrix based on functional connectivity between pairs of said ROIs in the fMRI data, each pair comprising any two ROIs; andderiving the gradient using a diffusion map determined based on the non-negative affinity matrix.

10. The method according to any one of claims 1 to 9, wherein observation of the Cross-ROI regions and Cross-Time regions is spatially generalized across different ROIs or temporally generalized across timesteps.

11. The method according to any one of claims 1 to 10, wherein observation of the Double-Cross regions is generalized to unseen ROIs at unencountered timesteps.

12. The method according to any one of claims 1 to 11, comprising shuffling the ROIs prior to producing the patch-level representation.

13. The method according to any one of claims 1 to 12, wherein training the model comprises minimizing a training loss defined as an averaged L2distance between the predicted latent representations and the corresponding target latent representations.

14. A one-dimensional brain dynamics foundation model token produced according to the method of claim 5.

15. A system for generating a brain dynamics foundation model, the system comprising memory and at least one processor, the memory comprising instructions that, when executed by the at least one processor, cause the system to perform the method according to any one of claims 1 to 13.

Images