Brain foundation models adapted to downstream tasks

Abstract

A method and system are provided for adapting a pre-trained functional neuroimaging data model to a new task. The method includes producing a plurality of projected input fMRI by mapping input functional neuroimaging data data to a prompt space conditioned on a given task using a prompt mapping, wherein the input corresponds to a set of modular prompts (MoPs). One or more participant-wise phenotype prompts (PhePs) are modeled by interpolating between the modular prompts based on the projected input. Adaptation of the pre-trained model to the new task is then performed by blending a target prompt, associated with the new task, with the PhePs using the same prompt mapping.

Description

[0001] Brain Foundation Models Adapted to Downstream Tasks

[0002] Technical Field

[0003] The present invention relates generally to systems and methods for adapting pre-trained brain foundation models, such as functional magnetic resonance imaging (fMRI) language models, to new tasks. More particularly, the invention concerns prompt-based tuning of such models with high parameter efficiency and semantic interpretability, suitable for low-resource neuroimaging tasks such as disease diagnosis and personality prediction.

[0004] Background

[0005] Functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) has emerged as a non-invasive neuroimaging technique capable of capturing brain activity over time. With the proliferation of large-scale fMRI datasets and advances in machine learning, pre-trained models have been developed to extract meaningful representations from brain signals for downstream applications such as cognitive state decoding, disease prognosis, and personality trait prediction.

[0006] Fine-tuning of pre-trained fMRI models for downstream tasks has been a commonly adopted approach. However, full fine-tuning requires updating all parameters of the pre-trained model, which can be computationally expensive and prone to overfitting when only limited task-specific data is available, which is a common situation in neuroimaging domains such as rare diseases or small cohort studies. Moreover, full fine-tuning may inadvertently distort the feature space learned during pre-training, degrading generalization performance.

[0007] To address these limitations, prompt tuning has been introduced in natural language processing (NLP) as an efficient adaptation strategy that keeps the backbone model frozen while learning a small set of task-specific soft prompts. Despite its parameter efficiency, standard prompt tuning approaches often yield inferior performance compared to full fine-tuning and lack mechanisms for interpreting how the model utilizes prompt information, which limits their utility in high-stakes clinical settings.

[0008] Recent research in NLP has explored multi-task prompt tuning frameworks that aim to transfer knowledge from high-resource tasks to low-resource ones. These approaches involve initializing prompts through training on multiple source tasks. However, such frameworks often do not adequately model the complex relationship between the input and the prompt, and generally treat the prompt space as task-agnostic, which reduces their capacity to adapt meaningfully to task-specific input features. Additionally, soft prompts are generally treated as opaque embedding vectors with limited semantic interpretability.

[0009] Accordingly, there remains a need for a parameter-efficient and interpretable framework for adapting large-scale pre-trained fMRI models to downstream tasks, particularly in low-data regimes. Such a framework should be capable of transferring learned representations from high-resource domains to target tasks, while maintaining compatibility with clinical interpretability requirements and minimizing the number of parameters to be updated during training.

[0010] Summary

[0011] The present invention provides a method for generating a prompt for adapting a pre-trained brain dynamics model (f) to a new task, comprising: producing a plurality of projected input functional neuroimaging data by mapping input functional neuroimaging data corresponding to modular prompts (MoPs), to a prompt space conditioned on a task, using a first prompt mapping; modelling a plurality of participant-wise phenotype prompts (PhePs) based on the plurality of projected input functional neuroimaging data and the corresponding MoPs, by interpolating between the MoPs; and adapting f to the new task by blending a target prompt, corresponding to the new task, with the PhePs, using a second prompt mapping that maps the PhePs to a prompt space condition of the new task.

[0012] Preferably, interpolating between MoPs comprises applying weights to MoPs, each weight corresponding to a computed attention between each MoP and the respective input functional neuroimaging data.

[0013] Furthermore, blending the target prompt with the PhePs, comprises blending the target prompt with a plurality of group-wise phenotype prompts generated by the PhePs by common phenotype.

[0014] Additionally, blending the task with the PhePs comprises interpolating between the group-wise phenotype prompts and the new task, to generate a vertex prompt.

[0015] Preferably, interpolating between the group-wise phenotype prompt and the new task comprises interpolating based on the second prompt mapping.

[0016] Furthermore, the vertex prompt is a 1-dimensional token that can be appended or prepended to an input, to condition f on the new task.

[0017] Additionally, each PheP corresponds to a respective high-resource task, and adapting f to the new task comprises maximising a likelihood of labelling the new task with a label associated with a plurality of the high-resource tasks, and mapping the new task to the PhePs based on the label.

[0018] Further, adding a set of learnable deep conditioning tokens (DC) to the input functional neuroimaging data, to guide mapping of the input functional neuroimaging data to the prompt space. Preferably, an output of mapping each input functional neuroimaging data to the prompt space is fed into a projection layer and layer norm, to produce the projected input functional neuroimaging data.

[0019] Furthermore, DC is injected into each layer of f, to guide mapping of the input functional neuroimaging data to the prompt space.

[0020] Additionally, a modality of the functional neuroimaging data is functional magnetic resonance imaging (fMRI).

[0021] The present invention provides a vertex prompt generated according to the methos described above.

[0022] Another aspect of the present invention provides a system, comprising a processor and memory, the memory storing instructions that, when executed by the processor, cause the system to perform the method according to any one described above.

[0023] Brief description of the drawings

[0024] Embodiments of the present invention will now be described, by way of nonlimiting example, with reference to the drawings in which :

[0025] Figure 1 illustrates a framework of the present invention, also known as Scaffold Prompt Tuning (ScaPT), showing Source Training and Target Training stages, where prompts are generated and adapted using attention-based interpolation and a Deeply-conditioned Input-Prompt (DIP) mapping module according to an embodiment of the present invention.

[0026] Figure 2 illustrates the method for adapting a pre-trained functional magnetic resonance imaging (fMRI) language model to a new task, according to an embodiment of the present invention.

[0027] Figure 3 shows a two-dimensional visualization of phenotype prompts generated during source training, illustrating unsupervised clustering into groups corresponding to personality, social emotion, and cognition domains.

[0028] Figure 4 illustrates the distribution of attention scores between input data and phenotype prompts during target task training, providing insight into which prompt groups contribute most to model predictions.

[0029] Figure 5 shows results of an ablation study evaluating the impact of removing specific components of the ScaPT framework, including phenotype prompts, modular prompts, and the DIP mapping module.

[0030] Figure 6 presents a comparison of model performance relative to the number of trainable parameters, demonstrating the parameter efficiency of ScaPT compared to baseline methods.

[0031] Detailed description

[0032] The present invention, referred to interchangeably herein as Scaffold Prompt Tuning or ScaPT, relates to systems and methods for adapting a pre-trained model to downstream tasks using prompt-based learning techniques. For illustration purposes, the discussion below will be applied to functional magnetic resonance imaging (fMRI) language models. However, the skilled person will understand, in light of present teachings, that the same framework can be applied to data generated by other functional neuroimaging modalities including EEG, Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Positron Emission Tomography (PET), functional near-infrared spectroscopy (fNIRS) and others. The application to other imaging modalities does not require a change to the substance of the invention.

[0033] The present framework enables high-performance adaptation with limited training data by introducing only a small number of trainable parameters or prompts on top of the frozen original pre-trained fMRI language model, while offering enhanced interpretability and scalability across tasks. Thus, in some embodiments, the entire pre-trained network is frozen. In other embodiments, most of the pre-trained weight of the network are frozen, and only a small number of weights are updated. In general, downstream tasks refer to specific classification or regression objectives that rely on brain imaging data, including but not limited to neurodegenerative disease diagnosis or prognosis, and behavioural trait prediction such as personality or cognitive functions. These tasks are often characterized by limited labelled data availability, especially in clinical or longitudinal studies. The present invention addresses this challenge by enabling knowledge transfer from data-rich (high-resource) tasks to data-scarce (low- resource) target tasks.

[0034] In one or more embodiments, the invention provides a hierarchical prompt tuning framework that incorporates three types of prompts: modular prompts, phenotype prompts, and vertex prompts. Modular prompts correspond to generalizable building blocks or latent skills learned from the pre-trained fMRI model. These are interpolated based on attention mechanisms to generate participant-specific phenotype prompts, which are then averaged into group- wise prompts that reflect shared characteristics for a given behavioural or clinical phenotype.

[0035] To adapt the model to a target task, a newly initialized task-specific prompt is blended with one or more pre-trained phenotype prompts to form a vertex prompt. This vertex prompt is prepended to the input and used during the inference or training phase to guide task-specific prediction. The blending is guided by attention scores computed between the input data and the stored phenotype prompts.

[0036] To facilitate the attention calculation between input data and prompts, the invention introduces a Deeply-conditioned Input-Prompt (DIP) mapping module. This module projects the input fMRI data into a task-conditioned prompt space, a representational space in which the input can meaningfully interact with prompt embeddings., The mapping is guided by a series of learnable deep conditioning tokens injected into the layers of a frozen, pre-trained model. By conditioning the input representation on the target task, the DIP module enables precision alignment within the prompt space between the input and the most relevant prompt representations.

[0037] The framework 100 for this process is illustrated in Figure 1. The ScaPT framework 100 operates in two main stages: Source Training (ST) 102 and Target Training (TT) 104. During the Source Training stage, a plurality of trainable modular prompts 106 are generated or, in some embodiments, pretrained modular prompts are provided. The modular prompts 106 are randomly initialized parameter vectors that are trained through backpropagation using the prediction loss from source tasks. During source training, these prompts are jointly optimized across multiple phenotype prediction tasks, such as behavioural, emotional, and cognitive trait classification. Through this process, each modular prompt learns to encode fundamental, task-relevant knowledge or abstract "skills" that assist the frozen pre-trained model in generating accurate predictions. For instance, a given prompt may become specialized in recognizing neural patterns associated with attention, emotional regulation, or memory function. These skills are latent in the sense that they are not explicitly labelled or predefined but emerge through the learning process as meaningful contributors to the model's predictive capabilities.

[0038] For a given input, a plurality of modular prompts 106 are dynamically combined through weighted interpolation, wherein the interpolation weights are derived from attention scores computed between the projected input fMRI data and each modular prompt. This mechanism enables the system to generate a participantwise phenotype prompt 108, which captures phenotype-relevant features tailored to the specific subject. The resulting participant-wise phenotype prompt 108 is then used to condition the output of the frozen fMRI model / (•) , facilitating phenotype prediction for tasks such as age, sex, or behavioral trait classification. In the context of the claimed invention, an individual task refers to a distinct prediction objective where the system is required to generate a specific output based on input fMRI data. Each task is defined by its unique target label, such as behavioural phenotype, demographic attribute, or clinical diagnosis. For example, during the Source Training (ST) stage, individual tasks include predicting 38 different phenotypes, as well as age and sex, using resting-stage fMRI data from the Human Connectome Project Aging (HCP-A) dataset. In the Target Training (TT) stage, individual tasks include classification of mild cognitive impairment (MCI) versus cognitively normal (CN) individuals, detection of amyloid positivity, and regression of neuroticism scores. Although these tasks are trained and evaluated separately, they shared common input data structures and utilize a shared set of modular prompts.

[0039] Soft prompts have a tendency not to match the input embedding spaces. This makes direct attention between input and prompts unreliable. A network may be used to project inputs into prompt spaces. In general, however, such networks are inaccurate in multitasking scenarios as they project inputs uniformly. Uniform projection is task agnostic and thus fails to capture taskspecific information. Such networks also increase training parameters due to the addition of a separate network and rely on linear projections, which may not effectively represent the complex relationships between inputs and prompts.

[0040] To remedy this, some embodiments the weights for interpolation between modular prompts 106, to generate participant-wise phenotype prompts 108, are derived from attention scores computed between the input X, and the modular prompts 106. These attention scores reflect the relevance of each modular prompt 106 to a given input sample.

[0041] To support this process, the present invention incorporates a Deeply- conditioned Input-Prompt (DIP) mapping module 116. This module maps the input fMRI data to a task-conditioned prompt space using learnable deep conditioning (DC) tokens. These DC tokens are injected into layers of the pre- trained model and serve to align the input embedding with the appropriate prompt representations, enabling accurate and task-relevant interpolation. In particular, to acquire the attention mechanism, the Deeply-conditioned Input- Prompt (DIP) mapping module, denoted as M, is developed by reusing / (•) as illustrated in Figure 1. The DIP mapping module learns weights to project an input to a prompt space used by the modular prompts 106. These weights are injected at the inputs of one or more layers of / (•) to modify the output of those one or more layers. The learned weights of the DIP mapping module performing task-specific modification of the output of each of the one or more layers of / (•), without modifying the hyperparameters of / (•). More specifically, given an fMRI pre-trained model with parameters 3, and a high-resource fMRI dataset T> = {(x;,yi)} , where each data paired corresponding to N high-resource tasks T = {7), ..., Tn, ..., TN], the goal is to learn a new low-resource task Ttargetby efficiently updating parameters <p given the target task dataset D' = {(x',y')} (|D| > IT'D . The number of updating parameters <p is much smaller than that of trainable parameters 3 in the pretrained model / (•). This is achieved using attention mechanisms that inherently match the contributions made by modular prompts, and group-wise phenotype prompts, to producing an output from / ■(■). learnable tokens, herein referred to as deep conditioning cN|cne Rexm} is introduced and associated with, or prepended to, the input X, of / (•). The tokens C guide / (•) to map the input to an appropriate prompt space conditioned on the given task. The prompt space is conditioned on the given task, by the modular prompts 106 being designed to cause / (•) to perform the given task. The prediction output from / (•) is then fed into a linear projection layer and a Layer Norm LW( ) to avoid gradient explosion.

[0042] To deeply guide the conditional mapping from input to prompt spaces, the DC tokens are injected to every layer of / (•) ■ Formally, the (first) input-prompt mapping is defined as: (1) where X; e R(xmis the "text-like" representation generated from fMRI, describing the signal of each of m brain networks for each time point I, X, prepended by R<(+e)xmis input to the frozen / ■(•), e R / lis the conditional output from / ■(•), with h representing the hidden dimension, W e R / !XZlis the projection parameter to be updated during training, and H(n Qe Rhis the projected input.

[0043] The attention mechanism comprises, for each phenotype prompt 108, a number of weights equal to the number of modular prompts 106. The weights may be represented a phenotype-wise vector ak, for each phenotype prompt k, forming rows of a matrix of weights for interpolating between modular prompts 106 and phenotype prompts 108. Once trained, the tokens C are used as weights for interpolation between modular prompts 106 and phenotype prompts 108.

[0044] Subsequently, the participant-wise prompts associated with a common behavioural or clinical phenotype are averaged to produce a group-wise phenotype prompt 110. Each group-wise prompt serves as a distilled representation of high-resource task knowledge and is retained for reuse during target task adaptation. Each group-wise phenotype prompt 110 embeds knowledge specific to the respective phenotype. In particular, each group-wise phenotype wise prompt 110 is generated from the averaged weighted contributions of the modular prompts 106 that would produce the result for the corresponding group-wise phenotype when the input X;is fed to / (•). The group- wise phenotype prompts 110 therefore inherently embed the conditioning of / (• )■

[0045] In the Target Training stage, the group-wise phenotype prompts 110 generated during ST are blended with a newly randomly initialized target prompt 112, which is specific to the downstream task. The blending produces a vertex prompt 114, which is used to adapt the pre-trained model to the target task. The interpolation between the group-wise phenotype prompts 110 and the target prompt 114 is guided by attention scores 111 (a second prompt mapping) computed between the input and the stored phenotype prompts.

[0046] The attention scores 111 (i.e., weights for interpolation) are learned by appending or prepending tokens c' to X', and training c' so that the contributions of the group-wise phenotype prompts 110, when applied to X', cause the network / (•) to produce the same result a P*. The resulting P* is a one-dimensional token prompt 114 that embeds phenotype-specific and task specific information into a single prompt. When appended to prepended to X', P* causes / (•) to generate an output comparable to the output for modular prompts designed to produce the same result.

[0047] Experiment Datasets

[0048] In the Source Training (ST) stage, resting-state functional magnetic resonance imaging (fMRI) data from 656 participants in the Lifespan Human Connectome Project Aging (HCP-A) were utilized to predict 38 phenotypic traits, in addition to age and sex. For the Target Training (TT) stage, the ScaPT framework was evaluated on two classification tasks related to neurodegenerative disease diagnosis and prognosis, using data from the Alzheimer's Disease Neuroimaging Initiative (ADNI), as well as on a regression task for personality trait prediction using data from the UK Biobank (UKB). All fMRI data were preprocessed in Montreal Neurological Institute (MNI) space and parcellated into 1,024 brain networks using the DiFuMo atlas, implemented via the Nilearn library.

[0049] The TT experiments were designed under limited training data conditions. In the first classification task, distinguishing between Control Normal (CN) and Mild Cognitive Impairment (MCI), 340 ADNI participants were used, evenly split between the two classes (170 each). An independent test set comprising 100 CN and 100 MCI participants was allocated. From the remaining 140 participants, training sets of 3, 5, or 10 samples per class were randomly selected, with the remainder used for validation. Each sampling condition was repeated 10 times, and performance was reported in terms of mean accuracy and Fl score with standard deviation. The second classification task involved distinguishing between amyloid-positive and amyloid-negative individuals, based on a cohort of 100 cognitively normal participants from ADNI (50 per class). The same sampling and evaluation procedure was applied, with a test set consisting of 25 participants per class. For the regression task, neuroticism scores— ranging from 0 to 12 and normalized to a 0-1 scale— were predicted using data from 1,000 participants in the UKB. Of these, 800 participants were used for training and 200 for testing. Training subsets of size 30, 50, and 100 were randomly selected from the training set under the same conditions as the classification tasks.

[0050] The present framework utilizes a fMRI language model employing a causal sequence modelling architecture (see, e.g., Thomas et al., Advances in Neural Information Processing Systems, vol. 35, pp. 21255-21269, 2022) for downstream task adaption. The language model was pre-trained using a large- scale dataset comprising 11,980 fMRI runs of 1,726 individuals across 34 datasets. The pretrained model contains 4 GPT-2 layers, with 12 attention heads in each self-attention module. To utilize the pre-trained model, the input must be parcellated by Dictionaries of Functional Modes (DiFuMo) atlas. The pre- processed input to the model is represented as a matrix X e R.(xmobtained by DiFuMo, where I is the input sequence length and m = 1024 networks. The hidden dimension in the model is h = 768. The number of modular prompts K = 5. Both prompt length d and the DC length e is set to d = e = 5.

[0051] Source Training (ST)

[0052] A healthy cohort comprises a large number of participants, each associated with various phenotypic attributes corresponding to high-resource tasks. In the first stage, Source Training (ST), the objective is to train a set of phenotype prompts (PheP) that encapsulate behaviourally relevant brain phenotypes. These trained phenotype prompts serve as a transferable source of knowledge for subsequent downstream tasks.

[0053] To more effectively capture the relationship between input fMRI data and the prompts, and to enhance the capacity of the PheP to reflect the complexity of the input, each participant-wise PheP 108 is modelled as an interpolation of a set of modular prompts (MoP) 106, where the interpolation weights are determined by the attention scores 109 computed between the input 107 and the respective MoPs 106, as illustrated in Figure 1. Each MoP can be seen as a basic skill; solving a task involves combining these fundamental skills. Formally, for a high-resource tas the training objective is to maximize the likelihood of predicting the label y(ni0as follows: where P(n0 e Rdxmis the participant-wise PheP with length d for task Tn. X, prepended by the prompt [P^yX,] e R(I+d)xmis input to the frozen / (■) to predict P(n.i) is generated from the interpolation of K MoP : where the attention score 109 between is computed

[0054] (maxpool of Pfcfrom Rdxmto Rm, followed by a linear transformation to Rft) and is the temperature. P(n,i) is a participant-wise prompt for X;. To transfer the learned prompts to the next stage, P(n>i)is averaged to formulate one group- wise PheP P„, which is utilized subsequently for next stage.

[0055] Target Training (TT)

[0056] Prompts for a new task 112 could be blended with pretrained prompts 110 to incorporate gained knowledge. In the second stage - Target Training (TT), a target prompt Ptargettailored for a target task is first initialized. To capitalize on the insights embedded in Pn, a vertex prompt P* 114 is learned, by interpolating Pnand Ptarget given attention computed by M as shown in Figure 1. Similar to ST, the goal of TT is to maximize the likelihood of predicting the correct target task label y', given the concatenation of P* 114 and input X' 113 that the new task Ptarget is designed to operate on: max

[0057] Ptarget where P*114 is the interpolation of Ptarget and Pn-Pn is the attention weights or score 111 between X' 113 and the Pncomputed by M . Thus, each PheP corresponds to a respective high-resource task, and the goal is to adapt / (•) to the new task by finding some combination of the PhePs - e.g., by interpolation - that responds to the new task. That combination is found by maximizing the likelihood of labelling the new task with a label associated with the high-resource tasks, and mapping the new task to the PhePs based on that label.

[0058] In other words, the blending of the group-wise phenotype prompts 110 and the task specific target prompt 112 is governed by attention weights 111 computed between the input fMRI data, X' 113 and the stored group-wise phenotype prompts 110. The attention mechanism is configured to evaluate the relevance of each phenotype prompt to a given input, thereby generating a set of attention weights 111. The vertex prompt P* 114 is then generated by interpolating between the group-wise phenotype prompts 110 and the target prompt according to the computed attention weights 111. The vertex prompt is a 1- dimensional (ID) prompt that conditions the pre-trained model / (•) on a given task, Ptarget- This ID token-based architecture can be extended to integrate and leverage diverse textual information, including but not limited to clinical records, demographics, genomics, blood markers, and digital tracking data, thus significantly enriching its interpretive and predictive capabilities. In the TT stage, the objective is to adapt the pre-trained fMRI language model to a new, low-resource task by predicting a target label y(nfor participant i under task n . This is accomplished through the construction of a vertex prompt is then prepended to the input X', ch is subsequently fed into the frozen es this input and produces a predicted output y(n,i), representing the estimated label for the target task. The parameters remain fixed during this process, and only the prompts and projection layers are trained, to enable parameter-efficient adaptation.

[0059] Figure 2 illustrates the method for adapting a pre-trained functional magnetic resonance imaging (fMRI) language model to a new task. In step 202, a plurality of projected input fMRI representations is produced by mapping input fMRI data, corresponding to a set of MoPs as described above, into a task-conditioned prompt space using a prompt mapping or using learnable DC tokens of the DIP mapping module. In step 204, one or more PhePs are modelled based on the projected input fMRI and the corresponding MoPs, wherein the PhePs are generated by interpolating between the MoPs using attention-based weights derived from the input. Interpolating between MoPs comprises applying weights to MoPs, each weight corresponding to a computed attention between each MoP and the respective input fMRI as described above. In step 206, the pre-trained fMRI language model is adapted to a new task by constructing a vertex prompt through the blending of a task-specific target prompt with the previously generated PhePs.

[0060] Main Results

[0061] Under limited training data conditions, the performance of the ScaPT framework was evaluated in comparison with conventional fine-tuning and three multitaskbased prompt tuning approaches: SPoT, MP2, and ATTEMPT, as summarized in Tables 1 and 2. For fine-tuning, (•) underwent direct fine-tuning using TT datasets (ADNI / UKB). Meanwhile, for the multitask-based methods, prompts were initially trained on ST datasets (HCPA), which were then served as the prompt initialization in the TT stage.

[0062] The present framework, ScaPT, demonstrated superior performance over both fine-tuning and other prompt tuning methods across various sizes of training datasets, scaling well with the number of training data. This underscores ScaPT's effectiveness in transferring knowledge from high-resource tasks to those with scarce resources.

[0063] Table 1. Accuracy (Acc) and Fl score on CN v.s. MCI and Amyloid ap + ve v.s. a0 - ve classification (%, mean(standard deviation) for 10 independent runs), trained on varying training dataset size (I = 3, 5, 10 per class). The best results are in bold, with * denoting significant improvement (p < 0.05). Table 2. Mean Absolute Error (MAE,x lO -1) and Pearson Correlation (p) on Neuroticism score prediction.

[0064] A total of 40 phenotype prompts were generated during the Source Training (ST) stage, corresponding to 38 phenotype prediction tasks, as well as age and sex classification. Notably, these prompts exhibited an emergent clustering structure, forming three distinct groups associated with Personality, Social Emotion, and Cognition domains, despite the absence of prior supervision. This observation suggests that the phenotype prompts effectively capture different pillars of brain-behaviour associations as shown in Figure 3.

[0065] During the TT stage, the attention scores between input and phenotype prompts shown in Figure 4 aid in interpreting the target task. Attention score vectors were averaged across inputs, and then attributes within each group were averaged (without the one for Ptarget). These five values were normalized for analysis. Aligning well with the literature, ScaPT shows a focus on "cognition" or "age" in neurodegenerative disease diagnosis / prognosis task (t-test on age / sex in training set for two classes yielded a p-value greater than 0.05, indicating no significant difference), while it focuses on "personality" for neuroticism score prediction.

[0066] As shown in Figure 5, the performance of ScaPT was evaluated through a series of ablation studies, which included : (i) ScaPT without PheP, in which modular prompts (MoP) were used directly for vertex prompt (P*) formulation without prior training on high-resource tasks; (ii) ScaPT without MoP, where the group- wise phenotype prompts (Pn) were learned without applying prompt width expansion; and (iii) ScaPT in which the DIP mapping module was replaced with the ATTEMPT subnetwork architecture for input-prompt mapping. The absence of PheP led to a significant performance drop, underscoring the importance of high-resource task knowledge in boosting low-resource task performance. ScaPT outperformed its counterparts lacking MoP, demonstrating MoP's role in enhancing expressive capacity by widening prompts. Additionally, ScaPT's DIP module surpassed ATTEMPT'S subnetwork in mapping inputs to prompts, effectively capturing complex input-prompt relationships using / (•).

[0067] As illustrated in Figure 6, the performance of ScaPT was compared against other models with respect to the number of trainable parameters. ScaPT achieved superior performance relative to fine-tuning, MP2, and ATTEMPT, despite requiring updates to only approximately 2% of the total model parameters. Although SPoT exhibited the lowest number of trainable parameters among the evaluated methods, its performance was notably lower, which may be attributed to its limited capacity for capturing complex input features.

[0068] In another aspect, the present framework provides a system for adapting a pretrained functional magnetic resonance imaging (fMRI) language model to a new task. The system comprises at least one processor and a memory storing computer-readable instructions which, when executed by the processor, cause the system to perform the steps of the method as described in any of the embodiments herein. These steps include mapping input fMRI data into a task- conditioned prompt space, modelling participant-wise phenotype prompts based on modular prompts and attention-based interpolation, and adapting the model to a target task by generating a vertex prompt through blending of a target prompt and phenotype prompts.

[0069] The processor may be implemented as a central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU), or any combination thereof. The memory may comprise non-transitory computer-readable media including RAM, ROM, flash storage, or any persistent storage medium suitable for storing the instructions, model parameters, and input data. The system may further comprise interfaces for receiving neuroimaging data and outputting predicted labels or scores associated with downstream tasks.

[0070] It will be appreciated that many further modifications and permutations of various aspects of the described embodiments are possible. Accordingly, the described aspects are intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.

[0071] Throughout this specification and the claims which follow, unless the context requires otherwise, the word "comprise", and variations such as "comprises" and "comprising", will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps. The reference in this specification to any prior publication (or information derived from it), or to any matter which is known, is not, and should not be taken as an acknowledgment or admission or any form of suggestion that that prior publication (or information derived from it) or known matter forms part of the common general knowledge in the field of endeavour to which this specification relates.

Claims

Claims1. A method for generating a prompt for adapting a pre-trained brain dynamics model (f) to a new task, comprising: producing a plurality of projected input functional neuroimaging data by mapping input functional neuroimaging data corresponding to modular prompts (MoPs), to a prompt space conditioned on a task, using a first prompt mapping; modelling a plurality of participant-wise phenotype prompts (PhePs) based on the plurality of projected input functional neuroimaging data and the corresponding MoPs, by interpolating between the MoPs; and adapting f to the new task by blending a target prompt, corresponding to the new task, with the PhePs, using a second prompt mapping that maps the PhePs to a prompt space condition of the new task.

2. The method according to claim 1, wherein interpolating between MoPs comprises applying weights to MoPs, each weight corresponding to a computed attention between each MoP and the respective input functional neuroimaging data.

3. The method according to any one of claims 1 or 2, wherein blending the target prompt with the PhePs, comprises blending the target prompt with a plurality of group-wise phenotype prompts generated by the PhePs by common phenotype.

4. The method according to claim 3, wherein blending the task with the PhePs comprises interpolating between the group-wise phenotype prompts and the new task, to generate a vertex prompt.

5. The method according to claim 4, wherein interpolating between the group-wise phenotype prompt and the new task comprises interpolating based on the second prompt mapping.

6. The method according to claim 4 or 5, wherein the vertex prompt is a 1- dimensional token that can be appended or prepended to an input, to condition f on the new task.

7. The method according to any one of claims 1 to 6, wherein each PheP corresponds to a respective high-resource task, and adapting f to the new task comprises maximising a likelihood of labelling the new task with a label associated with a plurality of the high-resource tasks, and mapping the new task to the PhePs based on the label.

8. The method according to any one of claims 1 to 7, comprising adding a set of learnable deep conditioning tokens (DC) to the input functional neuroimaging data, to guide mapping of the input functional neuroimaging data to the prompt space.

9. The method according to claim 8, wherein an output of mapping each input functional neuroimaging data to the prompt space is fed into a projection layer and layer norm, to produce the projected input functional neuroimaging data.

10. The method according to any one of claims 8 or 9, wherein DC is injected into each layer of f, to guide mapping of the input functional neuroimaging data to the prompt space.

11. The method according to any one of claims 1 to 10, wherein a modality of the functional neuroimaging data is functional magnetic resonance imaging (fMRI).

12. A vertex prompt generated according to the method of claim 4.

13. A system, comprising a processor and memory, the memory storing instructions that, when executed by the processor, cause the system to perform the method according to any one of claims 1 to 10.

Images