Large language model with brain processing tools

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining a mental state of a patient based on a natural language input and determining whether a relevant subset of brain data is anomalous. One of the methods includes receiving a natural language input describing at least one aspect of a mental state or of a behavior of an individual; generating a prompt based on at least in part the natural language input; submitting the prompt to a large language model; receiving at least one functional network that influences the at least one aspect of a mental state or of a behavior; for each network of the at least one functional network, analyzing MRI data for the individual to determine whether the network is anomalous; displaying to a user each network and whether it is anomalous; and taking an action in response to the displaying.

Description

[0001] LARGE LANGUAGE MODEL WITH BRAIN PROCESSING TOOLS

[0002] CROSS-REFERENCE TO RELATED APPLICATION

[0003] This application claims the of priority to U.S. Application No. 18 / 540,775, filed on December 14, 2023, the contents of which are hereby incorporated by reference.

[0004] BACKGROUND

[0005] Technical Field

[0006] This specification relates to processing data related to a subject’s brain using large language models. The present invention also relates to a method and apparatus for processing data related to a subject’s brain, and to a computer program product including a computer readable medium having recorded thereon a computer program for processing data related to a subject’s brain.

[0007] Background

[0008] Brain functional connectivity data characterizes, for each of one or more pairs of locations within the brain of a patient, the degree to which brain activity in the pair of locations is correlated.

[0009] One can gather data related to the brain of the patient by obtaining and processing images of the brain of the patient, e.g., using magnetic resonance imaging (MRI).

[0010] Data related to the brain of a single patient can be highly complex and highdimensional, and therefore difficult for a clinician to manually inspect and parse, e.g., to plan a surgery or diagnose the patient for a brain disease or mental disorder.

[0011] Large language models (LLMs) are a type of machine learning model that can be trained to receive a natural human language query and generate text and / or images in response to the query.

[0012] SUMMARY

[0013] This specification describes technologies for determining a mental state or behavior of a patient that is of interest based on a natural language input and then determining whether a relevant subset of brain data is anomalous. These technologies generally involve processing a natural language input that describes aspects of the mental state or behavior of a patient using a large language model to identify and display regions of the patient’s brain that may have abnormal activity levels.

[0014] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of receiving a natural language input describing at least one aspect of a mental state or of a behavior of an individual; generating a prompt based at least in part the natural language input, the prompt configured to generate at least one functional network that influences the at least one aspect of a mental state or of a behavior above a threshold; submitting the prompt to a large language model; receiving an indication of at least one functional network from the large language model that influences the at least one aspect of a mental state or of a behavior; for each network of the at least one functional network, analyzing MRI data for the individual to determine whether the network is anomalous; displaying to the user each network of the at least one functional network and whether it is anomalous; and taking an action in response to the displaying. At least one aspect of a mental state or of a behavior can include more than one aspect of mental state(s), more than one aspect of behavior(s), or a combination of aspect(s) of mental state(s) and aspect(s) of behavior(s).

[0015] Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0016] The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. In particular, one embodiment includes all the following features in combination.

[0017] In some implementations, the method further comprises receiving a user input in response to the displaying, wherein the response includes selection of at least one functional network to produce at least one selected functional network; and taking an action comprises submitting to a large language model a request for how to address an anomaly in the at least one selected functional network.

[0018] In some implementations, taking an action further comprises generating a plurality of treatment plans for how to address the anomaly in the at least one selected functional network.

[0019] In some implementations, a treatment plan comprises (i) surgeries, (ii) behavioral therapies, (iii) brain editing techniques, or a combination thereof.

[0020] In some implementations, the natural language input comprises one or more questions.

[0021] In some implementations, the method further comprises in response to submitting the prompt to the large language model, receiving a mental state or a behavior.

[0022] In some implementations, the method further comprises displaying the mental state or behavior along with an indication of whether that mental state or behavior is anomalous.

[0023] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. A set of brain data characterizing the brain of a single patient can often be incredibly large and complicated, and thus it can be difficult and time consuming for a user to extract useful information from the set of brain data. Using techniques described in this specification, a system can quickly identify one or more networks in a patient’s brain that may have abnormal activity and that influence the mental state or behavior of a patient. The system can then display data characterizing anomalies in the patient’s brain activity the user, so that the user is not forced to search through and analyze a large amount of data that is not clinically relevant. Therefore, the amount of time that a user must spend to discover the portion of the brain data that is useful can be drastically reduced, resulting in improved outcomes for patients, users and / or clinicians, especially when effective care requires time sensitive investigations.

[0024] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figs. 1 A and IB form a schematic block diagram of a computer system upon which arrangements described can be practiced.

[0026] FIG. 1C is a block diagram of an example mental state determination system.

[0027] FIG. 2A and 2B show a window of a graphical user interface showing a graphical representation of a treatment plan in response to an anomaly detected in a subject.

[0028] FIG. 3 shows another window of a graphical user interface showing a template to submit a natural language input and generate a treatment plan.

[0029] FIG. 4 shows the window of FIG. 3 with a submitted natural language input.

[0030] FIG. 5 shows the window of FIG. 3 with a selected anomaly.

[0031] FIG. 6A and FIG. 6B are diagrams of an example connectivity data system.

[0032] FIG. 7 illustrates example connectivity matrices.

[0033] FIG. 8 illustrates an example anomaly connectivity matrix.

[0034] FIG. 9 is a flow diagram of an example method for determining a mental state of a patient that is of interest based on a natural language input and then determining whether a relevant subset of brain data is anomalous.

[0035] FIG. 10 a flowchart of an example process for determining an anomalous subset of brain data.

[0036] Like reference numbers and designations in the various drawings indicate like elements.

[0037] DETAILED DESCRIPTION

[0038] A computing device can perform the arrangements described. Figs. 1A and IB depict a computer system 100, upon which one can practice the various arrangements described.

[0039] As seen in Fig. 1A, the computer system 100 includes: a computer module 101; input devices such as a keyboard 102, a mouse pointer device 103, a scanner 126, a camera 127, and a microphone 180; and output devices including a printer 115, a display device 114 and loudspeakers 117. An external Modulator-Demodulator (Modem) transceiver device 116 may be used by the computer module 101 for communicating to and from a communications network 120 via a connection 121. The communications network 120 may be a wide-area network (WAN), such as the Internet, a cellular telecommunications network, or a private WAN. Where the connection 121 is a telephone line, the modem 116 may be a traditional “dial-up” modem. Alternatively, where the connection 121 is a high capacity (e.g., cable) connection, the modem 116 may be a broadband modem. A wireless modem may also be used for wireless connection to the communications network 120.

[0040] The computer module 101 typically includes at least one processor unit 105, and a memory unit 106. For example, the memory unit 106 may have semiconductor random access memory (RAM) and semiconductor read only memory (ROM). The computer module 101 also includes an number of input / output (I / O) interfaces including: an audiovideo interface 107 that couples to the video display 114, loudspeakers 117 and microphone 180; an RO interface 113 that couples to the keyboard 102, mouse 103, scanner 126, camera 127 and optionally a joystick or other human interface device (not illustrated); and an interface 108 for the external modem 116 and printer 115. In some implementations, the modem 116 may be incorporated within the computer module 101, for example within the interface 108. The computer module 101 also has a local network interface 111, which permits coupling of the computer system 100 via a connection 123 to a local-area communications network 122, known as a Local Area Network (LAN). As illustrated in Fig. 1A, the local communications network 122 may also couple to the wide network 120 via a connection 124, which would typically include a so-called “firewall” device or device of similar functionality. The local network interface 111 may comprise an Ethernet circuit card, a Bluetooth® wireless arrangement or an IEEE 802.11 wireless arrangement; however, numerous other types of interfaces may be practiced for the interface 111.

[0041] The module 101 can be connected with an image capture device 197 via the network 120. The device 197 can capture images of a subject brain using each of diffusor tension imaging and magnetic resonance imaging (MRI) techniques. The captured images are typically in standard formats such as DICOM format and OpenfMRI format respectively. The module 101 can receive DTI and MRI images the device 197 via the network 120. Alternatively, the DTI and MRI images can be received by the module 101 from a remote server, such as a cloud server 199, via the network 120. In other arrangements, the module 101 may be an integral part of one of the image capture device 197 and the server 199. The I / O interfaces 108 and 113 may afford either or both of serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standards and having corresponding USB connectors (not illustrated). Storage devices 109 are provided and typically include a hard disk drive (HDD) 110. Other storage devices such as a floppy disk drive and a magnetic tape drive (not illustrated) may also be used. An optical disk drive 112 is typically provided to act as a non-volatile source of data. Portable memory devices, such optical disks (e.g., CD-ROM, DVD, Blu ray DiscTM), USB- RAM, portable, external hard drives, and floppy disks, for example, may be used as appropriate sources of data to the system 100.

[0042] The components 105 to 113 of the computer module 101 typically communicate via an interconnected bus 104 and in a manner that results in a conventional mode of operation of the computer system 100 known to those in the relevant art. For example, the processor 105 is coupled to the system bus 104 using a connection 118. Likewise, the memory 106 and optical disk drive 112 are coupled to the system bus 104 by connections 119. Examples of computers on which the described arrangements can be practised include IBM-PC’s and compatibles, Sun Sparcstations, Apple Mac™ or like computer systems.

[0043] The method described may be implemented using the computer system 100 wherein the processes of Figs. 3 to 5, to be described, may be implemented as one or more software application programs 133 executable within the computer system 100. In particular, the steps of the method described are effected by instructions 131 (see Fig. IB) in the software 133 that are carried out within the computer system 100. The software instructions 131 may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules performs the described methods and a second part and the corresponding code modules manage a user interface between the first part and the user.

[0044] The software may be stored in a computer readable medium, including the storage devices described below, for example. The software is loaded into the computer system 100 from the computer readable medium, and then executed by the computer system 100. A computer readable medium having such software or computer program recorded on the computer readable medium is a computer program product. The use of the computer program product in the computer system 100 preferably effects an advantageous apparatus for providing a display of a neurological image.

[0045] The software 133 is typically stored in the HDD 110 or the memory 106. The software is loaded into the computer system 100 from a computer readable medium, and executed by the computer system 100. Thus, for example, the software 133 may be stored on an optically readable disk storage medium (e.g., CD-ROM) 125 that is read by the optical disk drive 112. A computer readable medium having such software or computer program recorded on it is a computer program product. The use of the computer program product in the computer system 100 preferably effects an apparatus for providing a display of a neurological image.

[0046] In some instances, the application programs 133 may be supplied to the user encoded on one or more CD-ROMs 125 and read via the corresponding drive 112, or alternatively may be read by the user from the networks 120 or 122. Still further, the software can also be loaded into the computer system 100 from other computer readable media. Computer readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to the computer system 100 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, DVD, Blu-ray™ Disc, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computer module 101. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and / or data to the computer module 101 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like.

[0047] The second part of the application programs 133 and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the display 114. Through manipulation of typically the keyboard 102 and the mouse 103, a user of the computer system 100 and the application may manipulate the interface in a functionally adaptable manner to provide controlling commands and / or input to the applications associated with the GUI(s). Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilizing speech prompts output via the loudspeakers 117 and user voice commands input via the microphone 180.

[0048] Fig. IB is a detailed schematic block diagram of the processor 105 and a “memory” 134. The memory 134 represents a logical aggregation of all the memory modules (including the HDD 109 and semiconductor memory 106) that can be accessed by the computer module 101 in Fig. 1A.

[0049] When the computer module 101 is initially powered up, a power-on self-test (POST) program 150 executes. The POST program 150 is typically stored in a ROM 149 of the semiconductor memory 106 of Fig. 1 A. A hardware device such as the ROM 149 storing software is sometimes referred to as firmware. The POST program 150 examines hardware within the computer module 101 to ensure proper functioning and typically checks the processor 105, the memory 134 (109, 106), and a basic input-output systems software (BIOS) module 151, also typically stored in the ROM 149, for correct operation. Once the POST program 150 has run successfully, the BIOS 151 activates the hard disk drive 110 of Fig. 1A. Activation of the hard disk drive 110 causes a bootstrap loader program 152 that is resident on the hard disk drive 110 to execute via the processor 105. This loads an operating system 153 into the RAM memory 106, upon which the operating system 153 commences operation. The operating system 153 is a system level application, executable by the processor 105, to fulfil various high level functions, including processor management, memory management, device management, storage management, software application interface, and generic user interface.

[0050] The operating system 153 manages the memory 134 (109, 106) to ensure that each process or application running on the computer module 101 has sufficient memory in which to execute without colliding with memory allocated to another process. Furthermore, the different types of memory available in the system 100 of Fig. 1 A must be used properly so that each process can run effectively. Accordingly, the aggregated memory 134 is not intended to illustrate how particular segments of memory are allocated (unless otherwise stated), but rather to provide a general view of the memory accessible by the computer system 100 and how such is used. As shown in Fig. IB, the processor 105 includes a number of functional modules including a control unit 139, an arithmetic logic unit (ALU) 140, and a local or internal memory 148, sometimes called a cache memory. The cache memory 148 typically includes a number of storage registers 144 - 146 in a register section. One or more internal busses 141 functionally interconnect these functional modules. The processor 105 typically also has one or more interfaces 142 for communicating with external devices via the system bus 104, using a connection 118. The memory 134 is coupled to the bus 104 using a connection 119.

[0051] The application program 133 includes a sequence of instructions 131 that may include conditional branch and loop instructions. The program 133 may also include data 132 which is used in execution of the program 133. The instructions 131 and the data 132 are stored in memory locations 128, 129, 130 and 135, 136, 137, respectively. Depending upon the relative size of the instructions 131 and the memory locations 128-130, a particular instruction may be stored in a single memory location as depicted by the instruction shown in the memory location 130. Alternately, an instruction may be segmented into a number of parts each of which is stored in a separate memory location, as depicted by the instruction segments shown in the memory locations 128 and 129.

[0052] In general, the processor 105 is given a set of instructions which are executed therein. The processor 105 waits for a subsequent input, to which the processor 105 reacts to by executing another set of instructions. Each input may be provided from one or more of a number of sources, including data generated by one or more of the input devices 102, 103, data received from an external source across one of the networks 120, 102, data retrieved from one of the storage devices 106, 109 or data retrieved from a storage medium 125 inserted into the corresponding reader 112, all depicted in Fig. 1 A. The execution of a set of the instructions may in some cases result in output of data. Execution may also involve storing data or variables to the memory 134.

[0053] The described arrangements use input variables 154, which are stored in the memory 134 in corresponding memory locations 155, 156, 157. The described arrangements produce output variables 161, which are stored in the memory 134 in corresponding memory locations 162, 163, 164. Intermediate variables 158 may be stored in memory locations 159, 160, 166 and 167. Referring to the processor 105 of Fig. IB, the registers 144, 145, 146, the arithmetic logic unit (ALU) 140, and the control unit 139 work together to perform sequences of microoperations needed to perform “fetch, decode, and execute” cycles for every instruction in the instruction set making up the program 133. Each fetch, decode, and execute cycle comprises: a fetch operation, which fetches or reads an instruction 131 from a memory location 128, 129, 130; a decode operation in which the control unit 139 determines which instruction has been fetched; and an execute operation in which the control unit 139 and / or the ALU 140 execute the instruction.

[0054] Thereafter, a further fetch, decode, and execute cycle for the next instruction may be executed. Similarly, a store cycle may be performed by which the control unit 139 stores or writes a value to a memory location 132.

[0055] Each step or sub-process in the processes of FIG. 6 is associated with one or more segments of the program 133 and is performed by the register section 144, 145, 147, the ALU 140, and the control unit 139 in the processor 105 working together to perform the fetch, decode, and execute cycles for every instruction in the instruction set for the noted segments of the program 133.

[0056] FIG. 1C is a block diagram of an example mental state determination system 170.

[0057] The mental state determination system 170 includes a prompt engineering engine 174, a large language model 180, and a brain imaging system 178.

[0058] The prompt engineering engine 174 receives a natural language input 172 and generates a prompt 176. The natural language input 172 can describe at least one aspect of a mental state or of a behavior of an individual. The natural language input can be any description of one or more aspects of an individual’s mental state or of an individual’s behavior in any human language. For example, the natural language input can be a question or a statement. The prompt 176 can be any prompt that is configured to generate at least one functional network that influences the at least one aspect of a mental state or of a behavior above a threshold and can be submitted to a large language model. The prompt 176 can be based at least in part on the natural language input. Generating the prompt 176 is described in further detail below with reference to FIG. 3.

[0059] The large language model 180 receives the prompt 176 from the prompt engineering engine 174 and processes the prompt to generate one or more functional networks 182. The large language model 180 can be any appropriate neural network that receives an input sequence made up of text tokens selected from a vocabulary and auto-regressively generates an output sequence made up of text tokens from the vocabulary. For example, the large language model 180 can be a Transformer-based language model neural network or a recurrent neural network-based language model neural network.

[0060] The one or more functional networks 182 can be any functional networks that influence the at least one aspect of the mental state or of the behavior. Each functional network can include several smaller networks.

[0061] The brain imaging system 182 receives the one or more functional networks 182 from the large language model 180 and generates a display 184. For each network of the one or more functional networks, the brain imaging system 178 analyses MRI data for the individual to determine whether the network in anomalous. The network can be considered to be anomalous if the MRI data of the individual relevant to the network is outside a normal range, e.g., as defined by the brain data of other patients. The normal range can be, for example, defined by a standard deviation from the median of the distribution, defined by a range of values that cover a percentage (e.g., 95%) of the population, or defined by a distance from the mean of the distribution.

[0062] The display 184 can display each network of the at least one functional network and whether it is anomalous. The display can be a window of a graphical user interface such as the graphical user interfaces in FIGS. 2A - 5.

[0063] FIGS. 2A and 2B show a graphical user interface showing a graphical representation of a treatment plan in response to an anomaly detected in a subject. A variety of techniques can be used for anomaly detection. For example, anomaly detection techniques described in US patent No. 11,087,877, listing Michael Sughrue, Stephane Doyen and Peter Nicholas as inventors, entitled “identifying anomalous brain data” and incorporated herein by reference in its entirety, can be used for anomaly detection. The window 200 includes six tabs 232 that each represent a system of the human brain. A user can select a tab to view details regarding the mental state or behavior of an individual associated with the system represented by the tab. The six tabs represent “The Decision-Making System”, “The Fear& Worry System”, “The Satisfaction System”, “The Sensation & Movement System”, “The Social System”, and “The Sleep System” as described in the research domain criteria (RDoC) framework. The RDoC framework organizes the human brain into different domains. In this example, a user has selected “The Social System” tab 202.

[0064] Each system breaks down further into a set of constructs associated with the system. A user can select a construct associated the with selected tab from a construct menu 204 to view details regarding the selected construct. For example, according to the RDoC Framework, the constructs associated with the “The Social System” tab 102 include “Affiliation and Attachment”, “Social Communication”, “Perception and Understanding of Self’, and “Perception and Understanding of Others”. In this example, a user has selected to view details regarding “Affiliation and Attachment”.

[0065] Once the user has selected a system from the six tabs 232 and a construct from the construct menu 204, the window displays an assessment 214 of brain circuits for the individual based on a subset of brain data for the individual regarding the selected construct. The brain data can be any data characterizing the brain of a patient. For example, brain data can include one or both of i) direct measurement data of the brain of the patient, e.g., images of the brain collected using brain imaging techniques, or ii) data that has been derived or generated from initial measurement data of the brain of the patient, e.g., correlation matrices.

[0066] The assessment 214 displays a description 216 of the brain circuit that is associated with the construct. In this example, the description 216 associated with the brain circuit for “Affiliation and Attachment” reads “The ability to form relationships with other people”.

[0067] The assessment 214 displays whether the subset of brain data for the individual indicates that the subset of brain data shows an anomaly 220 or does not show an anomaly 218. The subset of brain data is considered to show an anomaly, e.g., if a subset of brain data of the patient is outside a normal range, e.g., as defined by the brain data of other patients. The normal range can be, for example, defined by a standard deviation from the median of the distribution, defined by a range of values that cover a percentage (e.g., 95%) of the population, or defined by a distance from the mean of the distribution. Determining that a subset of brain data shows an anomaly is described in further detail below with reference to FIGS. 6 A and 6B.

[0068] Additionally, with reference to FIGS. 2 A and 2B, the assessment 214 displays anomaly data 222, 224, 226, 228, and 230 for the individual that shows activity levels for different portions of the brain. In this example, the assessment 214 displays four brain images 222, 224, 226, and 228 that show regions of abnormal activity as well as a table 230 that describes activity levels for different parcels of the brain as well as a summary for each parcel.

[0069] The brain images 222, 224, 226, and 228 are discretized using a brain map to reduce the brain data into smaller portion related to specific behaviors for the particular brain circuit e.g., the language region, the motion region, etc. The regions for which the quantity of activation is outside a normal range when compared across a proprietary database of brain data captured from hundreds, thousands, or millions of other individuals are highlighted in the assessment 214.

[0070] The table 230 organizes the individual’s brain data according to multiple different parcellations of the individual’s brain. For example, the brain data can include multiple different time series characterizing the activity of a respective different region of the brain of the individual over time, e.g., a time series corresponding to each three-dimensional voxel of the brain that can be measured by an MRI machine. The table 230 can organize the different time series signals by parcellation according to a brain atlas of the brain. A brain atlas is data that defines one or more parcellations of a brain of a patient, e.g., by defining in a common three-dimensional coordinate system the coordinates of the outline of the parcellation or the volume of the parcellation.

[0071] The window also displays an analysis 206 for the individual based on the individual’s description 208 of their own mental state or behavior as well as a treatment plan 212 for the individual. In this example, the individual’s description 208 indicates that they are less attached to other people than most people. The individual’s description is a subjective description and can be obtained as a natural language description or as an answer to a survey.

[0072] The analysis 206 also displays symptoms 210 that the individual may experience as well as how the individual compares to others 208. In this example, symptoms for having lower affiliation and attachment with most people are described as “difficulty forming close relationships, which can manifest as challenges building romantic, platonic, and work relationships”.

[0073] The treatment plan 212 describes several possible actions that the individual can take to improve symptoms 210 that the individual may be experiencing 208. In this example, the treatment plan 212 suggests “treatments such as Cognitive Behavioral Therapy (CBT) or Attachment-Based Family Therapy (ABFT) may be indicated”.

[0074] A large language model generates the treatment plan 212. A large language model (“LLM”) is a model that is trained to process and generate human language. LLMs are trained on massive datasets of text and code, and they can be used for a variety of tasks.

[0075] The large language model can be any appropriate neural network that receives an input sequence made up of text tokens selected from a vocabulary and auto-regressively generates an output sequence made up of text tokens from the vocabulary. For example, the large language model can be a Transformer-based language model neural network or a recurrent neural network-based language model neural network.

[0076] In some situations, the large language model can be referred to as an auto-regressive neural network when the neural network used to implement the large language model auto- regressively generates an output sequence of tokens. More specifically, the auto-regressively generated output is created by generating each particular token in the output sequence conditioned on a current input sequence that includes any tokens that precede the particular text token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token, and a context input that provides context for the output sequence.

[0077] For example, the current input sequence when generating a token at any given position in the output sequence can include the input sequence and the tokens at any preceding positions that precede the given position in the output sequence. As a particular example, the current input sequence can include the input sequence followed by the tokens at any preceding positions that precede the given position in the output sequence. Optionally, the input and the current output sequence can be separated by one or more predetermined tokens within the current input sequence. More specifically, to generate a particular token at a particular position within an output sequence, the large language model can process the current input sequence to generate a score distribution (e.g., a probability distribution) that assigns a respective score, e.g., a respective probability, to each token in the vocabulary of tokens. The large language model can then select, as the particular token, a token from the vocabulary using the score distribution.

[0078] As a particular example, the large language model can be an auto-regressive Transformer-based neural network that includes (i) a plurality of attention blocks that each apply a self- attention operation and (ii) an output subnetwork that processes an output of the last attention block to generate the score distribution.

[0079] The language model neural network 212 can have any of a variety of Transformerbased neural network architectures. The architectures can include OpenAI: ChatGPT, Falcon, Google Bard: LaMDA, Cohere or any of a variety of other large language models.

[0080] The large language model (e.g., when trained on a large dictionary database customized for brain data and / or when prompted appropriately) can generate a treatment plan in response to a request for how to address symptoms a user is experiencing and as verified by a brain data analysis system. A prompt generation system can generate a prompt from a dictionary database that includes, for each possible construct and level of reactivity (e.g., over-reactive, underactive), a list of symptoms and possible treatments for each symptom. A user can select symptom(s) to address and the prompt generation system can select a portion of the dictionary that is relevant to the selected system and parse the portion of the dictionary into a set of prompts. The large language model can process the set of prompts to generate the treatment plan 212.

[0081] FIG. 3 shows another window 300 of a graphical user interface showing a template to submit a natural language input and generate a treatment plan.

[0082] The window 300 includes a text box 302 to receive a natural language input from a user that describes one or more mental states or behaviors of an individual along with three buttons 304, 308 and 310 described below.

[0083] The natural language input can be any description of one or more aspects of an individual’s mental state or behavior in any human language. For example, the natural language input can be a question or a statement. The submission button 304, when pressed, can result in generating a prompt that is based on the natural language input. The prompt can be configured to generate one or more functional networks that can influence the one or more aspects of the mental state or of the behavior.

[0084] When the natural language input is a question, the prompt can be a prompt to receive a section of a dictionary. The prompt can include instructions find one or more constructs that are related to the natural language input from a list of possible constructs e.g., action perception, action planning, acute threat, affiliation attachment, agency ownership, etc. The prompt can include instructions to use the natural language input to receive a section of a customized dictionary relevant to the one or more relevant constructs.

[0085] The dictionary can define, for each construct, symptoms for “overactive” networks and “underactive” networks. For each symptom, the dictionary can define possible treatments. For example, when the construct is Affiliation and Attachment and the natural language prompt may indicate an overactive network, the dictionary can define “Clinginess” as a symptom and “CBT”, “Dialectical behavior therapy”, and “Group therapy” as treatments for clinginess.

[0086] An example dictionary is shown below: rdoc_ constructs_symtoms_treatments = { “Adjusting Expectations” : { “overactive” : {

[0087] “Unrealistic expectations” : [“CBT”, “Mindfulness techniques”, “Psychoeducation”] ,

[0088] “Perfectionism”: [“CBT”, “Mindfulness techniques”, “Acceptance and Commitment Therapy (Act)”]

[0089] } ,

[0090] “underactive”: {

[0091] “Low expectations” : [“CBT”, “Behavioral activation”, “Goal-setting”] , “Difficulty adapting to change”: [“CBT”, “Mindfulness techniques”, “See a healthcare professional”]

[0092] }

[0093] } ,

[0094] “Affiliation and Attachment” : {

[0095] “overactive”: {

[0096] “Clinginess” : [“CBT”, “Dialectical behavior therapy”, “Group therapy”] , “Fear of abandonment” : [“CBT”, “Behavioral activation”, “Group therapy”] } ,

[0097] “underactive” : {

[0098] “Difficulty forming attachments”: [“CBT”, “IPT”, “See a healthcare professional”] }

[0099] } ,

[0100] “Anticipation” : {

[0101] “overactive” : {

[0102] “Excessive worry” : [“CBT”, “Mindfulness techniques”, “Medication”] , “Fear of future events” : [“CBT” “Exposure therapy”, “Medication”]

[0103] } ,

[0104] “underactive” : {

[0105] “Lack of motivation” : [“CBT”, “Behavioral activation”, “See healthcare professional”] ,

[0106] “Difficulty planning for future” : [“CBT”, “Goal-setting”, “Time management”]

[0107] }

[0108] } ,

[0109] “Arousal” : {

[0110] “overactive” : {

[0111] “Hyperarousal”: [“CBT”, “Relaxation techniques”, “Medication”] , “Anxiety” : [“CBT”, “Mindfulness techniques”, “Medication”]

[0112] } ,

[0113] “underactive” : {

[0114] “Lethargy” : [“Behavioral activation”, “Physical activity”, “See a healthcare professional”] ,

[0115] “Difficulty staying awake” : [“Sleep hygiene”, “Medication”, “See a healthcare professional”]

[0116] }

[0117] The prompt can use the natural language input to query the dictionary. The prompt can include instructions to process the natural language input to determine if the symptoms described in the natural language input describe an underactive or overactive network relevant to one or more constructs and then return the section of the dictionary that is relevant to that construct. For example, if the natural language input is “Why is it difficult for me to be away from my friends when Em at work?”, the prompt can query the dictionary to find that clinginess is a symptom of an overreactive Affiliation and Attachment construct. The large language model can return the section of the dictionary that is relevant to Affiliation and Attachment. An example prompt is shown below: question_to_dict(user_input) : return [

[0118] {“role”: “system” , “content”: “You will only provide answers in the shape of a python dictionary. No additional text, just the python dictionary.”} ,

[0119] {“role”: “user”, “content”: “Do you know RDDC framework for mental health?”} , {“role”: “assistant”, “content”: “Yes, I’m familiar with the Research Domain Criteria (RDoC) framework. RDoC is a research framework developed by the National Institute of Mental Health

[0120] {"role": "user" , "content": ""What are all the potential RDOC constructs only related to the question in the next prompt. You will only provide short answers in the form of a python dictionary

[0121] Find in that sentence whether the construct is likely to be overactive or underactive and use this to build in the dictionary mentioned further.

[0122] The dictionary should follow the very strict structure as follows:

[0123] { { 'overactive':

[0124] 'construct- T

[0125] 'construct_2' etc.

[0126] ] ,

[0127] 'underactive': [

[0128] 'construct- 1'

[0129] 'construct-!' etc.

[0130] ]

[0131] } }

[0132] Constructs can only be drawn from that list: action perception action planning acute threat affiliation attachment agency ownership arousal attention circadian rhythm cognitive inhibition declarative memory effort frustrative nonreward goal selection habit PVS head injury headache interoception language loss grief others harm performance monitoring potential threat production facial communication reward anticipation reward delayed reward prediction error reward probability reward receipt reward satiation self agency selft harm self knowledge sleep wake sustained threat vistual perception wm active maintenance wm capacity wm flexible updating wm interference control Limit your answer to only those that are relevant to the questions below

[0133] {"role": "user", "content" : "Question: '{question}"'. format(question=user_input)}].

[0134] The window also includes an exploration button 308 to view more details regarding a functional network (e.g., brain images for the individual that highlight portions of the individual’s brain that show abnormal activity) and a treatment plan button 310 to generate possible treatments in response to the mental state or the behavior of the individual.

[0135] FIG. 4 shows the window of FIG. 3 with a submitted natural language input. The text box 202 receives a natural language input “Why can’t I sleep? Why do I easily get addicted? Why nothing motivates me? Why do I feel socially awkward? Why do I feel threatened by my boss?” that describes multiple aspects of a mental state of a user.

[0136] In some examples, the natural language input can describe one or more aspects of a behavior of an individual e.g., “Others don’t think I’m fun to be around”.

[0137] A prompt generation system can generate a prompt based on the natural language that is configured to generate at least one functional network that influences the at least one aspect of a mental state or a behavior above a threshold.

[0138] A large language model can process the prompt to generate the one or more functional networks that can influence one or more aspects of the mental state or of the behavior. Each functional network can include several smaller networks.

[0139] The window displays a summary 404 that describes symptoms associated with the network anomalies. For example, a summary can read “Arousal: Sleep: Difficulty falling or staying asleep”. In this example, the large language model generates five sub-symptoms that may influence the mental state or behavior.

[0140] For each sub-symptom, an anomaly detection system analyzes MRI data for the individual to determine whether the network(s) correlated with the sub-symptom is anomalous in that particular individual. The window includes an anomaly display 306 that identifies whether each network is anomalous or not for the individual.

[0141] A user can take an action in response to the anomaly display 406. For example, when a user selects the treatment plan button 310, the user can select one or more functional networks and / or sub-symptoms to explore further. A large language model can receive a request for how to address an anomaly in the selected functional network and generate a relevant treatment plan.

[0142] FIG. 5 shows the window of FIG. 3 with a selected anomaly. In this example, the summary 404 of a selected anomaly reads “Arousal: Sleep: Difficulty falling or staying asleep”. When a user selects exploration button 308, the window can display brain images for the individual 502, 504, 506, and 508 that highlight portions of the individual’s brain that show abnormal activity.

[0143] FIG. 6A and FIG. 6B are diagrams of example connectivity data system 600 and 601 , respectively. The connectivity data systems 600 and 601 are examples of systems implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. As depicted in FIG. 6A, the connectivity data system 600 is configured to determine one or more anomalies in correlation data characterizing the correlation between the brain activity of different pairs of parcellations in the brain of the patient. As depicted in FIG. 6B, the connectivity data system 601 is configured to determine one or more anomalies in tractography data characterizing a number of tracts connecting different pairs of parcellations in the brain of the patient.

[0144] Referring to FIG. 6A, the connectivity data system 600 is configured to obtain patient brain data 602 characterizing the brain of a patient and process the patient brain data 602 to generate anomaly correlation data 642, which identifies one or more regions of the brain of the patient for which the patient brain data 602 was anomalous. For example, the patient brain data 602 can include one or more of blood-oxygen-level-dependent imaging data, fMRI data, or EEG data captured from the brain of the patient.

[0145] The connectivity data system 600 includes a pre-processing system 610, a parcellation correlation system 620, a connectivity data set 630, and an anomaly detection system 640.

[0146] The pre-processing system 610 is configured to obtain the patient brain data 602 and process the patient brain data 602 to generate patient parcellation data 612 which organizes the patient brain data 602 according to multiple different parcellations of the brain of the patient. For example, the patient brain data 602 can include multiple different time series characterizing the activity of a respective different region of the brain of the patient over time, e.g., a time series corresponding to each three-dimensional voxel of the brain that can be measured by an MRI machine. The pre-processing system 610 can organize the different time series signals by parcellation according to a brain atlas of the brain. The pre-processing system can then, for each parcellation, combine the different time series signals corresponding to the parcellation, e.g., by determining an average of the different time series signals

[0147] In this specification, a brain atlas is data that defines one or more parcellations of a brain of a patient, e.g., by defining in a common three-dimensional coordinate system the coordinates of the outline of the parcellation or the volume of the parcellation.

[0148] In some implementations the pre-processing system 610 performs one or more additional pre-processing steps to generate the patient parcellation data 612. For example, the pre-processing system 610 can perform smoothing on the patient brain data 602, e.g., to remove components of the patient brain data 602 that are not clinically relevant such as brain activity data related to the heartbeat or breathing of the patient. As another example, the preprocessing system 610 can perform skull stripping on the patient brain data 602. As another example, the pre-processing system 610 can remove one or more slices in brain data 602 in order to allow for signal stabilization. As another example, the pre-processing system 610 can perform slice timing correction on the brain data 602. As another example, the preprocessing system 610 can perform motion correction on the brain data 602. As another example, the pre-processing system 610 can perform gradient distortion correction on the brain data 602. As another example, the pre-processing system 610 can perform global intensity normalization on the brain data 602. As another example, the pre-processing system 610 can calculate one or more confounds using the brain data 602. As another example, the pre-processing system 610 can apply a whitening transform to the brain data 602.

[0149] The parcellation correlation system 620 is configured to obtain the patient parcellation data 612 and to process the patient parcellation data 612 to generate patient correlation data 622 that characterizes, for each pair of parcellation of the multiple parcellations in the patient parcellation data 612, a correlation between brain activity of the first parcellation and brain activity of the second parcellation in the brain of the patient.

[0150] For example, the patient parcellation data 612 can include one or more time series signals corresponding to each parcellation, and the parcellation correlation system 620 can determine, for each pair of parcellations, a correlation between the vales of the time series of the first parcellation and the time series of the second parcellation.

[0151] The anomaly detection system 640 is configured to obtain the patient correlation data 622 and to process the patient correlation data 622 to generate the anomaly correlation data 642. The anomaly correlation data 642 identifies one or more pairs of parcellations whose correlation in the patient correlation data 622 is anomalous.

[0152] In some implementations, the anomaly detection system 640 determines the anomaly correlation data 642 by processing the patient correlation data 622 using a machine learning model that is configured to process correlation data and identity one or more pairs of parcellations that are anomalous. For example, the machine learning model can be trained using correlation data obtained from the respective brain of multiple other patients, e.g., training data stored in the connectivity data set 630.

[0153] In some other implementations, the anomaly detection system 640 can determine the one or more anomalous pairs of parcellations according to normal correlation data 632 obtained from the connectivity data set 630. In some implementations, the normal correlation data 632 identifies, for each pair of parcellations in the patient parcellation data 622, a range of values for the correlation between the pair of parcellations that is considered “normal.” The normal range can be determined by the connectivity data set 630 according to the correlation between the pair of parcellations measured in the respective brain of multiple other patients. For example, the normal correlation data 632 can be determined from brain data captured from hundreds, thousands, or millions of other patients. As a particular example, the normal correlation data 632 might identify, for each pair of parcellations, an average correlation between the pair of parcellations and a standard deviation of correlations between the pair of parcellations, as determined from the correlations measured in the brains of the other patients.

[0154] The anomaly detection system 640 uses the normal correlation data 632 to determine the one or more anomalous pairs of parcellations in the brain of the patient in the anomaly correlation data 642. As a particular example, the anomaly detection system 640 might determine that the correlation between a pair of parcellations is anomalous if the correlation is outside a range of values defined by the average correlation and standard deviation of correlations of the pair of parcellations. For example, the correlation between a pair of parcellations may be anomalous if it is outside one, two, three, or four standard deviations of the average correlation.

[0155] As another particular example, the anomaly detection system 640 might determine that a set of two or more pairs of parcellations are anomalous only if each of the pairs in the set of pairs of parcellations is outside of a normal range of values for the set of pairs of parcellations. That is, the anomaly detection system 640 might maintain data that identifies sets of multiple pairs of parcellations that are related to each other, e.g., that are each related to the same disease, such that the anomaly detection system 640 should only determine that an anomaly has occurred if each pair of parcellations in the set of parcellations is anomalous. In this example, normal correlation data 632 might identify a range of normal correlation values corresponding to each pair in the set of pairs of parcellations, where the range of normal correlation values of a first pair in the set depends on the correlation value of one or more other pairs in the set. For example, for any combination of correlation values for the other pairs in the set, the normal correlation data 632 might define a particular normal range of correlation values for the first pair.

[0156] In some implementations, the anomaly detection system 640 might identify one or more pairs of parcellations whose “normal” correlations, as identified in the normal correlation data 632, are too variable or too unpredictable to draw a conclusion about the corresponding correlation identified in the patient correlation data 622. For example, the anomaly detection system 640 might determine that the correlations between a pair of parcellations is too variable if the standard deviation or variance of the correlations of the pair of parcellations, as identified in the normal correlation data 632, exceeds a predetermined threshold. As another example, the anomaly detection system 640 might rank the standard deviations of correlations of each pair of parcellations identified in the normal correlation data 632, and determine that the pairs of parcellations with the highest standard deviation or variance are too variable, e.g., the pairs of parcellations in the top 10%, 20%, 30%, or 50%. The anomaly detection system 640 would therefore not identify any of the pairs of parcellations identified to be too variable in the anomaly correlation data 642.

[0157] The connectivity data system 600 can provide the anomaly correlation data 642 to one or more downstream systems, e.g., a graphical user interface 650 and / or a machine learning system 660.

[0158] The graphical user interface 650 can display data characterizing the one or more pairs of parcellations whose correlations were determined to be anomalous, as identified in the anomaly correlation data 642, to a user. For example, the graphical user interface 650 can display a list of the one or more pairs of parcellations. As another example, the graphical user interface 650 can display an anomaly correlation matrix characterizing the brain of the user. In this specification, an anomaly correlation matrix is a correlation matrix that visually identifies one or more pairs of parcellations, corresponding to respective elements in the anomaly correlation matrix, whose correlation has been determined to be anomalous. This process is discussed in more detail below with respect to FIG. 7 and FIG. 8. As another example, the graphical user interface 650 can display a text summary of the anomaly correlation data 642 and / or a text summary of the corresponding anomaly correlation matrix. As another example, the graphical user interface 650 can display a score calculated according to the anomaly correlation data 642 that characterizes a degree of anomaly in the brain of the patient, e.g., a value between 0 and 1.

[0159] In some implementations, the graphical user interface 650 can determine to display a subset of the anomaly correlation data 642 that is determined to be clinically relevant to the user. In this specification, a model output is “clinically relevant” if the model output represents an answer to a question that a trained clinician might ask in clinical practice treating patients, e.g., a question asked by a clinician in order to treat a patient with a specific disease. For example, the graphical user interface 650 might determine one or more particular parcellations that are identified multiple times in the anomaly correlation data, i.e., one or more particular parcellations where such parcellation is a member of multiple different pairs of parcellations whose correlation is anomalous. The graphical user interface can therefore display a portion of the anomaly correlation data 642 related to the particular parcellations, e.g., a subset of the anomaly connectivity matrix. The graphical user interface can also display other data related to the particular parcellations, e.g., a three-dimensional model of tractography data corresponding to particular parcellations.

[0160] The machine learning system 660 can include one or more machine learning models that are configured to process the anomaly correlation data 642 and generate a model output that is clinically relevant for a user. For example, a machine learning model can process the anomaly correlation data 642 to generate a prediction for whether the patient has a particular brain disease, e.g., autism, depression, or schizophrenia.

[0161] As a particular example, the machine learning system 660 might determine that the patient is at risk for a particular disease according to one or more pairs of parcellations that are known to often be anomalous in brains of patients who have the particular disease. That is, the machine learning system 660 might have determined, through training using brain data of patients with the particular disease, one or more particular pairs of parcellations whose correlations may be indicators of the particular disease. The machine learning system 660 can then determine whether one or more of the particular pairs of parcellations are identified in the anomaly correlation data 642 of the patient, e.g., whether the number of the particular pairs of parcellations identified in the anomaly correlation data 642 exceeds a predetermined threshold. The machine learning system 660 can then recommend to the user to further analyze one or more regions of the brain or disease indicators of the patient.

[0162] Referring to FIG. 6B, the connectivity data system 601 is configured to obtain patient brain data 604 characterizing the brain of a patient and process the patient brain data 602 to generate anomaly tractography data 692, which identifies one or more regions of the brain of the patient for which the patient brain data 604 was anomalous. For example, the patient brain data 604 can include one or more of blood-oxygen-level-dependent imaging data, fMRI data, or EEG data captured from the brain of the patient.

[0163] The connectivity data system 601 includes a pre-processing system 670, a connectivity data set 680, and an anomaly detection system 690.

[0164] The pre-processing system 670 is configured to obtain the patient brain data 604 and process the patient brain data 604 to generate patient tractography data 672 which characterizes neural tracts connecting pairs of parcellations of the multiple parcellation in the brain of the patient. In some implementations the pre-processing system 670 performs one or more additional pre-processing steps to generate the patient tractography data 672. For example, the pre-processing system 670 can perform one or more pre-processing steps described above with respect to the pre-processing system 610 depicted in FIG. 6A.

[0165] The anomaly detection system 690 is configured to obtain the patient tractography data 672 and to process the patient tractography data 672 to generate the anomaly tractogrpahy data 692. The anomaly tractography data 692 identifies one or more pairs of parcellations for which the number of connections in the patient tractography data 672 is anomalous.

[0166] In some implementations, the anomaly detection system 690 determines the anomaly tractography data 692 by processing the patient tractography data 672 using a machine learning model that is configured to process tractography data and identity one or more pairs of parcellations that are anomalous. For example, the machine learning model can be trained using tractography data obtained from the respective brain of multiple other patients, e.g., training data stored in the connectivity data set 680.

[0167] In some other implementations, the anomaly detection system 690 can determine the one or more anomalous pairs of parcellations according to normal tractography data 682 obtained from the connectivity data set 680. In some implementations, the normal tractography data 682 identifies, for each pair of parcellations in the patient tractography data 672, a range of values for the number of tracts connecting the pair of parcellations that is considered “normal.” The normal range can be determined by the connectivity data set 680 according to the number of tract connecting the pair of parcellations measured in the respective brain of multiple other patients. For example, the normal tractography data 682 can be determined from brain data captured from hundreds, thousands, or millions of other patients. As a particular example, the normal tractography data 682 might identify, for each pair of parcellations, an average number of tracts between the pair of parcellations and a standard deviation of the number of tracts between the pair of parcellations, as determined from the neural tracts measured in the brains of the other patients.

[0168] The anomaly detection system 690 uses the normal tractography data 682 to determine the one or more anomalous pairs of parcellations in the brain of the patient in the anomaly tractography data 692. As a particular example, the anomaly detection system 690 might determine that the number of tracts between a pair of parcellations is anomalous if the number of tracts is outside a range of values defined by the average number of tracts and the standard deviation of the number of tracts connecting the pair of parcellations. For example, the number of tracts connecting a pair of parcellations may be anomalous if it is outside one, two, three, or four standard deviations of the average number of tracts.

[0169] As another particular example, the anomaly detection system 690 might determine that a set of two or more pairs of parcellations are anomalous only if the number of tracts between each pair in the set of pairs of parcellations is outside of a normal range of values defined for the set of pairs of parcellations. That is, the anomaly detection system 690 might maintain data that identifies sets of multiple pairs of parcellations that are related to each other, e.g., that are each related to the same disease, such that the anomaly detection system 690 should only determine that an anomaly has occurred if each pair of parcellations in the set of parcellations is anomalous. In this example, normal tractography data 682 might identify a normal range of numbers of tracts corresponding to each pair in the set of pairs of parcellations, where the range of normal values of a first pair in the set depends on the value of one or more other pairs in the set. For example, for any combination of numbers of tracts connecting each of the other pairs in the set, the normal tractography data 682 might define a particular normal range of the number of tracts between the first pair.

[0170] In some implementations, the anomaly detection system 690 might identify one or more pairs of parcellations whose “normal” number of connecting tracts, as identified in the normal tractography data 682, are too variable or too unpredictable to draw a conclusion about the corresponding number of tracts identified in the patient tractography data 672. For example, the anomaly detection system 690 might determine that the number of tracts between a pair of parcellations is too variable if the standard deviation or variance of the number of tracts between the pair of parcellations, as identified in the normal tractography data 682, exceeds a predetermined threshold. As another example, the anomaly detection system 690 might rank the standard deviations of the numbers of tracts connecting each pair of parcellations identified in the normal correlation data 682, and determine that the pairs of parcellations with the highest standard deviation are too variable, e.g., the pairs of parcellations in the top 10%, 20%, 30%, or 50%. The anomaly detection system 690 would therefore not identify any of the pairs of parcellations identified to be too variable in the anomaly tractography data 692.

[0171] The connectivity data system 601 can provide the anomaly tractography data 692 to one or more downstream systems, e.g., a graphical user interface 650 and / or a machine learning system 660.

[0172] The graphical user interface 650 can display data characterizing the one or more pairs of parcellations whose number of connecting tracts were determined to be anomalous, as identified in the anomaly tractography data 692, to a user. For example, the graphical user interface 650 can display a list of the one or more pairs of parcellations. As another example, the graphical user interface 650 can display a model of the tracts of the brain of the patient. As another example, the graphical user interface 650 can display a text summary of the anomaly tractography data 692. As another example, the graphical user interface 650 can display a score calculated according to the anomaly tractography data 692 that characterizes a degree of anomaly in the brain of the patient, e.g., a value between 0 and 1.

[0173] In some implementations, the graphical user interface 650 can determine to display a subset of the anomaly tractography data 692 that is determined to be clinically relevant to the user. For example, the graphical user interface 650 might determine one or more particular parcellations that are identified multiple times in the anomaly tractography data, i.e., one or more particular parcellations where such parcellation is a member of multiple different pairs of parcellations whose number of tracts is anomalous. The graphical user interface 650 can therefore display a portion of the anomaly tractography data 692 related to the particular parcellations. The graphical user interface can also display other data related to the particular parcellations, e.g., a portion of an anomaly connectivity matrix corresponding to particular parcellations.

[0174] The machine learning system 660 can include one or more machine learning models that are configured to process the anomaly tractography data 692 and generate a model output that is clinically relevant for a user. For example, a machine learning model can process the anomaly tractography data 692 to generate a prediction for whether the patient has a particular brain disease, e.g., autism, depression, or schizophrenia.

[0175] As a particular example, the machine learning system 660 might determine that the patient is at risk for a particular disease according to one or more pairs of parcellations that are known to often be anomalous in brains of patients who have the particular disease. That is, the machine learning system 660 might have determined, through training using brain data of patients with the particular disease, one or more particular pairs of parcellations whose number of connecting tracts may be indicators of the particular disease. The machine learning system 660 can then determine whether one or more of the particular pairs of parcellations are identified in the anomaly tractography data 692 of the patient, e.g., whether the number of the particular pairs of parcellations identified in the anomaly tractography data 692 exceeds a predetermined threshold. The machine learning system 660 can then recommend to the user to further analyze one or more regions of the brain or disease indicators of the patient.

[0176] FIG. 7 is an illustration of an example raw connectivity matrix 710 and an example anomaly connectivity matrix 720.

[0177] The raw correlation identifies, for each pair of parcellations of multiple parcellations in the brain of a patient, the correlation between brain activity of the first parcellation of the pair of parcellations and the brain activity of the second parcellation of the pair of parcellations. That is, each row and column of the raw connectivity matrix 710 corresponds to a parcellation, and each element has a value identifying the correlation between the parcellation corresponding to the row of the element and the parcellation corresponding to the column of the element. In some implementations, the raw connectivity matrix 710 can include ranges of two different colors, where the first color corresponds to negative correlations and the second color corresponds to positive correlations, and the intensity of a color corresponds to a magnitude of the negative or positive correlation.

[0178] It is difficult for a user to inspect a raw connectivity matrix and determine one or more correlations that are anomalous. The user cannot simply identify elements that have a high intensity, because the intensity corresponds to the magnitude of correlation, not the magnitude of anomaly; that is, it may be normal for the correlation between two parcellations to have a large positive or negative correlation.

[0179] As described above, an anomaly connectivity matrix identifies one or more pairs of parcellations whose correlation is anomalous. The anomaly matrix 720 visually identifies elements that correspond to pairs of parcellations whose correlations are determined to be too variable (in black), elements that correspond to pairs of parcellations whose correlations have been determine to be “normal” (in white), and elements that correspond to pairs of parcellations whose correlations have been determined to be anomalous (in grayscale). An example anomaly connectivity matrix is discussed in more detail below in reference to FIG. 4.

[0180] It is much easier for a user to identify anomalous pairs of parcellations using the anomaly connectivity matrix 720 than the raw connectivity matrix 710. In particular, a user can know to ignore the elements that are black or white, and focus on the elements that are grayscale (or color coded) and therefore indicate anomaly.

[0181] FIG. 8 is an illustration of an example anomaly connectivity matrix 800.

[0182] The anomaly matrix 800 includes elements, e.g., element 806, that correspond to pairs of parcellations whose correlation has been determined to be too variable to be clinically useful. That is, the correlations of the pair of parcellations determined from brain data gathered from multiple different patients is noisy, e.g., has a relatively large standard deviation compared to correlations of other pairs of parcellations, and therefore is not useful in predicting a mental health status of a patient. As depicted in FIG. 8, these elements have a dark gray color; however, in general the elements can be visually identified in any way, e.g., any color or pattern.

[0183] The anomaly matrix 800 includes elements, e.g., element 802, that correspond to pairs of parcellations whose correlation is not too variable, and has been determined to be within a normal range. For example, the correlations may be within a threshold defined by the average correlation and standard deviation of correlations in brain data of multiple different patients. As depicted in FIG. 8, these elements have a light gray color; however, in general the elements can be visually identified in any way, e.g., any color or pattern.

[0184] The anomaly matrix 800 includes elements, e.g., element 804, that correspond to pairs of parcellations whose correlation has been determined to be anomalous because it is higher than the normal range. As depicted in FIG. 8, these elements have a checkered pattern; however, in general the elements can be visually identified in any way, e.g., any pattern or range of colors. As a particular example, the elements identifying a higher correlation than normal can be identified by a range of colors whose intensity corresponds to the degree to which the correlation is higher than normal. The anomaly matrix 800 includes elements, e.g., element 808, that correspond to pairs of parcellations whose correlation has been determined to be anomalous because it is lower than the normal range. As depicted in FIG. 8, these elements have a striped pattern; however, in general the elements can be visually identified in any way, e.g., any pattern or range of colors. As a particular example, the elements identifying a lower correlation than normal can be identified by a range of colors whose intensity corresponds to the degree to which the correlation is lower than normal.

[0185] FIG. 9 is a flow diagram of an example method 900 for determining a mental state or a behaviorr of a patient that is of interest based on a natural language input and then determining whether a relevant subset of brain data is anomalous. For convenience, the method 900 will be described as being performed by a system, such as the system illustrated in FIG. 1A.

[0186] The system can receive a natural language input describing at least one aspect of a mental state or of a behavior of an individual (step 902). The natural language input can be a question. For example, the prompt can be “Why am I so sad?” when the prompt describes an aspect of a mental state. As another example, the prompt can be “Why do others think I’m not fun to be around?” when the prompt describes an aspect of a behavior.

[0187] The system can generate a prompt based at least in part the natural language input (step 904). The prompt can be configured to generate at least one functional network that influences the at least one aspect of a mental state or of a behavior above a threshold.

[0188] The system can submit the prompt to a large language model (step 904). The large language model can be any appropriate neural network that receives an input sequence made up of text tokens selected from a vocabulary and auto-regressively generates an output sequence made up of text tokens from the vocabulary. For example, the large language model can be a Transformer-based language model neural network or a recurrent neural networkbased language model neural network.

[0189] The system can receive at least one functional network from the large language model that influences the at least one aspect of a mental state or of a behavior (step 906).

[0190] The system can, for each network of the at least one functional network, analyze MRI data for the individual to determine whether the network is anomalous (step 908). The network is considered to be anomalous if a MRI data regarding the network is outside a normal range , e.g., as defined by the brain data of other patients. The normal range can be, for example, defined by a standard deviation from the median of the distribution, defined by a range of values that cover a percentage (e.g., 95%) of the population, or defined by a distance from the mean of the distribution.

[0191] The system can display to the user each network of the at least one functional network and whether it is anomalous (step 910). Determining if a network is anomalous is described in further detail below with reference to FIG. 10.

[0192] In some implementations, the system can receive a user input in response to the displaying. The response can include the selection of at least one functional network to produce at least one selected functional network.

[0193] The system can take an action in response to the displaying (step 912).

[0194] In some implementations, taking an action includes submitting a request for how to address an anomaly in the at least one selected functional network to a large language model. The large language model can generate one or more treatment plans for how to address the anomaly in the at least one selected functional network. The treatment plan can include surgeries, behavioral therapies, medical treatments (e.g., medications), brain editing techniques (e.g., brain stimulation, pharmacology, etc.), and at-home treatments.

[0195] FIG. 10 is a flowchart of an example process 1000 for determining an anomalous subset of brain data. The process 1000 can be implemented by one or more computer programs installed on one or more computers and programmed in accordance with this specification. For example, the process 1000 can be performed by the computer server module depicted in FIG. 1A. For convenience, the process 1000 will be described as being performed by a system of one or more computers.

[0196] The system obtains brain data captured by one or more sensors characterizing the brain of a patient (step 1001). The brain data can include one or more of blood-oxygen- level-dependent imaging data, fMRI data, EEG data, or tractography data.

[0197] For each of multiple pairs of parcellations formed from a set of parcellations, the system processes the brain data to generate a correlation in the brain of the patient between the brain activity of the first parcellation in the pair of parcellations and the brain activity of the second parcellation in the pair of parcellations (step 1002). In some implementations, the brain data can include, for each of multiple voxels in the brain of the patient, a time series data sequence characterizing the brain activity at the voxel. In these implementations, the system can assign, for each voxel in the brain of the patient, the voxel to a particular parcellation of the plurality of parcellations, and combine, for each parcellation, the time series data sequences corresponding to the voxels assigned to the parcellation to generate a respective parcellation time series data sequence.

[0198] The system obtains second connectivity data that characterizes, for each first parcellation and second parcellation of the set of parcellations, a normal range of correlations between the brain activity of the first parcellation and the second parcellation (step 1004). The second connectivity data can be generated from connectivity data corresponding to multiple other patients.

[0199] The second connectivity data can include, for each pair of parcellations, data characterizing i) a measure of central tendency of the correlation between the brain activity of the pair of parcellations, and ii) a measure of variance of the correlation between the brain activity of the pair of parcellations. As a particular example, the normal range between the brain activity of a pair of parcellations can be defined by i) a first value that specifies a maximum correlation and ii) a second value that specifies a minimum correlation. The first value and the second value can be linear combinations of the corresponding measure of central tendency and the corresponding measure of variance.

[0200] The system identifies one or more of the pairs of parcellations for which the correlation between brain activity of the first parcellation and the second parcellation of the pair is outside of the corresponding normal range of correlations specified in the second connectivity data (step 1006). For example, the system can identify one or more pairs of parcellations for which i) the correlation between brain activity of the pair of parcellations in the brain of the patient specified in the first connectivity data is outside of the corresponding normal range of correlations specified in the second connectivity data, and ii) the measure of variance corresponding to the pair of parcellations in the second connectivity data is below a threshold value.

[0201] The system provides data characterizing the one or more identified pairs of parcellations for display to a user on a graphical interface (step 1008). For example, the system can display an anomaly connectivity matrix generated from the identified pairs of parcellations to the user.

[0202] In some implementations, the system can determine an area of the brain of the patient that includes one or more particular parcellations that are in the one or more identified pairs of parcellations, and display data corresponding to the determined area of the brain of the patient.

[0203] In some implementations, the system can provide data characterizing the one or more identified pairs of parcellations as input to a machine learning model, e.g., a machine learning model that is configured to predict whether the patient has a particular disease.

[0204] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine -readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0205] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0206] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0207] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0208] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0209] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto -optical disks; and CD-ROM and DVD-ROM disks.

[0210] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone, running a messaging application, and receiving responsive messages from the user in return.

[0211] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0212] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0213] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0214] What is claimed is:

Claims

CLAIMS1. A method comprising: receiving a natural language input describing at least one aspect of a mental state or of a behavior of an individual; generating a prompt based on at least in part the natural language input, the prompt configured to generate at least one functional network that influences the at least one aspect of a mental state or of a behavior above a threshold; submitting the prompt to a large language model; receiving at least one functional network that influences the at least one aspect of a mental state or of a behavior; for each network of the at least one functional network, analyzing MRI data for the individual to determine whether the network is anomalous; displaying to a user each network of the at least one functional network and whether it is anomalous; and taking an action in response to the displaying.

2. The method of claim 1 , wherein the method further comprises receiving a user input in response to the displaying, wherein the response includes selection of at least one functional network to produce at least one selected functional network; and taking an action comprises submitting to a large language model a request for how to address an anomaly in the at least one selected functional network.

3. The method of claim 2, wherein taking an action further comprises generating a plurality of treatment plans for how to address the anomaly in the at least one selected functional network.

4. The method of claim 3, wherein a treatment plan comprises (i) surgeries, (ii) behavioral therapies, (iii) brain editing techniques, or a combination thereof.

5. The method of claim 1, wherein the natural language input comprises one or more questions.

6. The method of claim 1 , wherein the method further comprises in response to submitting the prompt to the large language model, receiving a mental state or a behavior.

7. The method of claim 6, wherein the method further comprises displaying the mental state or behavior along with an indication of whether that mental state or behavior is anomalous.

8. One or more computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: receiving a natural language input describing at least one aspect of a mental state or of a behavior of an individual; generating a prompt based on at least in part the natural language input, the prompt configured to generate at least one functional network that influences the at least one aspect of a mental state or of a behavior above a threshold; submitting the prompt to a large language model; receiving at least one functional network that influences the at least one aspect of a mental state or of a behavior; for each network of the at least one functional network, analyzing MRI data for the individual to determine whether the network is anomalous; displaying to a user each network of the at least one functional network and whether it is anomalous; and taking an action in response to the displaying.

9. The computer-readable storage media of claim 8, wherein the operations further comprise receiving a user input in response to the displaying, wherein the response includes selection of at least one functional network to produce at least one selected functional network; and taking an action comprises submitting to a large language model a request for how to address an anomaly in the at least one selected functional network.

10. The computer-readable storage media of claim 9, wherein taking an action further comprises generating a plurality of treatment plans for how to address the anomaly in the at least one selected functional network.

11. The computer-readable storage media of claim 10, wherein a treatment plan comprises (i) surgeries, (ii) behavioral therapies, (iii) brain editing techniques, or a combination thereof.

12. The computer-readable storage media of claim 8, wherein the natural language input comprises one or more questions.

13. The computer-readable storage media of claim 8, wherein the operations further comprise in response to submitting the prompt to the large language model, receiving a mental state or a behavior.

14. A system comprising: a user device; and one or more computers configured to interact with the user device and to perform operations comprising: receiving a natural language input describing at least one aspect of a mental state or of a behavior of an individual; generating a prompt based on at least in part the natural language input, the prompt configured to generate at least one functional network that influences the at least one aspect of a mental state or of a behavior above a threshold; submitting the prompt to a large language model; receiving at least one functional network that influences the at least one aspect of a mental state or of a behavior; for each network of the at least one functional network, analyzing MRI data for the individual to determine whether the network is anomalous; displaying to a user each network of the at least one functional network and whether it is anomalous; and taking an action in response to the displaying.

15. The system of claim 14, wherein the operations further comprise receiving a user input in response to the displaying, wherein the response includes selection of at least one functional network to produce at least one selected functional network; and taking an action comprises submitting to a large language model a request for how to address an anomaly in the at least one selected functional network.

16. The system of claim 15, wherein taking an action further comprises generating a plurality of treatment plans for how to address the anomaly in the at least one selected functional network.

17. The system of claim 16, wherein a treatment plan comprises (i) surgeries, (ii) behavioral therapies, (iii) brain editing techniques, or a combination thereof.

18. The system of claim 14, wherein the natural language input comprises one or more questions.

19. The system of claim 14, wherein the operations further comprise in response to submitting the prompt to the large language model, receiving a mental state or a behavior.

20. The system of claim 14, wherein the user device comprises a personal computer or smart phone running a web browser or a mobile telephone running a WAP browser.

Images