Adapted multi-modal sequence generation neural networks for performing medical tasks

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using an adapted multi-modal sequence generation neural network, e.g., a multi-modal language model neural network, to perform medical tasks. For example, the image encoder neural network of the multi-modal sequence generation neural network can be fine-tuned to encode two-dimensional medical images to allow the neural network to perform medical tasks that require reasoning about two-dimensional medical images. As another example, the video encoder neural network of the multi-modal sequence generation neural network can be fine-tuned to encode volumetric medical images to allow the neural network to perform medical tasks that require reasoning about volumetric medical images. As another example, the image encoder neural network of the multi-modal sequence generation neural network can be fine-tuned to encode non-image medical data, e.g., genomics data, to allow the neural network to perform medical tasks that require reasoning about non-image medical data.

Description

[0001] ADAPTED MULTI-MODAL SEQUENCE GENERATION NEURAL NETWORKS FOR PERFORMING MEDICAL TASKS

[0002] CROSS REFERENCE TO RELATED APPLICATION

[0003] This application claims priority to Greece Application No. 20240100300 filed on April 25, 2024. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.

[0004] BACKGROUND

[0005] This specification relates processing data using machine learning models.

[0006] As one example, neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to another layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of weights.

[0007] SUMMARY

[0008] This specification describes a system implemented as computer programs on one or more computers that adapts a multi-modal sequence generation neural network for performing one or more medical tasks, that uses an adapted multi-modal sequence generation neural network to perform one or more medical tasks, or both.

[0009] A “medical task” is one that requires processing a multi-modal input that includes two or more different types of medical data, e.g., text, two-dimensional medical images, volumetric medical images, non-image medical data, e.g., genomics data, and so on.

[0010] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0011] General -purpose multi-modal sequence generation neural networks, e.g., multimodal large language models (LLMs), have generally been trained on a large variety of multi-modal data and have shown significant progress on tasks that require multi-modal reasoning.

[0012] However, these general-purpose multi-modal sequence generation neural networks cannot be used off-the-shelf for medical tasks that require operating on multi-modal inputs. As one example, many of the inputs that are required to be processed to perform these tasks, e.g., volumetric medical images or genomics data, are not represented in a way that can be encoded by an off-the-shelf multi-modal sequence generation neural network.

[0013] As another example, the training of these multi-modal sequence generation neural networks may not be sufficiently tailored to the medical domain, preventing an off-the-shelf neural network from accurately performing medical tasks.

[0014] This specification describes techniques for addressing these issues and allowing a multi-modal sequence generation neural network to effectively perform tasks that require operating on different modalities of medical data.

[0015] For example, by representing genomics data as an input image, the system can use the image encoder neural network of a general-purpose multi-modal sequence generation neural network to effectively process medical inputs that include genomics data.

[0016] As another example, by representing a volumetric medical image as a video, the system can use the video encoder neural network of a general-purpose multi-modal sequence generation neural network to effectively process medical inputs that include volumetric medical images.

[0017] It will be appreciated that examples can be combined. For example, techniques described in the context of training a multi-modal sequence generation neural network may be combined with techniques for representing genomics data as an input image and / or techniques for representing a volumetric medical image as a video. As a particular example, the techniques described below for fine-tuning (at least) an image encoder can be used to both fine-tune the image encoder on medical images and on images generated from genomics data, e.g., the image encoder can be fine-tuned on a combined data set that includes some training examples that have medical images and some training examples that have images representing genomics data.

[0018] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below.

[0019] Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0020] BRIEF DESCRIPTION OF DRAWINGS

[0021] FIG. 1 shows an example multi-modal processing system.

[0022] FIG. 2 shows an example of adapting the multi-modal sequence generation neural network. FIG. 3 is a flow diagram of an example process for adapting a multi-modal sequence generation neural network by fine-tuning an image encoder of the multi-modal sequence generation neural network.

[0023] FIG. 4 is a flow diagram of an example process for processing genomics inputs using a multi-modal sequence generation neural network that has an image encoder.

[0024] FIG. 5 is a flow diagram of an example process for adapting a multi-modal sequence generation neural network to process genomics inputs.

[0025] FIG. 6 is a flow diagram of an example process for processing volumetric medical images using a multi-modal sequence generation neural network.

[0026] FIG. 7 is a flow diagram of an example process for adapting a multi-modal sequence generation neural network to process volumetric images.

[0027] FIG. 8 shows examples of data sets that can be used to adapt the multi-modal sequence generation neural network.

[0028] FIGS. 9-13 show various examples of the performance of the described techniques.

[0029] Like reference numbers and designations in the various drawings indicate like elements.

[0030] DETAILED DESCRIPTION

[0031] FIG. 1 shows an example multi-modal processing system 100. The multi-modal processing system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0032] The system 100 adapts a multi-modal sequence generation neural network 110 for performing one or more medical tasks using training data 120. As a result, the system 100 generates one or more adapted (or ‘‘medically tuned'’) multi-modal sequence generation neural networks that are each adapted to perform one or more medical tasks that require processing multi-modal data, e.g., that can process a medical input 130 that includes multimodal data 140 to generate a medical output 150 that carries out a medical task on the medical input 130 using the multi-modal data 140. Examples of medical tasks will be described below.

[0033] Generally, the multi-modal sequence generation neural network 110 includes a token generation neural network 150 having a set of token generation parameters that is configured to process a sequence of input tokens to generate a sequence of output tokens. In this specification, a "token" is a vector of numeric values that has a fixed dimensionality.

[0034] A token can be a discrete token, e.g., a one-hot representation of an input or output from a corresponding vocabulary or a continuous token, e.g., a continuous representation of at least a portion of a corresponding input or output.

[0035] For example, the token generation neural network 150 can be an auto-regressive generative neural network that generates each token in the output sequence conditioned on the preceding tokens in the output sequence and at least some of the tokens in the input sequence.

[0036] For example, the token generation neural network 150 can be a language model neural network, e.g., a large language model ("LLM") that is configured to process an input sequence of tokens from a vocabulary of tokens to generate an output sequence of tokens from the vocabulary.

[0037] More generally, the token generation neural network 150 can be any appropriate neural network that receives an input sequence made up of tokens selected from a vocabulary and auto-regressively generates an output sequence made up of tokens from the vocabulary. For example, the token generation neural network 150 can be a Transformerbased language model neural network or a recurrent neural network-based language model neural network.

[0038] In some situations, the token generation neural network 150 can be referred to as an auto-regressive neural network when the neural network used to implement the language model auto-regressively generates an output sequence of tokens. More specifically, the auto-regressively generated output is created by generating each particular token in the output sequence conditioned on a current input sequence that includes any tokens that precede the particular text token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token, and a context input that provides context for the output sequence.

[0039] For example, the current input sequence when generating a token at any given position in the output sequence can include the input sequence and the tokens at any preceding positions that precede the given position in the output sequence. As a particular example, the current input sequence can include the input sequence followed by the tokens at any preceding positions that precede the given position in the output sequence. Optionally, the input and the current output sequence can be separated by one or more predetermined tokens within the current input sequence.

[0040] More specifically, to generate a particular token at a particular position within an output sequence, the token generation neural network 150 can process the current input sequence to generate a score distribution (e.g., a probability distribution) that assigns a respective score, e.g.. a respective probability, to each token in the vocabulary of tokens. The token generation neural network 150 can then select, as the particular token, a token from the vocabulary using the score distribution. For example, the token generation neural network 150 can greedily select the highest-scoring token or can sample, e.g., using nucleus sampling or another sampling technique, a token from the distribution.

[0041] As a particular example, the token generation neural network 150 can be an autoregressive Transformer-based neural network that includes (i) a plurality of attention blocks, at least some of which apply a self-attention operation and (ii) an output subnetwork that processes an output of the last attention block to generate the score distribution.

[0042] The token generation neural network 150 can have any of a variety of Transformerbased neural network architectures. Examples of such architectures include those described in J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203. 15556, 2022; J.W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song. J. Aslanides. S. Henderson, R. Ring. S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Eisen, S. M. Jayakumar, E. Buchatskaya, D. Budden. E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens. X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toy ama, C. de Masson d’Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas. A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac. E. Lockhart, S. Osindero, L. RimelL C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112. 11446, 2021 ; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou. Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910. 10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha. Gaurav Nemade. Yifeng Lu. and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; Gemini (described in arXiv:2403.05530), Gemma (described in arXiv: 2403.08295), and PaliGemma (described in arXiv:2412.03555).

[0043] Generally, the output sequence of output tokens is used to generate the output for the task that the neural network 110 is performing, e.g., for a given medical task after adaptation by the system 100.

[0044] For example, the output sequence can be mapped to a text sequence and the text sequence can be provided as the output.

[0045] When the output sequence includes tokens representing other modalities other than text, those tokens can be mapped to outputs of the corresponding modality by a corresponding decoder and then provided, e.g., along with a text sequence also specified by the output sequence, as the output.

[0046] The multi-modal sequence generation neural network 110 also includes a respective encoder for each of the multiple modalities that the neural network 110 is configured to process. The encoder for a given modality is configured to receive an input of the modality and convert the input to one or more tokens representing the input. These tokens can then be included in the input sequence that is processed by the token generation neural network 150.

[0047] For example, for the text modality, the respective encoder can be a tokenizer 160 that tokenizes the text to generate one or more sparse tokens representing the text and, optionally, an embedding layer that maps each sparse token to a continuous token. The tokenizer 160 can tokenize the tokens using any appropriation tokenization technique, e.g., byte pair encoding (BPE), e.g., by applying a SentencePiece or other appropriate tokenization scheme to the input text.

[0048] For the image modality, the respective encoder can be an image encoder neural network 170 having a set of image encoder parameters and configured to process an input image to generate one or more tokens representing the input image.

[0049] For the video modality, the respective encoder can be a video encoder neural network 180 having a set of image encoder parameters and configured to process an input video to generate one or more tokens representing the input video.

[0050] In some cases, prior to adaptation by the system 100, the image and video encoder neural networks are the same neural network, i.e., the neural network 110 uses the same encoder to encode both images and videos, with each video frame in the video being represented by a respective set of one or more tokens. In other cases, prior to adaptation by the system 100, the image and video encoder neural networks are different neural networks, i.e., the neural network 110 uses one encoder to encode images and another, different encoder to encode videos.

[0051] Examples of image and video encoders include Flamingo encoders, CoCa encoders, MaMMUT encoders. SigLiP encoders, and PaLI encoders.

[0052] Other encoders for other modalities, e.g., audio, are possible.

[0053] The token generation neural network 150 can process the token(s) generated by the encoders for the modalities as input in any of a variety of ways.

[0054] As one example, the tokens can be included as part of the input sequence that is processed by the token generation neural network 150.

[0055] As another example, the token generation neural network 150 can include one or more cross-attention layers that each perform a cross-attention mechanism that uses, as queries, representations of the already-generated output tokens and, as keys and values, representations of tokens generated by one or more of the encoders. In some examples, the tokens for one or more of the encoders are both included as part of the input sequence and used as input to one or more cross-attention layers. Further, in some example, the tokens for one or more of the encoders are included as part of the input sequence while the tokens for others of the encoders are used as input to one or more cross-attention layers.

[0056] Examples of incorporating tokens of multiple modalities into the processing of an auto-regressive neural network are described in more detail in, e.g., Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022, and Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Saiz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. PaLi: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022.

[0057] Generally, prior to the training of the neural network 1 10 by the system 100, the neural network 110 can have already been trained across one or more training stages.

[0058] For example, the one or more training stages can include a pre-training stage. During the pre-training stage, the neural network 110 can have been trained by the system or a separate system on a next token prediction task, e.g., a task that requires predicting. given a current sequence of tokens, the next token that follows the current sequence in the training data.

[0059] As a particular example, the neural network 110 can have been trained on a maximum-likelihood objective on a large dataset of text in one or more natural languages, e.g., text that is publicly available from the Internet or another text corpus, a large dataset of computer code in one or more programming languages, e.g., Python, C++, C#, Java, Ruby, PHP, and so on, e.g., computer code that is publicly available from the Internet or another code repository, a large dataset of audio samples, e.g., audio recordings or waveforms that represent the audio recordings, a large dataset of images where each image includes an array of pixels, a large dataset of videos where each video includes a temporal sequence of frames, or a large multi-modal dataset that includes a combination of two or more of these datasets. In some cases, the system 100 modifies this pre-training stage to include some or all of the medical training data described below in order to improve the effectiveness of the pre-trained model on medical tasks.

[0060] As another example, the one or more training stages can include one or more additional training stages, e.g.. that occur after the pre-training stage. For example, the one or more additional training stages can include any one or more of: a supervised fine-tuning stage, a reinforcement learning stage, e.g., reinforcement learning from human or other feedback, a preference learning stage, an instruction tuning stage, and so on.

[0061] As described above, the system 100 adapts the multi-modal sequence generation neural network 1 10 for performing one or more medical tasks.

[0062] For example, the system 100 can fine-tune the image encoder neural network 170 of the multi-modal sequence generation neural network 110 to encode two-dimensional medical images to allow the neural network 110 to perform medical tasks that require reasoning about two-dimensional medical images.

[0063] Examples of such medical tasks include medical image classification, question answering, report generation, and risk prediction.

[0064] When the task is medical image classification, the input includes a medical image and the output classifies the medical image as depicting a particular object class. For example, the output can be a label for the particular object class or a respective score for each of multiple object classes that represents the likelihood that the medical image depicts the particular object class. Examples of classification tasks are depicted below in FIGS. 8- 13. When the task is question answering, the input includes a medical image and a question about the medical image and the output is a response, e.g., a natural language response, to the question.

[0065] When the task is report generation, the input includes a medical image and the output is a prediction of a report that would be generated by a clinician based on the medical image. For example, the input can be a radiology image, e.g., an X-ray. and the output can be a radiology report. A radiology report is a written document that summarizes the findings of an imaging study, such as an X-ray, MRI, or CT scan (described below with reference to volumetric images). The report is a communication tool between radiologists and other medical professionals, outlining a radiologist's interpretation of the images. The report includes details, e.g., the patient's information, the type of exam, the reason for the scan, the findings, and any impressions or conclusions the radiologist has about the images. The report can be a free-form document or can be a structured document arranged according to a specific structure or schema.

[0066] When the task is risk prediction, the input includes a medical image and the output is a prediction of a respective risk level for each of one more medical conditions for the patient corresponding to the medical image. For example, the prediction can be represented as a risk score, a respective score for each of multiple predetermined risk levels, or as a text description of the risk level. Examples of medical conditions are shown below in FIGS. 8- 13.

[0067] Depending on the task, the tw^o-dimensional medical images can include any of a variety7of different ty pes of medical images, e.g., CXR images, mammography images, pathology images, dermatology' images, retinal images, and so on. Moreover, two- dimensional medical images can include images captured by any of a variety of imaging devices, e.g., X-ray devices, fundus cameras, mobile phone or other user device cameras, and so on.

[0068] As another example, the system 100 can fine-tune the video encoder neural network 180 of the multi-modal sequence generation neural network 110 to encode volumetric medical images to allow the neural network 110 to perform medical tasks that require reasoning about volumetric medical images.

[0069] Examples of such medical tasks include medical image classification, question answering, report generation, and risk prediction. Inputs and outputs for these tasks are described above. Generally, a volumetric image is an image that contains information for each point in a three-dimensional coordinate space that includes a depth dimension in addition to height and width dimensions.

[0070] For example, a volumetric image can be a three-dimensional image, where each coordinate corresponds to a respective voxel of the image in a three-dimensional coordinate space, e.g., a three-dimensional grid, where the one or more respective values are one or more intensity values of the voxel.

[0071] Examples of such images include computer tomography (CT) image, a magnetic resonance imaging (MRI) image, an ultrasound image, an X-ray image, a mammogram image, a fluoroscopy image, a SPECT (Single Photon Emission Computed Tomography) image, an Optical Coherence Tomography (OCT) image, a Cone Beam CT (CBCT) image, or a positron-emission tomography (PET) image.

[0072] As another example, the system 100 can fine-tune the image encoder neural network 170 of the multi-modal sequence generation neural network 110 to encode non-image medical data, e.g., genomics data, to allow the neural network 110 to perform medical tasks that require reasoning about non-image medical data. For example, the genomics data can be represented as a set of genomics data risk scores, also known as polygenic risk scores (PRSs) or genetic risk scores, that are calculated by analyzing an individual's genetic information to estimate their predisposition to a specific disease or trait. These scores aggregate the effects of many genetic variants into a single number, representing the overall genetic risk.

[0073] Examples of such medical tasks include question answering, report generation, and risk prediction. Inputs and outputs for these tasks are described above.

[0074] In some implementations, after being adapted, the same multi-modal sequence generation neural network 110 can include multiple ones of the above encoders, allowing the same multi-modal sequence generation neural network 110 to operate on multi-modal data of different modalities, e.g., the same multi-modal sequence generation neural network 110 can process any of genomics data, two-dimensional medical images, or volumetric medical images by encoding each type of data with the appropriate encoder.

[0075] In some other implementations, the same multi-modal sequence generation neural network 110 can be separately fine-tuned to process each of the above data ty pes, yielding multiple different adapted multi-modal sequence generation neural networks that are each adapted for processing a different type of medical data. FIG. 2 shows an example 200 of the adaptation of the multi-modal sequence generation neural network 110.

[0076] In the example 200, a multi-modal sequence generation neural network 210 is separately fine-tuned to process three separate types of medical data 260 to yield three “medically tuned'’ or “adapted"’ neural networks 220.

[0077] While three separate types of medical training data 260 are described in FIG. 2, in general, the medical training data can include any appropriate types of medical training data. For example, the medical training data can include text-only medical training data, e.g., sequences of text relating to medical conditions. These text-only medical training examples can be incorporated as part of the training data that is used for any of the finetuning or pre-training described in this specification, e.g., to ensure that the fine-tuned medical neural network(s) are able to effectively respond to text-only inputs.

[0078] In the example 200, the multi-modal sequence generation neural network 210 is fine-tuned on medical training data that includes two-dimensional (2D) medical images to yield a 2D neural network 230 that performs tasks that require operating on 2D medical images.

[0079] Further, the multi-modal sequence generation neural network 210 is fine-tuned on medical training data that includes 3D medical images to yield a 3D neural network 240 that performs tasks that require operating on 3D medical images.

[0080] Additionally, the multi-modal sequence generation neural network 210 is fine-tuned on genomics data to yield a genomics neural network 250 that performs task that require operating on genomics data.

[0081] After fine-tuning, the system 100 or another system can use the medically -tuned neural networks 220 to perform a variety of tasks. Examples of such tasks that are shown in FIG. 2 include classification tasks, visual question answering tasks, 2D report generation tasks, i.e., a report generation task that operates on 2D medical images, 3D report generation tasks, i.e., a report generation task that operates on 3D medical images, and polygenic risk prediction tasks.

[0082] That is, after fine-tuning, the system can receive an input, and determine what modality of medical data is included in the input or required to be generated by processing the input.

[0083] The system can then select an appropriate one of the medically-tuned neural networks 220 that has been adapted to process the determined modality type, and then use the selected medically -tuned neural network 220 to generate the response to the input. Thus, the system can flexibly and accurately perform any of a variety7of tasks using the medically- tuned neural networks 220.

[0084] FIG. 3 is a flow diagram of an example process 300 for adapting a multi-modal sequence generation neural network by fine-tuning an image encoder of the multi-modal sequence generation neural network. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a multi-modal processing system, e.g.. the multi-modal processing system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

[0085] The system obtains data specifying a pre-trained multi-modal sequence generation neural network (step 302).

[0086] As described above, the multi-modal sequence generation neural network generally includes an image encoder neural network having a set of image encoder parameters and configured to process an input image to generate one or more tokens representing the input image and a token generation neural network having a set of token generation parameters configured to process a sequence of input tokens to generate a sequence of output tokens. Thus, the data specifying the pre-trained multi-modal sequence generation neural network includes data specifying the pre-trained values of the image encoder parameters and pretrained values of the token generation parameters.

[0087] The multi-modal sequence generation neural network can optionally include other components in addition to the image encoder neural network. Examples of such components include other encoders to encode other types of input data and decoders to decode the tokens generated by the token generation neural network.

[0088] The system obtains medical training data that includes a plurality of medical training examples (step 304).

[0089] Each of the medical training examples includes (i) a respective medical input that includes one or more medical images and (ii) a respective target medical output for a respective medical task. Generally, the medical training examples can include examples corresponding to multiple different medical tasks, e.g., multiple ones of the tasks described above. Depending on the task, the medical images can include any of a variety of different ty pes of medical images, e.g., CXR images, mammography images, pathology7images, dermatology7images, retinal images, and so on.

[0090] As described above and below, the medical training data can also optionally include other types of training examples. The system trains the pre-trained multi-modal sequence generation neural network on the medical training data to update at least the pre-trained values of the image encoder parameters (step 306).

[0091] In some cases, the system only trains the image encoder neural network, i.e., only updates the pre-trained values of the image encoder parameters, and holds the token generation neural network fixed.

[0092] In some other cases, the system also trains the token generation neural network. In these other cases, the system trains the pre-trained multi-modal sequence generation neural network to update at least the pre-trained values of the image encoder parameters and the pre-trained values of the token generation parameters.

[0093] The system can also optionally update other parameters of the multi-modal sequence generation neural network, e.g., the parameters of the text encoder, the parameters of one or more decoders required to generate the outputs for the medical task(s), or both.

[0094] Optionally, after performing step 306, the system can receive a new medical input that includes one or more new medical images (step 308) and process the new medical input using the (adapted) multi-modal sequence generation neural network to generate a new medical output (step 310).

[0095] As part of this processing, the system processes each of the one or more new medical images using the image encoder neural network to generate one or more tokens representing each of the one or more new medical images and then processes a new input that includes the one or more tokens representing each of the one or more new medical images using the token generation neural netw ork to generate a new7output sequence of tokens.

[0096] FIG. 4 is a flow diagram of an example process 400 for processing genomics inputs using a multi-modal sequence generation neural network that has an image encoder. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a multi-modal processing system, e.g., the multi-modal processing system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 400.

[0097] The system receives a medical input (step 402). The medical input includes a genomics input that includes genomics data or data derived from the genomics data. The medical input can also include data of one or more other modalities, e.g., text that specifies a query that relates to the genomics input. The system processes the medical input using a multi-modal sequence generation neural network to generate a medical output for a medical task (step 404).

[0098] As part of the processing, the system generates an image that represents the genomics input (step 406).

[0099] Generally, the image represents the genomics data as an image that has respective values for each of a set of pixels. That is, the image represents the genomics data by, for each of the pixels of the image, having a respective value for each of a plurality of channels, with the values for the channels being derived from the genomics data.

[0100] For example, the system can determine, from the genomics data, one or more risk scores for each of a plurality of traits and then project the risk scores for the plurality of traits to generate a plurality of patches of the image. Here, ‘"projecting'’ a risk score refers to projecting the risk score using a predetermined projection that projects scores to fall in a specified range of color values, e.g., [0,255], As a particular example, the respective value of each channel for each pixel can correspond to a projection of a corresponding risk score of a respective p-value threshold for the corresponding trait.

[0101] As a specific example of this and as descnbed above, the genomic data can include polygenic risk scores (PRSs) for multiple traits (and at multiple p-value thresholds). The system can generate an image from these PRSs by projecting into [0, 255] 8x8 pixel patch- aligned squares, and then stacking the projection of PRSs of 3 p-value thresholds as RGB channels of the image. For example, each patch of the image can correspond to a respective trait and the pixels of the patch can each have respective channels that include projected risk scores for the trait.

[0102] The system then processes the image that represents the genomics data using an image encoder neural network of the multi-modal sequence generation neural network to generate one or more tokens representing the genomics data (step 408). That is, by generating a representation of the genomics data as an image, the system is able to process the genomics data using the image encoder neural network to generate one or more tokens representing the genomics data.

[0103] When the medical input includes data of one or more other modalities, the system also processes the other data of the one or more other modalities to generate tokens representing the other data.

[0104] The system processes the input that includes the tokens representing the genomics data using a token generation neural network of the multi-modal sequence generation neural network to generate a sequence of output tokens (step 410). When the medical input includes data of one or more other modalities, the input to the token generation neural network also includes the tokens representing the other data.

[0105] As described above, in some cases, prior to performing the process 400, the system or another training system has adapted the multi-modal sequence generation neural network to effectively process the genomics inputs.

[0106] FIG. 5 is a flow diagram of an example process 500 for adapting a multi-modal sequence generation neural network to process genomics inputs. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a multi-modal processing system, e.g., the multi-modal processing system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 500.

[0107] The system obtains data specifying a pre-trained multi-modal sequence generation neural network (step 502).

[0108] As described above, the multi-modal sequence generation neural network generally includes an image encoder neural network having a set of image encoder parameters and configured to process an input image to generate one or more tokens representing the input image and a token generation neural network having a set of token generation parameters configured to process a sequence of input tokens to generate a sequence of output tokens. Thus, the data specifying the pre-trained multi-modal sequence generation neural network includes data specifying the pre-trained values of the image encoder parameters and pretrained values of the token generation parameters.

[0109] The multi-modal sequence generation neural network can optionally include other components in addition to the image encoder neural network. Examples of such components include other encoders to encode other types of input data and decoders to decode the tokens generated by the token generation neural network.

[0110] The system obtains medical training data that includes a plurality of medical training examples (step 504).

[0111] Each of the plurality’ of medical training examples includes (i) a respective medical input that includes a training image representing training genomics data and (ii) a respective target medical output for a respective medical task. Generally, the medical training examples can include examples corresponding to multiple different medical tasks, e.g., multiple ones of the tasks described above. For example, the system can have obtained, for each training example, the corresponding genomics data and can have generated the respective training image in the training example from the genomics data as described above with reference to FIG. 4.

[0112] As described above and below, the medical training data can also include other types of medical training examples.

[0113] The system trains the pre-trained multi-modal sequence generation neural network on the medical training data to update at least the pre-trained values of the image encoder parameters (step 506). In some cases, the system only trains the image encoder neural network, i.e., only updates the pre-trained values of the image encoder parameters, and holds the token generation neural network fixed. In some other cases, the system also trains the token generation neural network. In these other cases, the system trains the pre-trained multi-modal sequence generation neural network to update at least the pre-trained values of the image encoder parameters and the pre-trained values of the token generation parameters. The system can also optionally update other parameters of the multi-modal sequence generation neural network, e.g., the parameters of the text encoder, the parameters of one or more decoders required to generate the outputs for the medical task(s), or both.

[0114] FIG. 6 is a flow diagram of an example process 600 for processing volumetric medical images using a multi-modal sequence generation neural network. For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a multi-modal processing system, e.g., the multi-modal processing system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 600.

[0115] The system receives a medical input (step 602). The medical input includes a volumetric image, e.g., one of the types of volumetric image described above with reference to FIG. 1. As a particular example, the volumetric image can be a computed tomography (CT) volume. The medical input can also include data of one or more other modalities, e.g., text that specifies a query that relates to the volumetric image.

[0116] The system processes the medical input using a multi-modal sequence generation neural network to generate a medical output for a medical task (step 604). For example, the output for the medical task can be a prediction of at least a portion of a radiology report that would be generated by a radiologist from the volumetric image.

[0117] As part of the processing, the system processes the volumetric image using a video encoder neural network of the multi-modal sequence generation neural network to generate one or more tokens representing the volumetric image (step 606). Generally, the system can generate a plurality of video patches from the volumetric image by representing the depth dimension of the volumetric image as the time dimension of a video and then process the plurality of video patches using the video encoder neural network to generate the one or more tokens. In other words, the video encoder neural network is generally configured to process videos of the form h x w x / , where h x w are the spatial dimensions of the video frames and t is a time dimension that identifies the time stamp of each of the video frames of the video. By representing the depth dimension d of the volumetric image as the time dimension t of the video, the system is able to re-purpose the video encoder for accurately encoding volumetric images. In particular, the system represents the volumetric image as a sequence of 2D slices from the volumetric image, with each slice having a corresponding depth index. The system then provides these 2D slices as input to the video encoder, with each 2D slice corresponding to a video frame and the depth index of the slice corresponding to the time index of the corresponding video frame. This representation leverages the video understanding capabilities of the multi-modal sequence generation neural network for effectively processing volumetric images.

[0118] When the medical input includes data of one or more other modalities, the system also processes the other data of the one or more other modalities to generate tokens representing the other data.

[0119] The system processes the input that includes the tokens representing the volumetric image using a token generation neural network of the multi-modal sequence generation neural network to generate a sequence of output tokens (step 608). When the medical input includes data of one or more other modalities, the input to the token generation neural network also includes the tokens representing the other data.

[0120] As described above, in some cases, prior to performing the process 600. the system or another training system has adapted the multi-modal sequence generation neural network to effectively process the volumetric images.

[0121] FIG. 7 is a flow diagram of an example process 700 for adapting a multi-modal sequence generation neural network to process volumetric images. For convenience, the process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, a multi-modal processing system, e.g., the multi-modal processing system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 700.

[0122] The system obtains data specifying a pre-trained multi-modal sequence generation neural network (step 702). As described above, the multi-modal sequence generation neural network generally includes a video encoder neural network having a set of video encoder parameters and configured to process an input video to generate one or more tokens representing the input video and a token generation neural network having a set of token generation parameters configured to process a sequence of input tokens to generate a sequence of output tokens. Thus, the data specifying the pre-trained multi-modal sequence generation neural network includes data specifying the pre-trained values of the video encoder parameters and pretrained values of the token generation parameters.

[0123] The multi-modal sequence generation neural network can optionally include other components in addition to the video encoder neural network. Examples of such components include other encoders to encode other types of input data and decoders to decode the tokens generated by the token generation neural network.

[0124] The system obtains medical training data that includes a plurality of medical training examples (step 704).

[0125] Each of the plurality of medical training examples includes (i) a respective medical input that includes a volumetric image and (ii) a respective target medical output for a respective medical task. Generally, the medical training examples can include examples corresponding to multiple different medical tasks, e.g., multiple ones of the tasks described above.

[0126] The volumetric images are each represented by video patches, i.e.. sequences of 2D slices that each have a respective depth value, as described above, so that the volumetric images can be effectively processed by the video encoder. Thus, although the video encoder was pre-trained on video data that included videos having a time dimension, the video encoder is fine-tuned on volumetric image data where each 2D slice has a corresponding depth, rather than time, index.

[0127] The system trains the pre-trained multi-modal sequence generation neural network on the medical training data to update at least the pre-trained values of the video encoder parameters (step 706). In some cases, the system only trains the video encoder neural network, i.e., only updates the pre-trained values of the video encoder parameters, and holds the token generation neural network fixed. In some other cases, the system also trains the token generation neural network. In these other cases, the system trains the pre-trained multi-modal sequence generation neural network to update at least the pre-trained values of the video encoder parameters and the pre-trained values of the token generation parameters. The system can also optionally update other parameters of the multi-modal sequence generation neural network, e.g., the parameters of the text encoder, the parameters of one or more decoders required to generate the outputs for the medical task(s), or both.

[0128] While the above description describes adapting the video encoder neural network of a multi-modal sequence generation neural network for medical tasks, more generally, the described techniques can be used to adapt the multi-modal sequence generation neural network to process any type of volumetric image for any type of task, e.g., tasks outside of the medical domain.

[0129] Generally, the system can perform the training described above by optimizing an appropriate objective function, e.g., by minimizing an appropriate loss function. Examples of appropriate objectives include negative log likelihood and cross-entropy objectives. The system can optimize the objective by performing an iterative training process. At each iteration, the system can sample a set of one or more training examples from the set of training examples being used for the training and determine, using the sampled training example(s) gradients of the objective function with respect to the parameters that are being updated by the training. The system can then apply an optimizer to the gradients to update the corresponding parameters. Examples of appropriate optimizers include stochastic gradient descent (SGD), Adam, AdamW, Adafactor, and so on.

[0130] In some cases, the system performs any of the training techniques described above across multiple training phases. For example, the system can include any of the training examples described above in a pre-training phase, so that the pre-training phase includes training on both general-purpose and medical training data. As another example, after the pre-training, the system first performs the fine-tuning described above and then perform additional training phases. For example, the system can include any of the medical training data described above in a reinforcement learning phase, e.g., a reinforcement learning from human feedback (RLHF) phase or a reinforcement learning from Al feedback (RLAI) phase. As another example, the system can include any of the medical training data described above in training data for a distillation phase, where the system distills the token generation neural network, one or more of the encoder neural networks, or both, into a smaller neural network.

[0131] FIG. 8 shows an example 800 of various data sets that can be used to fine-tune the multi-modal sequence generation neural network 110. In particular, the example 800 shows examples of data sets that can be used to adapt the multi-modal sequence generation neural network 110 to process 2D medical images, e.g.. 2D radiology images, 2D pathology images, 2D dermatology images, 2D ophthalmology images, and a variety of images for performing medical visual question answering (VQA) tasks. The example 800 also shows an example of a data set that can be used to adapt the multi-modal sequence generation neural network 110 on 3D radiology images, e.g., 3D CT images. The example 800 also shows an example of a data set that can be used to adapt the multi-modal sequence generation neural network 110 on genomics data.

[0132] FIGS. 9-13 shows various examples 900-1300 of the performance of the described techniques (referred to as "‘Mosaic’7), e.g., relative to a multi-modal generative neural network that has not been adapted (the “Gemini Ultra” model), relative to human clinicians, and relative to other baseline neural network-based approaches.

[0133] For example, FIG. 9 shows an example 900 of the performance of the described techniques relative to Gemini Ultra on chest X-ray classification. As can be seen from the example 900, the described techniques matches or outperforms Gemini Ultra on a variety of classification tasks on the MIMIC CXR datataset.

[0134] As another example, FIG. 10 shows an example 1000 of the performance of the described techniques relative to Gemini Ultra on ophthalmology classification from images of the fundus of the eye of the patient.

[0135] As can be seen from the example 1000, the performance of described techniques outperforms Gemini Ultra on a variety of classification tasks on the EyePACS datataset.

[0136] As another example, FIG. 11 shows an example 1100 of the performance of the described techniques relative to Gemini Ultra on visual question answering tasks across diverse medical specialties, including radiology and pathology. The example 1 100 also shows the performance of state-of-the-art approaches on each task.

[0137] As can be seen from the example 1100, the described techniques outperform Gemini Ultra across the board on the visual question answering tasks. Moreover, the described techniques exceed or are comparable to state-of-the art performance on these tasks, despite using a single model to perform all of the tasks instead of a specialized model for the task.

[0138] As another example, FIG. 12 shows an example 1200 of the performance of the described techniques relative to the performance of state-of-the-art approaches on generating reports from chest X-rays. As can be seen from the example 1200, the described techniques outperform the existing state-of-the-art approach on a variety of evaluation metrics.

[0139] As another example, FIG. 13 shows an example 1300 of the performance on health outcome prediction using the described techniques compared to two baselines for both indistribution and out-of-distribution outcomes. “Demographics only” used a linear probe of age, sex, and BMI to predict each health outcome, and “Ensemble of PRSs and demographics" combined demographics with all 7,145 PRSs in a linear probe. In this example, the sequence generation neural network was prompted with both an individual’s PRS image and demographics. For out-of-distribution health outcomes, the linear probes ("Ensemble of PRSs and demographics" and "Demographics only") were trained to predict the most-correlated in-distribution outcome (Table A.9). and those predictions were then evaluated on the out-of-distribution outcome. As can be seen from the example 1300, the described techniques outperform the baselines across the variety of prediction tasks.

[0140] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly- embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0141] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0142] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0143] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

[0144] Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0145] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by. or incorporated in. special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0146] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory' devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0147] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory' feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return. Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0148] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a Jax framework.

[0149] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0150] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0151] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0152] Similarly, while operations are correspond toed in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0153] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes correspond toed in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0154] What is claimed is:

Claims

CLAIMS1. A method performed by one or more computers, the method comprising: obtaining data specifying a pre-trained multi-modal sequence generation neural network, the multi-modal sequence generation neural network comprising: an image encoder neural network having a set of image encoder parameters and configured to process an input image to generate one or more tokens representing the input image; and a token generation neural network having a set of token generation parameters configured to process a sequence of input tokens to generate a sequence of output tokens, wherein the data specifying the pre-trained multi-modal sequence generation neural network comprising data specifying pre-trained values of the image encoder parameters and pretrained values of the token generation parameters; obtaining medical training data comprising a plurality of medical training examples, each of the plurality of medical training examples comprising (i) a respective medical input that comprises one or more medical images and (ii) a respective target medical output for a respective medical task; and training the pre-trained multi-modal sequence generation neural network on the medical training data to update at least the pre-trained values of the image encoder parameters.

2. The method of claim 1. wherein training the pre-trained multi-modal sequence generation neural network on the medical training data comprises training the pre-trained multi-modal sequence generation neural network to update at least the pre-trained values of the image encoder parameters and the pre-trained values of the token generation parameters.

3. The method of any preceding claim, further comprising: after training the pre-trained multi-modal sequence generation neural network on the medical training examples: receiving a new medical input comprising one or more new medical images; and processing the new medical input using the multi-modal sequence generation neural network to generate a new medical output, comprising:processing each of the one or more new medical images using the image encoder neural network to generate one or more tokens representing each of the one or more new medical images; and processing a new input comprising the one or more tokens representing each of the one or more new medical images using the token generation neural network to generate a new output sequence of tokens.

4. The method of any preceding claim, wherein the one or more medical images comprise one or more of CXR images, mammography images, pathology' images, dermatology' images, or retinal images.

5. A method performed by one or more computers, the method comprising: receiving a medical input, the medical input comprising a genomics input, the genomics input comprising genomics data or data derived from the genomics data; and processing the medical input using a multi-modal sequence generation neural network to generate a medical output for a medical task, the processing compnsing: generating an image that represents the genomics input; processing the image using an image encoder neural network of the multimodal sequence generation neural network to generate one or more tokens representing the genomics data; and processing an input that comprises (i) the one or more tokens representing the genomics data using a token generation neural network of the multi-modal sequence generation neural network to generate a sequence of output tokens.

6. The method of claim 5, wherein: the medical input is a multi-modal medical input that comprises the genomics data and other data of one or more other modalities, the processing comprises processing the other data of the one or more other modalities to generate tokens representing the other data, and the input to the token generation neural network comprises (i) the one or more tokens representing the genomics data and (ii) the tokens representing the other data.

7. The method of claim 5 or claim 6, wherein generating an image that represents the genomics data comprises:determining, from the genomics data, one or more risk scores for each of a plurality of traits; and projecting the risk scores for the plurality of traits to generate a plurality of patches of the image.

8. The method of claim 7, wherein each pixel of the image has a respective value for each of a plurality of channels, and wherein the respective value of each channel corresponds to a projection of a corresponding risk score of a respective p-value threshold.

9. The method of claim any one of claims 5-8, further comprising: prior to receiving the medical input: obtaining data specifying pre-trained values of the image encoder parameters and pre-trained values of the token generation parameters; obtaining medical training data comprising a plurality of medical training examples, each of the plurality of medical training examples comprising (i) a respective medical input that comprises a training image representing training genomics data and (ii) a respective target medical output for a respective medical task; and training the multi-modal sequence generation neural network on the medical training data to update at least the pre-trained values of the image encoder parameters.

10. The method of claim 9, wherein training the multi-modal sequence generation neural network on the medical training data comprises training the multi-modal sequence generation neural network to update the pre-trained values of the image encoder parameters and the pre-trained values of the token generation parameters.

11. A method performed by one or more computers, the method comprising: receiving an input, the input comprising a volumetric image; and processing the input using a multi-modal sequence generation neural network to generate an output for a task, the processing comprising: processing the volumetric image using a video encoder neural network of the multi-modal sequence generation neural network to generate one or more tokens representing the volumetric image; and processing an input that comprises the tokens representing the volumetric image using a token generation neural network of the multi-modal sequence generationneural network to generate a sequence of output tokens.

12. The method of claim 11, wherein the input is a medical input, the volumetric image is a volumetric medical image and the task is a medical task.

13. The method of claim 12, wherein the output for the medical task is a prediction of at least a portion of a radiology report.

14. The method of any one of claims 11-13, wherein: the input is a multi-modal input that comprises the volumetric image and other data of one or more other modalities. the processing comprises processing the other data of the one or more other modalities to generate tokens representing the other data, and the input to the token generation neural network comprises (i) the tokens representing the volumetric image and (ii) the tokens representing the other data.

15. The method of any one of claims 11-14, wherein processing the volumetric image using a video encoder neural network of the multi-modal sequence generation neural network to generate one or more tokens representing the volumetric image comprises: generating a plurality of video patches from the volumetric image by representing a depth dimension of the volumetric image as a time dimension of a video; and processing the plurality of video patches using the video encoder neural network to generate the one or more tokens.

16. The method of any one of claims 11-15 when dependent on claim 12, further comprising: prior to receiving the medical input: obtaining data specifying pre-trained values of the video encoder parameters and pre-trained values of the token generation parameters; obtaining medical training data comprising a plurality of medical training examples, each of the plurality of medical training examples comprising (i) a respective medical input that comprises a training volumetric medical image and (ii) a respective target medical output for a respective medical task; and training the multi-modal sequence generation neural network on the medical trainingdata to update at least the pre-trained values of the video encoder parameters.

17. The method of claim 16. wherein the respective target medical output is at least a portion of a radiology report.

18. The method of claim 16 or 17, wherein training the multi-modal sequence generation neural network on the medical training data comprises training the multi-modal sequence generation neural network to update the pre-trained values of the video encoder parameters and the pre-trained values of the token generation parameters.

19. The method of any one of claims 11-18, when dependent on claim 12. wherein the volumetric medical image is a computed tomography (CT) volume.

20. The method of any preceding claim, when dependent on claim 6 or claim 14, wherein the other data of the one or more other data comprises text data and wherein processing the other data of the one or more other modalities to generate tokens representing the other data comprises: tokenizing the text data to generate tokens representing the text data.

21. The method of any preceding claim, wherein the medical task comprises one or more of: medical image classification; question answering; report generation; or risk prediction.

22. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1-21.

23. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the respective method of any one of claims 1-21.

Images