Language aligned medical image embeddings

Abstract

A method includes receiving training data comprising medical images and associated text reports. The method further includes training an image encoder to associate image embeddings generated by the image encoder when the image encoder is applied to the medical images with text embeddings generated by a text encoder when the text encoder is applied to the associated text reports. The method additionally includes training a vision-language model based on the image embeddings generated by the trained image encoder to represent the medical images and the associated text reports in a language-aligned embeddings space. The method further includes training an adaptor which connects the vision-language model to an LLM to generate LLM-aligned tokens which emphasize portions of medical image information to the LLM via an attention mechanism. The method further includes providing the trained image encoder, the trained vision-language model, and the trained adaptor to perform one or more inference tasks.

Description

Language Aligned Medical Image EmbeddingsCROSS-REFERENCE TO RELATED DISCLOSURE

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 517,336, filed August 2, 2023, which is incorporated herein by reference in its entirety.BACKGROUND

[0002] Machine learning models may be used to process various types of data, including images and text, to generate various desirable outputs. Improvements in the machine learning models and / or the training thereof allow the models to carry out the processing of data faster, to utilize fewer computing resources for the processing, and / or to generate outputs that are of relatively higher quality.SUMMARY

[0003] An example embodiment is a method which includes receiving training data comprising medical images and associated text reports; training an image encoder to associate image embeddings generated by the image encoder when the image encoder is applied to the medical images with text embeddings generated by a text encoder when the text encoder is applied to the associated text reports; training a vision-language model based on the image embeddings generated by the trained image encoder to represent the medical images and the associated text reports in a language-aligned embeddings space; training an adaptor which connects the visionlanguage model to a large language model (LLM) to generate LLM-aligned tokens which emphasize portions of medical image information to the LLM via an attention mechanism, wherein the LLM is configured to generate an output based on the LLM-aligned tokens; and providing the trained image encoder, the trained vision-language model, and the trained adaptor to perform one or more inference tasks.

[0004] A further example embodiment involves one or more non-transitory computer readable media comprising program instructions executable by one or more processors to perform functions comprising: receiving a medical image; applying an image encoder to the medical image to generate image embeddings; applying a vision-language model to the image embeddings generated by the image encoder to generate a representation of the medical image in a language-aligned embeddings space; and performing one or more inference tasks based onthe representation of the medical image in the language-aligned embedding space.

[0005] An additional example embodiment involves a system comprising one or more processors and one or more non-transitory computer readable media comprising program instructions executable by the one or more processors to perform functions comprising: receiving a medical image; applying an image encoder to the medical image to generate image embeddings; applying a vision-language model to the image embeddings generated by the image encoder to generate a representation of the medical image in a language-aligned embeddings space; and performing one or more inference tasks based on the representation of the medical image in the language-aligned embedding space.

[0006] A further example embodiment is a method which includes receiving a medical image; applying an image encoder to the medical image to generate image embeddings; applying a vision-language model to the image embeddings generated by the image encoder to generate a representation of the medical image in a language-aligned embeddings space; and performing one or more inference tasks based on the representation of the medical image in the language- aligned embedding space.

[0007] Another example embodiment involves one or more non-transitory computer readable media comprising program instructions executable by one or more processors to perform functions comprising: receiving a medical image; applying an image encoder to the medical image to generate image embeddings; applying a vision-language model to the image embeddings generated by the image encoder to generate a representation of the medical image in a language-aligned embeddings space; and performing one or more inference tasks based on the representation of the medical image in the language-aligned embedding space.

[0008] An additional example embodiment involves a system comprising one or more processors and one or more non-transitory computer readable media comprising program instructions executable by the one or more processors to perform functions comprising: receiving a medical image; applying an image encoder to the medical image to generate image embeddings; applying a vision-language model to the image embeddings generated by the image encoder to generate a representation of the medical image in a language-aligned embeddings space; and performing one or more inference tasks based on the representation of the medical image in the language-aligned embedding space.BRIEF DESCRIPTIONS OF THE FIGURES

[0009] Figure 1 illustrates training for the ELIXR-C model, in accordance with examplesdescribed herein.

[0010] Figure 2 illustrates inference for the ELIXR-C model, in accordance with examples described herein.

[0011] Figure 3 illustrates training for the ELIXR-B model, in accordance with examples described herein.

[0012] Figure 4 illustrates inference for the ELIXR-B model, in accordance with examples described herein.

[0013] Figure 5 illustrates the PRAWN model, in accordance with examples described herein.

[0014] Figure 6 illustrates a flow chart, in accordance with examples described herein.

[0015] Figure 7 illustrates a flow chart, in accordance with examples described herein.DETAILED DESCRIPTIONI. Overview

[0016] Artificial intelligence systems for medical imaging have traditionally focused on highly specific tasks and have generalized inconsistently to new problems. The combination of large language models (LLMs) and vision encoders offers the potential to address some of these challenges. Disclosed herein an approach that enables efficient training of multimodal models using routinely collected medical images and their associated text reports, and adds the ability to perform a diverse range of tasks with rich expressive outputs. This approach unlocks the potential for a new generation of medical Al applications, supporting workflows including high performance zero-shot and data-efficient classification, semantic search, visual question answering (VQA), and radiology report quality assurance (QA).

[0017] Examples are described with respect to chest X-rays, but examples are also contemplated with respect to a variety of medical imaging and radiological imaging. Additional examples are also contemplated in the context of magnetic resonance imaging (MRIs), computerized tomography (CT) scans, ultrasounds, mammograms, positron emission tomography (PET) scans, radiographs, pathology, diabetic retinopathy, other types of X-rays, as well as other types of imagery or contexts.

[0018] The machine learning models and methods for training provided herein exhibit a variety of improvements relative to prior models and training methods, such as:• Ability to achieve improved accuracy when trained on smaller datasets;• Ability to build models rapidly to prototype new use cases, adapt to distributionshifts with a small amount of new training data, or use alternative publicly available LLMs;• Ability to leverage types of data that are ubiquitous in healthcare, so that the training process does not require expensive manual label curation by experts.

[0019] The machine learning models described herein can be applied to a variety of applications, such as:• Zero-shot classification;• Data-efficient classification;• Semantic search;• Visual question answering (VQA);• Report quality assurance (QA);• VQA or QA involving a series of intermediate prompts from a first LLM and a final answer from a second LLM. Optionally, the first LLM may be image aware while the second LLM may only summarize text.

[0020] Further examples described herein involve whole-slide images (WSIs). Many important diagnostic and treatment decisions involve interpretation of histopathology slides, which when digitized produce gigapixel WSIs. Recent machine learning advances in visionlanguage modeling raise new opportunities for analysis of pathology images. The sheer size of WSIs introduces unique challenges to image-text alignment. Additionally the text from pathology reports might simultaneously aggregate interpretation across multiple slides while also highlighting key findings from small subregions. As such, pathology reports remain a largely untapped source of supervision in computational histopathology. Instead, most machine learning efforts have relied on supervision from region-of-interest annotations or selfsupervision at the patch-level. We describe a vision-language model based on the BLIP-2 framework using paired WSI and pathology reports. This enables applications utilizing a shared image-text embedding space along with grafting of the WSI encoder onto a pre-trained and frozen large language model (LLM) for WSLbased generative text capabilities. For model development and evaluation we utilize a de-identified real -world dataset of over 350,000 WSIs paired with diagnostic text from associated pathology reports spanning a wide range of diagnoses, procedure types, and tissue types. We evaluate our model on various slide-level tasks, including pathologist evaluation of text retrieval from WSI embeddings, WSL classification, text generation, and workflow prioritization (slide-level triaging). Modelgenerated text for WSIs was rated via pathologist review as accurate, without clinicallysignificant error or omission, for 78% of WSIs on average, demonstrating exciting potential capabilities of language-aligned WSI embeddings paired with LLMs.II. ELIXR Examples

[0021] Our approach, which we call Embeddings for Language / Image-aligned X-Rays, or “ELIXR”, leverages a language-aligned image encoder combined or “grafted” onto a fixed LLM, PaLM 2, to perform a broad range of tasks. We train this lightweight adapter architecture using images paired with corresponding free-text radiology reports from the MIMIC-CXR dataset. Evaluation of zero-shot and data-efficient classification was performed using the public CheXpert and ChestX-ray datasets, as well as a private dataset from five hospitals in India. Semantic search was evaluated across four themes using the MIMIC-CXR test set. VQA was evaluated using the VQA-RAD benchmark and the MIMIC-CXR test set. LLM output for report QA was evaluated on the MIMIC-CXR test set by a board-certified thoracic radiologist.

[0022] ELIXR achieved state-of-the-art performance on zero-shot chest X-ray (CXR) classification (mean AUC of 0.850 across 13 findings), data-efficient CXR classification (mean AUCs of 0.893 and 0.898 across five findings (atelectasis, cardiomegaly, consolidation, pleural effusion, and pulmonary edema) for 1% (-2,200 images) and 10% (-22,000 images) training data), and semantic search (0.76 normalized discounted cumulative gain (NDCG) across nineteen queries, including perfect retrieval on twelve of them). Compared to existing data- efficient methods including supervised contrastive learning (SupCon), ELIXR required two orders of magnitude less data to reach similar performance. ELIXR also showed promise on CXR vision-language tasks, demonstrating overall accuracies of 58.7% and 62.5% on visual question answering and report quality assurance tasks, respectively. These results suggest that ELIXR is a robust and versatile approach to CXR Al. LLM-aligned multimodal models can unlock the value of chest X-rays paired with radiology reports to solve a variety of previously challenging tasks.

[0023] The past decade has witnessed dramatic advances in artificial intelligence (Al) in medical imaging. Numerous deep learning systems have been developed that can achieve expert-level performance across a range of medical tasks. However, clinical and technical limitations have resulted in an implementation gap that has impeded the impact of Al in real world health applications at scale. Key challenges include the significant cost of curating high quality training datasets, restriction of Al development to narrow, highly specific tasks, difficulty in processing multimodal data, and limited interpretability that has hampered effective human-AI interaction

[0024] Until recently, Al systems for medical imaging have been largely built using vision- only models, including convolutional neural networks (CNNs) and vision transformers. Using a traditional fully supervised approach, training CNNs and vision transformers is an expensive and time-consuming process that requires large quantities of expertly annotated data. In addition, such networks are usually limited to performing discrete tasks, such as image classification, object detection, and segmentation. On the input side, these networks also take in only images, usually of just one modality. In contrast, healthcare workflows are typically multimodal in nature, with clinicians leveraging a diverse array of inputs (e.g. clinical notes, images, investigations) when making diagnoses and treatment decisions.

[0025] Large language models (LLMs) are part of a new generation of versatile transformerbased Al models that are trained on massive datasets and demonstrate previously unseen abilities to generalize to a range of tasks, despite requiring very little task-specific data. The multimodal combination of vision models and LLMs presents a range of exciting possibilities including zero-shot image-to-text generation that can follow natural language instructions. In medical imaging, these advances over the potential to address limitations of vision-only models by enabling model training using ubiquitous medical images that have paired free text reports, adding capabilities to carry out a diverse range of tasks, accurately coping with the long-tail of diagnoses, enabling true multimodal inference, and presenting new options for expressive human-computer interaction.

[0026] Our lightweight vision-language adapter model (ELIXR) combines or “grafts” a vision encoder with a frozen LLM to perform a wide range of vision-language tasks broadly relevant to medical imaging. The discussion herein focuses on chest X-rays (CXRs) due to the wide availability of image-text paired data, but the methods are also applicable to other image modalities. We describe a number of key advantages of ELIXR.

[0027] First, ELIXR achieves state-of-the-art performance for zero-shot classification, data- efficient classification, and semantic search of thoracic conditions across a range of datasets. These factors may enable a new class of models that can address the long-tails of diagnoses in the medical domain, and provide a path for more broadly useful Al tools in the diagnostic workflow.

[0028] Second, adapting an image encoder to an LLM using ELIXR is a fast and resourceefficient method of training compared to full finetuning of an LLM, leveraging a modest-sized frozen LLM and a data-efficient frozen vision encoder. Building models on top of ELIXR can be done rapidly to prototype new use cases, adapt to distribution shifts with a small amount ofnew training data, or use alternative publicly available LLMs.

[0029] Third, ELIXR’s synthesis of imaging and text unlocks a new generation of medical Al applications. We demonstrate semantic search, visual question answering (VQA), and radiology report quality assurance (QA), but there are countless potential applications across the medical domain that can be addressed using the proposed multimodal framework.

[0030] Fourth, ELIXR is trained using paired CXR and free text radiology reports - data that are ubiquitous in healthcare. The training process does not require expensive manual label curation by experts. Such an approach unlocks the value of routinely collected medical data to develop Al systems at far greater scale and at lower overall cost than previously possible.

[0031] The ELIXR system described herein was trained in two stages (ELIXR-C and ELIXR- B).

[0032] Figure 1 illustrates training and Figure 2 illustrates inference for the ELIXR-C model using Contrastive Language-Image Pre-training (CLIP). This uses images and radiology reports to align our previously published pre-trained supervised contrastive learning-based (SupCon) vision-only CXR model with a T5 language encoder. CLIP uses a contrastive loss function, which encourages the model to bring the representations of an image and its associated text (in this case, the radiology report) closer together in a high-dimensional space, while simultaneously pushing apart representations of mismatched images and text.

[0033] Figure 3 illustrates training and Figure 4 illustrates inference for an LLM-aligned adapter network ELIXR-B, based on the Bootstrapping Language-Image Pre-training 2 (BLIP- 2) architecture. ELIXR-B is built directly upon ELIXR-C, where it aims to extract location- aware features from the unpooled spatial ELIXR-C image embedding space and map them to the LLM’s language token space. In examples described herein, PaLM 2-S was used as the LLM. By serving as an adapter between the image encoder and the LLM, ELIXR-B passes information between vision and language encoders via an attention mechanism, and allows for leveraging the existing knowledge and reasoning abilities of the LLM to interpret the images and perform various vision-language tasks (e.g. captioning, VQA). For computation and data efficiency, ELIXR-C and PaLM 2-S are kept frozen, and only the adapter between them is trained. This can be thought of as a way of grafting an image encoder onto an LLM. More specifically, following the BLIP-2 architecture, there are two phases to ELIXR-B training: vision-language representation learning (phase 1) and vision-language generative learning (phase 2).

[0034] In the first phase, the vision-language model (the Q-Former) is trained to understandand represent both CXRs and reports in a shared embedding space by jointly employing three different tasks: a) image-text contrastive learning (ITC), b) image-grounded text generation (ITG), and c) image-text matching (ITM). Standard contrastive loss is applied for image-text contrastive learning; image-grounded text generation is modeled as a classification problem (e.g., which token in the vocabulary should be chosen at each output position) and optimized by cross-entropy loss; image-text matching is modeled as a binary classification problem (image-text matched / unmatched) and optimized by cross-entropy loss. This results in a model that can extract key information from the image embeddings and align it with the report text embedding space.

[0035] In the second phase, a multilayer perceptron (MLP) that connects the Q-F ormer with the LLM, and the Q-F ormer itself are further trained to generate the impressions section of radiology reports based upon the image embeddings from ELIXR-B using the LLM. The language modeling (standard cross-entropy) loss is used to guide the training. The result is that the Q-Former is able to produce LLM-aligned tokens based on the image and feed the most useful information to the LLM, while removing irrelevant visual information.

[0036] As illustrated in Figure 3, ELIXR-B is trained in two phases. In the first phase, the model bootstraps vision-language representation in the Q-Former with three learning objectives (image-text contrastive learning (ITC), image-grounded text generation (ITG), image-text matching (ITM) losses) to learn from embeddings from a frozen image encoder. In the second phase, the model bootstraps vision-to-language generation from a frozen large language model. ELIXR-C takes images and / or text as input and outputs embeddings; ELIXR- B takes images and / or text as input and outputs embeddings and / or text. Therefore, for text output tasks such as VQA and QA, only ELIXR-B is used.

[0037] For zero-shot and data-efficient classification of CXR findings, area under the receiver operating characteristic curve (AUC) for each finding class is reproduced since the tasks are all binary classifications. We compared ELIXR-C and ELIXR-B performance with SupCon and previous SOTA models CheXzero and ConVIRT as baselines. We also evaluated the effect of varying the training dataset size.

[0038] To perform zero-shot classification using ELIXR-C, we adopted a prompting strategy described previously. Note that because ELIXR-C was pretrained on images and reports that include the findings being classified, “zero-shot” refers more to open-ended classification without using explicit labels during training than it does to never having observed these findings during training. The positive and negative text prompts were passed through the textencoder to obtain their text embeddings. These text embeddings were each average pooled and normalized, resulting in one representative embedding for each prompt. A cosine similarity was then calculated between the image embedding and these two text embeddings. A softmax of the cosine similarity for the positive prompt was used to produce a classification score so that AUC can be calculated. When there are multiple prompts for the positive or negative case, the mean of the cosine similarities of these prompts was taken before computing the softmax.

[0039] For ELIXR-B, the image as well as positive and negative text prompts were passed through the Q-F ormer to get the output image query tokens and the sequence of text embeddings. The first in the sequence of text embeddings is the single special classification token ([CLS]). The cosine similarity is calculated between this classification token and each of the output image query tokens. The highest cosine similarity for the positive prompt and the highest cosine similarity for the negative prompt are passed to the softmax function to get the final classification score.

[0040] To perform data-efficient classification, a nonlinear classifier, an MLP consisting of two layers of 512 and 256 neurons, was trained on top of the frozen image encoder. We adopted a learning rate of 0.2, batch size of 512, and trained for 300 epochs with the layer-wise adaptive rate scaling (LARS) optimizer. To make results more directly comparable to CheXzero and ConVIRT, we also trained linear classifiers for the 1% and 10% training data samples.

[0041] For data-efficient classification, the AUC was averaged across 10 repeats of randomly subsampled training sets. To obtain an overall summary, the AUCs were (also) averaged across all tasks and all datasets (CheXpert, CXR-14, and INDI) for each training set size. 95% confidence intervals were calculated using twice the standard error across the 10 randomly drawn train set samples, and hypothesis testing was based on comparing the model closest to the mean performance across the 10. For the zero-shot models, 95% confidence intervals were estimated and AUCs were compared using the DeLong method.

[0042] For semantic search, confidence intervals were calculated by bootstrapping with 1,000 samples, p-values were calculated from two-sided permutation tests with 1,000 iterations.

[0043] For semantic search (also known as text-image retrieval), we provided queries across four topics, including queries for single findings, laterality-specific, severity-specific, and nuanced features. For each query, the analysis focused on the top five retrieved images based on model predictions. There were seven single finding queries, three laterality queries, four severity queries, and five nuanced queries. We compared ELIXR-C and ELIXR-B against the current state-of-the-art model MedCLIP25 using the publicly available code and modelcheckpoints. The MIMIC-CXR test set served as the data pool for retrieval.

[0044] For semantic search using ELIXR-C, we computed the cosine similarity between the query text embedding and the image embeddings from the data pool. The top five images with the highest cosine similarity were retrieved. For semantic search using ELIXR-B, we adopted the two-stage method in BLIP-210. In the first stage, we retrieved 128 images with the highest cosine similarity similar to CLIP as the candidates. In the second stage, the image-text matching score (i.e. the matched class probability) was computed to rerank these 128 candidates. The top five images with the highest image-text matching score from these 128 candidates were then returned.

[0045] For evaluation, a board- certified thoracic radiologist (CL) scored the semantic search results as follows: 0 = irrelevant or factually incorrect, 1 = close fit to the query, but not the one intended (e.g., retrieved “bilateral effusion” for “right effusion” query), and 2 = reasonable match. Precision at five (precision@5) and normalized discounted cumulative gain at 5 (NDCG@5) were calculated to evaluate the quality of retrieval. For the ideal rank normalization, we assumed the data pool contained at least five matched queries, and thus we set the relevance scores to be all 2s for the ideal relevance. Precision for cases that had reasonable match (score = 2) and precision for at least somewhat fit (score >1) were also measured.

[0046] To establish the context of the difficulties of the retrieval, we estimated the total count of images with corresponding pathologies. To do so, we adopted an LLM-based approach to detect mislabeling within the MIMIC-CXR test set. Candidates for correction were first identified by a keyword search on the radiology reports. Next, a medically tuned LLM (Med- PaLM 229) was applied to ensure that the label was consistent with the report, and a board- certified thoracic radiologist (CL) adjudicated cases where the LLM results differed from the ground truth in MIMIC-CXR.

[0047] We used two different datasets for visual question answering (VQA) evaluation of ELIXR-B: a subset of the MIMIC-CXR test set, which is from the same domain as a part of the training dataset, and the subset of chest x-ray cases (IMAGEORGAN == “CHEST” AND ANSWER != “ultrasound”) in the VQA-RAD dataset, from a different domain than the trainingdataset. A small held-out tuning set consisting only of “yes” or “no” answers from VQA-RAD was used for model checkpoint selection. Specifically, we used our own test and tuning splits for the 793 VQA-RAD CXR question and answer pairs, since the original test set (column Phrase_type / QID_para == “test freeform” or “test_para”) shares images with the development set (QID_para == “freeform” or “para”). By contrast, the images in our test set (584 questions on 73 images) are disjoint from the tune set images, and thus were unseen by the model until evaluation time. Moreover, we did not train the model on any VQA-RAD question and answer pairs, but only used it for checkpoint selection based on our smaller tuning set of 209 VQA pairs across 25 images. We compared ELIXR-B against the SOTA MedVInT30 model using the publicly available code and a model checkpoint that, like ours, was not finetuned on VQA- RAD.

[0048] The second VQA dataset is a subset of cases with findings from MIMIC-CXR test set, as labeled in the MIMIC-CXR- JPG project, with eight cases for each for the following findings: “No Finding”, “Pneumothorax”, “Pleural Effusion”, “Edema”, “Consolidation OR Pneumonia” (collected as a single category), and “Lung Lesion”. For each case with a finding present (with finding presence confirmed by a board-certified thoracic radiologist (CL)), we queried the model with a set of finding-specific questions, covering its presence, location, and its severity, size or type, where applicable. In addition, we asked two additional findingindependent questions per case, and three finding-independent questions for cases without any of the above findings.

[0049] Since both phases of ELIXR-B were trained to generate the impression section of a radiology report, but neither was tuned to follow instructions, we utilized both the impression generation results from phase 1 (same ITG setup as BLIP-2,10) and LLM-aligned tokens from phase 2 to facilitate the VQA use case. Specifically, after generating the impression with phase 1, we then ran inference for the phase 2 model, using the following dialog prompt for the PaLM 2-S LLM wherein the LLM-aligned tokens from phase 2 inference were placed at the beginning of the prompt at ({aligned LLM tokens}), the phase 1-generated impression was fed to ({impression}), and the specific question ({question}) was added afterwards:{aligned LLM tokens}[Bot] I'm a helpful Chest X-ray assistant, I can help you interpret the above image.[User] What are the findings?[Bot] {impression}[User] Based on the above chest x-ray, the findings, and / or your medical knowledge, answer the following question: {question} [Bot]

[0050] In terms of answer grading, model answers that could programmatically be mapped to “yes” or “no” and compared against “yes”-or-“no”-expected answers were automatically graded with 1.0 for a match, and 0.0 otherwise. A board-certified thoracic radiologist (CL) graded the remaining model answers using correctness scores 0.0, 0.5 or 1.0. For the MIMIC- CXR test set samples, the same radiologist using the same scoring rubric evaluated all 216 answers generated by the model based on their assessment directly of the CXR image and original radiology report. Sensitivity, specificity and accuracy values were calculated from the average radiologist grades on respective subsets where the condition was positive, negative or both.

[0051] In the report quality assurance task, we simulated the situation where a radiology report contained errors and used ELIXR-B to identify and suggest corrections to these errors. Errors that we evaluated included swapping laterality of a finding (“left” to “right” or vice versa), adding an erroneous finding that was not clearly present in the image, or omitting a major finding that was present in the image.

[0052] To evaluate ELIXR’s performance, we first identified a subset of cases with findings from the MIMIC-CXR test set, as labeled in the MIMIC-CXR-JPG project, that would be most relevant for evaluating quality assurance: “No Finding”, “Pneumothorax”, “Pleural EDusion”, “Edema”, “Consolidation OR Pneumonia” (collected as a single category), and “Lung Lesion”. We randomly selected eight cases per finding. For each case we defined the “primary finding” as the finding that was used in the query that yielded that given case, even if other findings, including more clinically significant findings, were present in the case. For example, if we searched for cases with pleural effusion and discovered a case with both pleural effusion and pulmonary edema present, the primary finding for that case was considered to be pleural effusion. We filtered cases to ensure that (1) an impression section was present in the report (true for approximately 83% of cases) and (2) the primary finding was present unambiguously in the image, based on the impression of a board-certified thoracic radiologist (CL).

[0053] Within the set of eight cases for each finding category, we made an alteration per case as follows: two cases we left with unaltered impression sections (the “control” cases), two cases had the primary finding’s laterality swapped, two cases had the primary finding removed, and two cases had an erroneous finding added. For each alteration, we made further minimalmodifications to the impression text as needed to ensure that it was internally consistent (e.g., if we added an extraneous finding of “moderate pulmonary edema,” we confirmed that the rest of the report was clinically consistent with this alteration). For ’’Edema”, because pulmonary edema tends to be a bilateral finding, we did not include any cases with swapped laterality and instead included three control cases, three cases where an extraneous finding was added, and two cases where the primary finding of ’’Edema” was removed, still resulting in eight cases total. For the cases labeled as "No Finding", modifications like laterality swapping and finding removal were not feasible. Therefore, we chose four control cases and four cases in which we introduced a false finding, providing us with a total of eight cases. After making these alterations, we had 48 total cases, each of which had an associated CXR image and an impression section that was either unaltered (control) or altered according to the above process.

[0054] To generate the model outputs for evaluation, we first ran ELIXR-B phase 1 inference on the image to produce a report with ELIXR’s findings. We then fed the image, the generated report, and a series of prompts that covered the main possible set of findings in chest x-rays to ELIXR-B phase 2.

[0055] For each prompt, the Q-F ormer-encoded image embeddings preceded the text hard prompt as input into PaLM 2-S, and the control or altered impressions section was inlined in place of the variable {altered impression}. ELIXR provided an assessment as to whether each finding existed, and if so, assessed the laterality in order to support the laterality swap detection task. The prompts were as follows.1. If there's an endotracheal tube (ET tube) in the chest x-ray, tell me whether it's mal- positioned or well-positioned. If there's no ET tube, respond 'no'2. Is there any evidence of pneumothorax in the chest x-ray? If so, on which side(s)?3. Are there any signs of pleural effusion present in the x-ray? If so, on which side(s)?4. Are there any visible signs of pulmonary edema in the CXR? If so, on which side(s)?5. Are there any signs of pneumonia or lung infection? If so, on which side(s)?6. Are there any signs of consolidation or lung infection in this patient's chest x-ray? If so, on which side(s)?7. Are there any signs of atelectasis in the lungs? If so, on which side(s)?8. Are there any signs of fibrosis in the lungs? If so, describe it9. Are there signs suggestive of a nodule or mass in this patient's chest x-ray? If so, on which side(s)?10. Is the cardiac silhouette size normal or enlarged?11. Is a hiatal hernia present? If so, on which side(s)?12. Are there any signs of acute skeletal fracture? If so, where?

[0056] This initial part of the workflow was essentially comprehensive VQA. We concatenated these responses into a single piece of text to constitute ELIXR’s comprehensive findings. We then fed the concatenated questions and answers into a non-image-aligned LLM, Med-PaLM 229, to determine whether there were any missing findings, erroneously added findings, or laterality swaps. The two prompts to do this were as follows. Note that while MIMIC-CXR contains full reports for most cases, we altered and evaluated only the impression section of the reports.First Prompt: You are an expert radiologist. These are your responses to a comprehensive assessment of a patient's chest x-ray (CXR).ASSESSMENT: {questions and ELIXR answers separated by new lines}.A radiology resident has written the following radiology report for the same CXR. RESIDENT'S REPORT: {altered impression}.Are there any findings that you mark positive or abnormal in your assessment but that the resident either marks absent / negative or simply does not mention? If so, what are the findings?Second Prompt: You are an expert radiologist. These are your responses to a comprehensive assessment of a patient's chest x-ray (CXR).ASSESSMENT: {questions and ELIXR answers separated by new lines}.A radiology resident has written the following radiology report for the same CXR. RESIDENT'S REPORT: {altered impression}.Are there any findings that you mark negative or normal in your assessment but that the resident marks positive / abnormal in his report? If so, what are the findings?

[0057] LLM responses were graded by a board-certified thoracic radiologist (CL). If the LLM correctly described the alteration (or identified an unaltered, control report as being correct and complete), the LLM output was scored as correct; if the LLM failed to identify the alteration, even if it gave an otherwise correct response, the output was scored as incorrect.

[0058] ELIXR-B and ELIXR-C demonstrated zero-shot classification performance on five findings (“atelectasis”, “cardiomegaly”, “consolidation”, “pleural effusion”, and “pulmonary edema”) that was comparable to SupCon’s classification performance when trained on the entirety of the CheXpert train set (-224,000 examples). Note that although ELIXR’s vision encoder was initialized from a SupCon checkpoint, CheXpert was not used for pretrainingSupCon at all, only for training small downstream classifiers. Thus, CheXpert is completely held out from ELIXR-B and ELIXR-C zero-shot. Across the 13 findings (excluding “No Finding”) from the CheXpert test set, ELIXR-C and ELIXR-B both surpassed the state-of-the- art zero-shot performance from CheXzero.

[0059] ELIXR-B and ELIXR-C both set a new state-of-the-art for data-efficient linear probe classification on CheXpert’s five main findings (“atelectasis”, “cardiomegaly”, “consolidation”, “pleural effusion”, “pulmonary edema”) using 1% and 10% of the train set, outperforming even the fully-finetuned ConVIRT. ELIXR-B and ELIXR-C also both demonstrated data-efficient performance superior to SupCon or, to put it another way, demonstrated data-efficient performance equivalent to SupCon using roughly two orders of magnitude less training data.

[0060] ELIXR-B outperformed both ELIXR-C and the state-of-the-art MedCLIP on the retrieval quality of top-5 retrieved images. For each query group, we computed the average metrics across the queries. NDCG@5 and Precision@5 of ELIXR-B were consistently better than ELIXR-C across all query groups. ELIXR-B scored higher than MedCLIP on NDCG@5 for all query groups and on Precision@5 (score = 2) for three out of four query groups.

[0061] On twelve out of nineteen queries, ELIXR-B demonstrated perfect retrieval, including for laterality-specific queries like “right pleural effusion,” severity-specific queries like “moderate cardiomegaly,” and nuanced queries like “nasogastric tip reaches stomach.” Notably, we found both ELIXR-C and ELIXR-B performed worse on fracture- and pneumothorax-related queries than queries related to other findings. This could be partially due to the low prevalence of these two pathologies in the MIMIC test set (4% for fracture and 3% for pneumothorax) compared to >10% prevalence for other pathologies. In some cases, retrieval improved when using more specific queries, e.g. adding a laterality or severity modifier to a general finding. For example, ELIXR-C and ELIXR-B scored 0.723 and 0.83 for “left pneumothorax” as compared to 0.131 and 0.214 for “pneumothorax.” These results point to the sensitivity of these models to prompting style.

[0062] On more challenging text-generation tasks, ELIXR-B demonstrated overall accuracies of 58.7% and 54.8% on two visual question answering (VQA) datasets, VQA-RAD (CXR-only questions) and MIMIC-CXR test, as well as 62.5% on report quality assurance (QA) on MIMIC-CXR test. Notably, ELIXR-B surpassed the accuracy of the SOTA model MedVInT which wasn’t finetuned on VQA-RAD.

[0063] Examples described herein are based on ELIXR, a multimodal model that grafts alanguage-aligned vision encoder onto a frozen LLM. The model was trained using CXR images paired with their free-text radiology reports, without the requirement for expensive expert data curation. The model achieved state-of-the-art performance for zero-shot classification, data- efficient classification, and semantic search tasks, while also demonstrating potential in visual question answering and radiology report quality assurance. The modular architecture has the advantage of being easily adaptable to other tasks, with the ability to swap in different vision encoders and base LLMs as required.

[0064] The ELIXR architecture is data and computationally efficient. Previously we demonstrated that small nonlinear classifiers trained on a frozen SupCon vision encoder can outperform fully supervised models in low-data settings. With ELIXR, we have improved upon the data efficiency of supervised contrastive learning by two orders of magnitude. This offers the potential to train highly accurate models that are capable of addressing the long tail of diagnoses (including rare diseases), with only a fraction of the requirement for expert-curated training data. The process of prototyping a model for a new task also becomes simplified and more widely accessible, requiring only the design of positive and negative prompts using natural language without a requirement for machine learning expertise. We hope that these approaches will enable a wider range of researchers to engage in a broader array of diagnostic research questions and will allow medical professionals to develop models for underserved populations or understudied diseases.

[0065] By leveraging a pre-trained frozen vision encoder and a frozen LLM, we were able to train ELIXR in a highly compute-efficient manner. Backpropagation of gradients during the second stage of training is only performed for the Q-Former and MLP components, which are orders of magnitude smaller than the vision encoder and frozen LLM. In comparison to finetuning the vision encoder and / or LLM, this approach to training can be easily used by others who do not have access to substantial compute hardware. Furthermore, when adapting to a new task, or when newer generations of vision encoders and LLMs become available, it is straightforward and relatively inexpensive to train a new ELIXR model to take advantage of these advances.

[0066] LLMs offer a deep natural language understanding that enables a new range of possibilities for rich interaction between clinicians and Al. We demonstrate early promising capabilities in semantic search, vision question answering, and report quality assurance-all tasks that cannot be achieved easily using traditional vision-only models.

[0067] Semantic search unlocks the ability to search within an image for features of interestusing free text prompts. ELIXR demonstrated high retrieval precision across a broad range of queries, including perfect retrieval of five out of seven findings-related queries (e.g. “central venous catheter”), two out of three laterality-specific queries (e.g. “right pleural effusion”), two out of four severity-related queries (e.g. “moderate cardiomegaly”), and three out of five nuanced queries (e.g. “nasogastric tube reaches stomach”). This capability could be used by researchers to identify images for study datasets, by clinicians to search for specific images of interest from a patient’s historical record, by educators to find examples for teaching purposes, as well as in many other applications.

[0068] ELIXR also enables rich human-AI interaction through visual question answering. We benchmark our VQA performance on the chest X-ray subset of the VQA-RAD dataset, yielding accuracy of 69.1% across closed type questions, and 37.9% across open type questions. In contrast to others’ work, we do not train on VQA-RAD, as the standard VQA-RAD data splits exhibit an overlap with images in both train / test that risk artificially inflating performance. In addition, we report accuracy based on assessment by a board-certified thoracic radiologist rather than uni gram matching (BLEU-1), since BLEU-1 does not comprehensively reflect the quality of VQA.

[0069] Finally, we demonstrate the ability of ELIXR to use its understanding of CXRs to check for errors in written radiology reports. We envision that such a capability could have utility in the hospital setting, potentially acting as an advanced multimodal “spell check” to alert radiologists to suspected inconsistencies in their reports, improving both quality and consistency of care. It could also act as an “Al Mentor”, evaluating and mentoring more junior radiologists, including in settings where resource limitations result in less opportunity for oversight by senior radiologists. VQA and QA were heavily dependent on the specific prompts used, which were in turn affected by the LLM used by ELIXR. With further work into prompt engineering, it is expected that one could make the QA more specific, with fewer false positive reports. This high specificity is likely to be needed in a clinical environment, especially where the error rate is far lower than in this enriched simulated set.

[0070] There are a number of limitations to our work. Firstly, through employing a frozen LLM and vision encoder, we achieve our goals of training efficiency, but this might be at the expense of overall performance. Secondly, ELIXR inherits the current set of wider challenges of LLMs including fragility to changes in prompts, hallucinations, and unwarranted confidence in its answers, even when wrong. We expect that advances in future generations of LLMs will directly lead to improved performance when incorporated into ELIXR. Thirdly, a lack ofestablished, robust benchmarks makes it challenging to compare performance and establish state-of-the-art. Fourthly, we note that the MIMIC CXRs used were highly complex in an intensive care setting, containing multiple findings, which added more complexity than is typical compared to routine hospital X-rays. A non-intensive care CXR dataset that reflects routine wider hospital practice would be valuable to evaluate in future work. Finally, we selected different stages of ELIXR for different tasks based upon a performance versus effciency trade-off. For example, on zero-shot learning tasks, the benefits of ELIXR-B over ELIXR-C were limited. In contrast, VQA and report QA tasks are more text dependent, and benefited from longer training using larger amounts of text data. We found that some categories of findings were challenging across all tasks-for example pneumothorax and fractures-where the difficulty of these cases is only partly explained by their low prevalence and noisy reference standards in the training datasets.

[0071] In further examples, we explicitly contemplate extending these methods to other imaging modalities such as musculoskeletal X-ray, mammography, computed tomography (CT), and also beyond radiology in an effort we are calling Medical Information Adapters. It is also likely that temporal information can be incorporated into this architecture, widening the range of potential real world clinical applications.

[0072] As described herein, we developed and evaluated an efficient vision-language multimodal model for medical imaging that is trained using only medical images paired with free-text radiology reports obtained from routine clinical practice. The method is highly compute and data efficient to train, and we demonstrated promising performance across a range of multimodal radiology tasks. This work is an initial step towards a general purpose X-ray artificial intelligence system.III. PRAWN Examples

[0073] Regarding whole-slide images (WSIs), recent work in the field of digital histopathology has moved beyond training task-specific image classifiers or even image-only foundation models to advances that make use of image-text data for vision-language modeling. The training data for such efforts have predominantly been based on small “patches” or regions-of- interest (ROIs) extracted from within a Whole Slide Image (WSI), paired with associated patchlevel text descriptions. For example, the captions for histopathology images in PubMed articles or educational resources can provide useful image-text pairs. While such sources can provide useful description of local histological features, many clinical and research tasks involve slidelevel or case-level interpretation. Additionally, curated WSI-level text descriptions accuratelypaired with specific slides are less readily available than patch-level captions, particularly at the scale necessary for machine learning based approaches. Even when pathology reports are available, it can be challenging to identify the specific slides that are associated with the reported findings. This is because reporting is typically done for the entire case, but there may be many slides for each case, some of which contribute more meaningfully to the diagnosis and reported findings than others. At least in part due to this data-curation challenge, robust strategies to develop visual -language models for WSIs in pathology have been limited to a small number of recent examples.

[0074] The emergence of large language models (LLMs) and large multimodal models (LMMs) has created an entirely new field of general purpose multimodal and generative Al systems. The ability to utilize both images and text to develop models has benefited from several recent approaches including CLIP, BLIP-2, LLaVa, CoCa, and others. For pathology specifically, a number of recent works describe promising results for vision-language models. These include utilization of a variety of different data sources for image-text pairs, including social media posts by experts, YouTube video and caption curation, pathology reports with patch extraction, and large-scale curation of figure-caption pairs from medical literature and educational resources. Additionally, Sun et al. recently evaluated many publicly available LMMs on a large, patch-based visual question answering (VQA) dataset, which was itself generated and curated with the help of the GPT-4V LMM14. One thing these works have in common is the focus on patch-level modeling. Recently, initial strategies have been described to represent WSIs via different patch-aggregation approaches, and parallel efforts have explored new strategies for establishing useful WSLlevel embeddings and WSLlanguage alignment. These contributions represent important milestones in the field and highlight some of the unique challenges for combining large images and text information at the WSI-level.

[0075] As described herein, we help bridge this gap by presenting Pathology Report Aligned Whole-slide eNcoder (PRAWN), which performs vision-language alignment using approximately 10,000 patches per gigapixel WSI, paired with the corresponding diagnostic findings in associated pathology reports. This approach enables image-text alignment capabilities at a slide-level, bringing us closer to the possibility of automatic report generation and case-level visual question answering for digital histopathology. Specifically, we utilize embeddings from a patch-level foundation model fed into a BLIP-2 framework. We train two models, including one variant trained only with the image-text contrastive loss, and a second variant using the standard two-stage BLIP-2 training that further integrates a frozen LLM toenable WSI-level text generation and basic visual question-answering capabilities. In addition to embedding-based classification performance, we present results for pathologist evaluation of cross-modal retrieval and text generation along with a practical use case demonstration of slide prioritization.

[0076] The primary data used for this work consists of a de-identified dataset (“DS1”) of 354,089 WSIs from a teaching hospital paired with diagnostic text from pathology reports. On visual inspection, the vast majority are hematoxylin and eosin (H&E) stained, with a smaller portion of immunohistochemical (IHC) stained slides also present.

[0077] DS1 is a real -world dataset that is largely representative of the distribution of surgical and biopsy procedures and histopathology findings seen by pathologists in the U.S. As a result, it contains a large number of biopsy specimens (e.g. skin, cervix, colon), many of which are benign or non-cancerous, with cancer cases representing a relatively small portion of total slides, especially for less common types of cancer.

[0078] In order to further enrich our training and evaluation data for cancer cases, we also included de-identified data from The Cancer Genome Atlas (TCGA), which contains WSIs and case-level metadata from a variety of cancer types. For this work, we utilized the set of 12,268 diagnostic WSIs in TCGA across all 32 TCGA study types.

[0079] Pathology specimens are typically processed and accessioned by their case, part, and block. A single case may have several different parts and a single part may have several different blocks, with each block sectioned (ie. cut) to provide one or more slides for histopathology review. This results in the challenge of pairing any given slide with the portion of the pathology report that actually describes the findings on that particular slide. For digitized slides, this means a report may summarize information across anywhere from small regions to 100s of gigapixels.

[0080] As is typical for pathology reports, each case in DS1 has an associated diagnostic text, corresponding to the final diagnosis (ie. “bottom line” text) from the pathology report. These diagnostic texts are structured into part-level text sections. Because each case may have several different parts, we parse these to the part-level via regular expressions. For each part-level text section, there is a “label” (description of tissue site or surgical procedure) and a “finding” (description of diagnostic findings). Because the “label” text is typically based on the specimen processing and preparation, it often includes information such as anatomic location and laterality that may not readily be determined from the WSI alone. Occasionally the findings section also includes this type of information. As such, we apply a further set of regularexpressions to remove information from both the labels and findings that cannot reliably be determined from the WSI.

[0081] As described herein, we focus on modeling at the single WSI-level, rather than across multiple WSIs that may comprise a part or case. As a result, we pair each slide with its associated, part-level subsection of the report based on part indicators present in both the text and the WSI metadata. Of note, as diagnostic reporting in pathology often exhibits a common style across pathologists, many free text descriptions for different slides will be the same when the findings are essentially the same.

[0082] Due to the part, block, slide hierarchy and variability in accessioning, there are three high-level categories of association between slides and part-level text. The part-level text in the diagnostic report corresponds to one of three categories: (1) a single slide from a single block, (2) multiple slides from a single block, or (3) one or more slides per block across multiple blocks. The probability that some of the information in the part-level text does not actually apply to a given slide (because that particular slide is not representative of the final diagnostic finding) increases from category 1 to category 3. Within the framework of these categories, it is additionally possible that the pathologist reviewed more slides than those that were archived and subsequently digitized, so even category 1 has some possibility of the text not corresponding specifically to the image. While we believe this possibility to be a rare occurrence in DS1, complete accessioning metadata from each case was not available, so formal quantification of “missing slides" could not be performed. In light of these considerations, we separated the DS1 dataset into a “clean” set, consisting of category 1 and select category 2 slides where these ambiguities are mitigated (DS1-CLEAN), and a “noisy” set consisting of all categories (DS1-NOISY). We describe these subsets in detail in the “Data splits” section below.

[0083] For TCGA, although pathology reports are available (as PDFs), they are submitted from a variety of source sites with significant variability in structure and detail. Additionally, they do not specify which portion of the report corresponds to the available images. Instead of parsing these heterogeneous reports, we utilized structured case-level metadata available for TCGA to generate a basic description in the “label : finding” format, analogous to the typical structure of part-level text in DS1. Specifically, we used the tissue type, histological type, and histologic grade (when available) to generate captions such as “bladder, resection : histological type: invasive urothelial carcinoma; tumor grade: high grade”. We associate each slide with a caption generated from the metadata in this way. For TCGA, the possibility that a given slidein isolation is not representative of all this metadata is at least partially mitigated because typically only one or two diagnostic slides per case were submitted and these slides were selected to contain substantial tumor content and be representative of the case-level diagnosis.

[0084] For DS1, slides were split taking the three different categories of image-text pairings into account. Specifically, category 1 (one WSI per part-level diagnostic text) was combined with one WSI per part from category 2 (multiple WSIs from the same block per part-level diagnostic text) to create DS 1 -CLEAN (n=92,321 WSI-text pairs). Note that since category 2 WSIs from the same part are by definition from the same block, they typically represent slides of similar tissue from serial or subsequent sections. The individual slides from category 2 parts for DS 1 -CLEAN were selected based on rules applied to image metadata (eg. scanning protocol used, stain type when available) to maximize likelihood of high quality H&E images. Slides in DS 1 -CLEAN were split by case into train, validation, and test sets with a 90 / 5 / 5 split. All slides from cases not included in the validation and test splits of DS 1 -CLEAN were combined with remaining category 2 and category 3 WSIs to form DS1-NOISY, a larger dataset used for training only (n=344,532 WSIs).

[0085] For TCGA, formalin-fixed, paraffin-embedded (FFPE) H&E slides across all TCGA study types (n=12,268) were split into train, validation and test sets by tissue source site (TSS) on a per-study basis. This enables better evaluation of performance generalization across variability in tissue and image processing. TSSs were assigned with a target split ratio of 2: 1 : 1 across train, validation and tests splits within each study, though the final ratios varied due to site size variability.

[0086] Figure 5 illustrates PRAWN, which is based on the BLIP-26 vision-language model architecture and training method, where the Q-F ormer architecture includes image and text encoding submodules. In the first stage, the WSI and text encoders are trained to align their representations, using learned query vectors to cross-attend to the WSI data. For input to the WSI encoder, WSIs are represented via sequences of patch-level embeddings produced from a patch-level SSL-trained histopathology foundation model along with their positional coordinates. In the second stage, we discard the text encoder submodule from stage 1, and graft the pretrained WSI-encoder to a frozen generative LLM via an adapter with further fine tuning for text generation. We train one stage 1 model with the image-text contrastive loss only, referring to this variant as PRAWN-R (for retrieval, based on use of this model for cross-modal retrieval tasks). The second variant is trained using the standard two-stage BLIP-2 training procedure along with LLM integration for text generation. We refer to this variant as PRAWN-G (for generation), and use a frozen PaLM 2-S25 model as the LLM.

[0087] We represent each WSI by a set of up to 10,240 tissue-containing patches of size 224x224 at 10X magnification (~1 micron-per-pixel). Tissue masks were generated via a sequence of image processing operations. These consist of transforming RGB images to HSV- space, performing morphological operations to group together connected regions, thresholding on pixel intensity, and post-processing with an erosion operation to remove remaining noise. Using these tissue masks, we identify all tissue containing patches of size 224x224 pixels, with a stride of 192 pixels (32 pixels of overlap). We use at most 10,240 patches per WSI (sampling without replacement when the total number of tissue-patches exceeds this count), which covers all patches for 97.8% percent of DS1 WSIs, and 91.4% of TCGA WSIs.

[0088] We pretrained a pathology-specific patch-level encoder via self-supervised learning using the train split of DS1, using Masked Siamese Networks as the SSL method along with RandStainNA color augmentation. This patch encoder uses a ViT-S23 architecture and maps 224x224 pixel patches into embeddings of size 384.

[0089] The WSLencoder is based on the BLIP-2 training method utilizing the Q-former with an image-encoder submodule. The input to the WSLencoder is the sequence of up to 10,240 frozen patch-embeddings from the patch-encoder, with non-learnable sine and cosine position encodings for the patch coordinates incorporated for both x and y axes. The learned query vectors in the Q-F ormer are used to cross-attend to the input WSI data.

[0090] We use the two stage BLIP-2 methodology in order to produce language aligned WSL level embeddings (stage 1) and to align WSLlevel visual soft tokens to a frozen LLM (stage 2), largely following the design choices outlined in BLIP-2.

[0091] To avoid false negatives from similar diagnostic findings within training batches in the image-text contrastive (ITC) loss and image-text matching (ITM) loss, we mask out negative pairs ((image_i, text J) with i!=j) with high text similarity between text_i and text J using text embeddings from the Universal Sentence Encoder model (cosine similarity > 0.985, which typically requires very high similarity with only slight differences in syntax or word choice). To further reduce the impact of false-negatives, we did not use hard-negative mining for the ITM loss.

[0092] For PRAWN-R, we chose not to use ITM reranking during retrieval for practical considerations, with efficiency in terms of potential model-serving and typical client-end API use in mind. When using the cosine similarity between contrastive embeddings for ranking, we found that training with the ITC loss alone and using a single learnable query in the Q-formerworked better for retrieval. For text generation (PRAWN-G), we found that the standard BLIP- 2 approach of training a stage 1 model including all losses worked best, along with 32 learnable queries.

[0093] Two U.S. board certified pathologists evaluated PRAWN-R on image-text retrieval and PRAWN-G on image-conditioned text generation (automatic evaluation was also performed during model development, see Appendix A). PRAWN-R was also evaluated on four text-based image-classification tasks using area under the receiver operating curve (AUC) and balanced accuracy metrics.

[0094] A subset of 120 test set WSIs from the DS1 test set was selected for evaluation by pathologists. For subset selection the test set was first divided into two categories: (1) the most common specimen types (colon, rectum, cervix, and skin biopsies) and (2) other specimen types. Then, 60 images from each of these two categories were randomly sampled. For each WSI, pathologists were presented with the image in a web-based digital pathology viewer along with five retrieved diagnostic texts (from PRAWN-R), a generated text (from PRAWN-G), and the original diagnostic text. Texts were provided in random order and pathologists were blinded to the source of each (ie. retrieved, generated, original) to avoid any bias in interpretation.

[0095] Text examples were rated based on concurrent review of the corresponding WSI. Ratings were provided on a five-point scale (1 : Completely inaccurate, 2: Partially accurate; 3 : Mostly accurate but clinically significant error or omission; 4: Mostly accurate without clinically significant error or omission, and 5: Highly accurate). At evaluation, 5 images were dropped due to pathologist review indicating poor image quality (n=l) or need for immunohistochemistry (IHC) (n=4) for confident interpretation.

[0096] For image-to-text retrieval, PRAWN-R takes a WSI and corpus of texts as input, and scores the texts according to embedding similarity with that of the input image. In our case, the text corpus consists of all unique diagnostic texts from WSI-text pairs in the DS1 test set (n=3,176 unique diagnostic texts). The scoring was done using the cosine-similarity between the model’s embedding for the input WSI and the model’s embeddings for each diagnostic text in the corpus. The top-5 retrieved diagnostic texts for each WSI in the pathologist evaluation set were rated as described above. We evaluated the retrieved results by computing the rates at which the best scoring retrieved text received a rating of at least 4 (i.e. mostly accurate without clinically significant error or omission). This is analogous to top-k accuracy. To address the fact that retrieval results could be influenced by the frequency of similar cases in the corpus, we limited retrieval to unique diagnostic texts (ie. duplicates removed before retrieval), and wealso performed subanalysis on the “common” and “less common” specimen types defined in the “Pathologist Evaluation” section above.

[0097] Text generated from PRAWN-G based on the input WSI was also rated by pathologists as described above. The generated text was evaluated by comparing ratings for generated text to those for the original diagnostic text.

[0098] To perform classification, PRAWN-R is given a WSI and a class is assigned based on similarity of the image embeddings to the text embeddings for classes of interest, using a curated set of texts per class. This can be thought of as a highly constrained version of image- to-text retrieval, except that instead of scoring the similarity between the model’s WSI embedding and embeddings for the corpus of texts, the scoring is done using the average similarity between the WSI embeddings and the set of texts associated with each class. This approach is often referred to as zero-shot classification in the literature, but since the concepts in these tasks are contained in the training data, we refer to this simply as text-based imageclassification here. We evaluate four separate classification tasks including:- NSCLC subtyping: non-small cell lung cancer subtyping using the “LU AD” and “LUSC” studies in TCGA.- RCC subtyping: renal cell carcinoma subtyping using the “KIRC”, “KIRP” and “KICH” studies in TCGA.- BRCA subtyping: breast cancer subtyping of “ductal” versus “lobular” carcinoma using the BRCA study and associated metadata for subtype in TCGA.- Procedure classification: “biopsy” vs. “resection” classification using a subset of DSl.

[0099] For the three subtyping tasks, texts for each class were curated to represent common diagnostic texts for WSIs of each subtype in the training set (identified via regular expression matching). For the procedure classification task, the biopsy class is represented by just the single word “biopsy”, while the resection class is represented by “resection” as well as a variety of tissue specific resection types (e.g. “lobectomy”, “mastectomy”, “nephrectomy”, etc). WSIs for the procedure classification task were selected by randomly sampling 250 WSIs with a diagnostic text containing the word “biopsy” and selecting all WSIs containing any of the resection texts (n=63) from the DS1 test set. We evaluate classification performance with both macro-averaged AUC (using average similarity directly) as well as balanced accuracy (taking the max similarity score across classes). Confidence-intervals for classification tasks werecomputed via bootstrapped resampling with replacement over 1000 replicates.

[0100] Image-to-text retrieval for DS1 images was evaluated by two U.S. board- certified pathologists. Top-1, top-3, and top-5 retrieval accuracy were 73.5%, 91.3%, and 92.6%, respectively, on average across the two raters (based on an evaluation score of 4 or 5). For reference, the original diagnostic text had an “accuracy” (rated 4 or 5) of 86.5%, further illustrating the relative accuracy of retrieved texts.

[0101] Subanalysis by the “common” and “less common” specimen-type categories showed similar performance, without evidence for significant bias favoring retrieval of common cases. Automatic evaluation of image-text retrieval (as well as text-image and imageimage retrieval) was also performed, using a text-based similarity score threshold to determine accurate retrievals.

[0102] Generated diagnostic text for DS1 images were also evaluated by pathologist scoring. For images where either original text or Al generated text was rated 4 or above, the Al generated text was determined to be equivalent or better than the original text in 75% of cases on average. For all WSIs (n=115), generated text was rated to be 4 or 5 (ie. mostly or highly accurate without clinically significant error or omission) for 78% of images, on average across the two pathologist evaluations. For images considered to be normal, (ie. no pathologic findings based on the original diagnostic report), generated text was rated 4 or 5 in 92% of images on average. For images with mild diagnostic findings, generated text was rated 4 or 5 in 79% of images on average. For images with significant diagnostic findings, generated text was rated 4 or 5 in 61% of images on average.

[0103] To understand the ability perform classification using the embedding similarity between images and texts, we evaluate PRAWN-R on three cancer subtyping tasks using TCGA (lung, breast and kidney cancer subtyping, referred to as LUAD, RCC, and BRCA) and on procedure type classification (i.e. biopsy vs resection) using DS1. AUCs [95% CI] were 0.945 [0.921 - 0.965] for NSCLC, 0.971 [0.954 - 0.985] for RCC, 0.879 [0.823 - 0.926] for BRCA. AUC was 0.987 [0.976 - 0.996] for procedure classification. Balanced accuracy scores were 0.875 [0.838 - 0.909], 0.889 [0.832 - 0.941] and 0.775 [0.706 - 0.836] for NSCLC, RCC, and BRCA subtyping, respectively, and 0.942 [0.912 - 0.978] for procedure classification.

[0104] While our predominant focus in this work has been on applications involving image-text alignment and text-generation for WSIs and diagnostic text, the use of a pre-trained and frozen LLM suggests further interesting applications. To showcase potential uses for such abilities, we provide and evaluate a case prioritization example.

[0105] Prioritizing cases for review: We randomly select a set of 200 colon biopsies representing a theoretical case load and use PRAWN to return the image list sorted by likely “severity”of the findings. For the simple implementation here, we prompt the LLM with each image, the generated text, and the following prompt: “Question: On a scale of 1 to 3, where 1 is benign or low-risk, 2 are pre-cancerous polyps and adenomas, and 3 is cancerous or highly suspicious for cancer, can you rate the pathological findings for this image? Answer:” We then use the output score from the LLM to sort the images. The results demonstrate the promising potential to organize or group cases with flexible natural language queries.

[0106] Many important applications in histopathology involve interpretation of WSIs. Leveraging advances in efficient vision-language pretraining and self-supervised patch-level encoders, we develop a pathology report aligned WSLencoder using a real -world dataset of over 350,000 gigapixel WSIs with diagnostic text from associated pathology reports. We evaluate this model for classification, cross-modal retrieval, and generation of text describing pathologic findings.

[0107] Our work complements recent efforts on language aligned WSLencoders such as PathM3, PRISM and GigaPath. Compared to prior work, we explore an alternative method for efficient image-text alignment with WSIs based on the BLIP-2 approach. This enables us to align our WSLencoder with a powerful pre-trained LLM (PaLM 2-S) for generating text from WSIs. While evaluation of the other recent models has been limited to classification tasks, automated scoring, and qualitative review of text generation, we report the first quantitative pathologist evaluation of cross-modal retrieval and text generation.

[0108] The text generation evaluations provide several interesting insights. On one hand, they reflect the impressive capability of the domain-specific WSLencoder to align with a pretrained LLM even when images are gigapixel-sized. The generated texts are generally quite accurate at reflecting important information about the WSI, often showcasing important slide-level capabilities by providing information that requires aggregating information and context from multiple patches. Examples of this include the type of procedure or biopsy, and perhaps more impressively, the concept of low grade versus high grade cervical dysplasia, which is defined in part by the extent of the epithelial thickness that is affected, and thus likely requires contextual information within the slide. On the other hand, there are clearly still some shortcomings in the details provided by the generated text, such as specific grades for prostate and breast cancer. We also observe some confabulations, particularly in the specimen type when specimen information cannot be readily inferred from the image (eg. “neck contents, lymphnode”). This is likely due, in part, to imperfect removal of this type of specimen information when we processed part labels, but also reflects the inherent importance of context when reviewing slides and writing reports. While prompting the model with the specimen information along with the image is one strategy that might reflect realworld availability of the part label during slide review, we did not find this to significantly improve text generation in our study. Efforts to more effectively clean training data and to more thoroughly evaluate confabulations and optimize prompting strategies using available metadata are opportunities for future work.

[0109] While PRAWN-R performs well on the cancer subtyping tasks, direct comparison to other image-text pathology models (e.g. CONCH, GigaPath) cannot be made directly due to different image splits, as well as our inclusion of TCGA data in training (albeit from different TSS than those used for testing).

[0110] While our training datasets are comparable in size to prior work on WSI-level image-text alignment, they are relatively small compared to datasets that have been used for image-text alignment from natural images (>400M). In initial experiments, we observed that training on the full set of training data (DSl-Noisy) provided significantly better performance than training on only the cleaner, but smaller train split of DS 1 -Clean, further supporting the potential for data scaling. Because pathology reports are in principle available for all WSIs that have gone through clinical workflows, we hope to see future work build on our approach with larger-scale datasets of WSIs paired with pathology reports.

[0111] This work has several limitations. A primary challenge in aligning pathology WSIs with diagnostic text is the “many-to-one” nature of slides to the associated portion of the diagnostic report. In this work, we curated our dataset in a manner that minimized this issue for the validation and test splits, but this also resulted in a relative enrichment in these splits for the types of cases that typically only have one slide per submitted pathology “part”, such as colon, skin, and cervical biopsies along with other small specimens. Additionally, there are at least some instances for which there are “missing” slides, such that the single slide used for inference may not contain all the information represented in the paired text. Curation of datasets to include only the representative slides for the reported findings and modeling at the level of multiple slides remain as opportunities to further address this issue.

[0112] The use of TCGA, while useful for enriching DS1 with cancer cases and providing increased training diversity, also introduces specific limitations. For example, we used available structured metadata to generate “synthetic” diagnostic captions for these images.While an effort was made to diversify the language used for describing any given entity, these captions still do not necessarily represent the reporting style for these types of cancer cases in real clinical reports. For example, an actual diagnostic report for a cancer resection might describe many aspects of the tumor grading, staging, and subtyping in a manner that is more extensive than the available structured metadata from TCGA.

[0113] Due to lack of available out-of-distribution datasets for this study, image-to-text retrieval and text generation tasks were only evaluated on in-distribution data for DS1. Experiments to evaluate generalization of this model to diverse data sources are warranted. Analysis on additional tasks such as text-to-image retrieval could also be performed and existing evaluations could be improved by increasing the evaluation set size, including greater diversity of cases and findings, and including a larger number of raters.

[0114] Future work could explore additional vision-language modeling strategies coupled with different LLMs and further instruction tuning. While we benefited from the lower computational costs involved in cross-attention by using relatively fewer number of Q-former queries to the long patch-sequence, direct patch-to-patch interaction is potentially not fully captured in the cross-attention mechanism and might be improved further such as through efficient self-attention. Modeling at the level of multiple slides across entire parts or cases, higher magnifications, or a pyramid of multiple magnifications could also further enable useful applications.

[0115] As described herein, we demonstrate the novel development of multimodal pathology models using WSIs paired with curated portions of original diagnostic reports along with a pre-trained patch encoder and a LLM. These initial results highlight the potential for WSI-text alignment in a manner that can incorporate the reasoning capabilities of large multimodal models.IV. Example Methods

[0116] Figure 6 illustrates a flow chart of operations, which may be carried out by a computing system or a computing device. The embodiments of Figure 6 may be simplified by the removal of any one or more of the features shown therein. Further, these embodiments may be combined with features, aspects, and / or implementations of any of the previous figures or otherwise described herein.

[0117] Block 602 involves receiving training data comprising medical images and associated text reports.

[0118] Block 604 involves training an image encoder to associate image embeddingsgenerated by the image encoder when the image encoder is applied to the medical images with text embeddings generated by a text encoder when the text encoder is applied to the associated text reports.

[0119] Block 606 involves training a vision-language model based on the image embeddings generated by the trained image encoder to represent the medical images and the associated text reports in a language-aligned embeddings space.

[0120] Block 608 involves training an adaptor which connects the vision-language model to a large language model (LLM) to generate LLM-aligned tokens which emphasize portions of medical image information to the LLM via an attention mechanism, where the LLM is configured to generate an output based on the LLM-aligned tokens.

[0121] Block 610 involves providing the trained image encoder, the trained visionlanguage model, and the trained adaptor to perform one or more inference tasks.

[0122] In some examples, the image encoder is trained using a contrastive loss function.

[0123] In some examples, the vision-language model is trained by jointly employing three different tasks: a) image-text contrastive learning (ITC), b) image-grounded text generation (ITG), and c) image-text matching (ITM).

[0124] In some examples, the adaptor is trained based on language modeling loss based on output image query tokens from the vision-language model and associated text.

[0125] In some examples, the image encoder is frozen while training the visionlanguage model and the adaptor.

[0126] In some examples, the LLM is frozen while training the vision-language model and the adaptor.

[0127] In some examples, the medical images comprise chest X-rays and wherein the associated text reports comprise radiology reports.

[0128] In some examples, the medical images comprise histopathology whole-slide images, wherein each of the histopathology whole-slide images comprises a plurality of smaller image patches. Such examples, may involve training a patch encoder to provide patch embeddings for each of the plurality of smaller image patches of each of the histopathology whole-slide images, where the image encoder is trained based on a sequence of patch embeddings from the patch encoder and associated positions. In such examples, the patch encoder may be frozen during training of the image encoder.

[0129] Some examples may involve one or more non-transitory computer readablemedia comprising program instructions executable by one or more processors to perform the functions of Figure 6.

[0130] Further examples may involve a system comprising one or more processors and one or more non-transitory computer readable media comprising program instructions executable by the one or more processors to perform the functions of Figure 6.

[0131] Figure 7 illustrates another flow chart of operations, which may be carried out by a computing system or a computing device. The embodiments of Figure 7 may be simplified by the removal of any one or more of the features shown therein. Further, these embodiments may be combined with features, aspects, and / or implementations of any of the previous figures or otherwise described herein.

[0132] Block 702 involves receiving a medical image.

[0133] Block 704 involves applying an image encoder to the medical image to generate image embeddings.

[0134] Block 706 involves applying a vision-language model to the image embeddings generated by the image encoder to generate a representation of the medical image in a language-aligned embeddings space.

[0135] Block 708 involves performing one or more inference tasks based on the representation of the medical image in the language-aligned embedding space.

[0136] In some examples, the medical image comprises a chest X-ray, and wherein the one or more inference tasks comprise generating an impressions section of a radiology report.

[0137] In some examples, the one or more inference tasks comprise zero-shot classification, wherein the one or more prompts comprise one or more positive or negative prompts, and wherein an output of the zero-shot classification is at least one score based on the one or more positive or negative prompts.

[0138] In some examples, the one or more inference tasks comprise data-efficient classification to generate a classification score based on application of a nonlinear classifier.

[0139] In some examples, the one or more inference tasks comprise semantic search based on a text description of a search term, and wherein an output of the semantic search is based on similarity between a query text embedding and image embeddings from a data pool.

[0140] Some examples further involve applying an adaptor to the representation of the medical image in the shared embeddings space to generate large language model (LLM)- aligned tokens which emphasize portions of the medical image to an LLM via an attention mechanism, wherein the one or more inference tasks are performed based on application of theLLM to the LLM-aligned tokens. In such examples, one or more prompts may be input into the LLM. In such examples, the one or more inference tasks comprise visual question answering (VQA), wherein an output of the VQA comprises answers given to questions based on the medical image.

[0141] In some examples, the one or more inference tasks comprise report quality assurance (QA), wherein the one or more prompts comprise a medical report, and wherein an output of the report QA comprises a description of the completeness and / or correctness of the medical report given the medical image, wherein the output of the report QA optionally further comprises rich text explanation of the decision.

[0142] In some examples, the medical image comprises a histopathology whole-slide image, where the histopathology whole-slide image comprises a plurality of smaller image patches. Such examples may involve applying a patch encoder to the plurality of smaller image patches of the histopathology whole-slide image to determine a plurality of patch embeddings; and determining a sequence of the plurality of patch embeddings and associated positions for the histopathology whole-slide image, wherein the image encoder is applied to the sequence of patch embeddings and associated positions for the histopathology whole-slide image.

[0143] Some examples may involve one or more non-transitory computer readable media comprising program instructions executable by one or more processors to perform the functions of Figure 7.

[0144] Further examples may involve a system comprising one or more processors and one or more non-transitory computer readable media comprising program instructions executable by the one or more processors to perform the functions of Figure 7.V. Conclusion

[0145] The particular arrangements shown in the figures, tables, and charts of the Appendix should not be viewed as limiting. It should be understood that other embodiments may include more or less of each element shown. Further, some of the illustrated elements may be combined or omitted. Yet further, an exemplary embodiment may include elements that are not illustrated in the figures, tables, and charts.

[0146] Additionally, while various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims. Other embodiments may be utilized, and other changes may be made, without departing from thespirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.

Claims

CLAIMSWhat is claimed is:

1. A method comprising: receiving training data comprising medical images and associated text reports; training an image encoder to associate image embeddings generated by the image encoder when the image encoder is applied to the medical images with text embeddings generated by a text encoder when the text encoder is applied to the associated text reports; training a vision-language model based on the image embeddings generated by the trained image encoder to represent the medical images and the associated text reports in a language-aligned embeddings space; training an adaptor which connects the vision-language model to a large language model (LLM) to generate LLM-aligned tokens which emphasize portions of medical image information to the LLM via an attention mechanism, wherein the LLM is configured to generate an output based on the LLM-aligned tokens; and providing the trained image encoder, the trained vision-language model, and the trained adaptor to perform one or more inference tasks.

2. The method of claim 1, wherein the image encoder is trained using a contrastive loss function.

3. The method of claim 1, wherein the vision-language model is trained by jointly employing three different tasks: a) image-text contrastive learning (ITC), b) image-grounded text generation (ITG), and c) image-text matching (ITM).

4. The method of claim 1, wherein the adaptor is trained based on language modeling loss based on output image query tokens from the vision-language model and associated text.

5. The method of claim 1, wherein the image encoder is frozen while training the visionlanguage model and the adaptor.

6. The method of claim 1, wherein the LLM is frozen while training the vision-language model and the adaptor.

7. The method of claim 1, wherein the medical images comprise chest X-rays and wherein the associated text reports comprise radiology reports.

8. The method of claim 1, wherein the medical images comprise histopathology wholeslide images, wherein each of the histopathology whole-slide images comprises a plurality of smaller image patches.

9. The method of claim 8, further comprising: training a patch encoder to provide patch embeddings for each of the plurality of smaller image patches of each of the histopathology whole-slide images, wherein the image encoder is trained based on a sequence of patch embeddings from the patch encoder and associated positions.

10. The method of claim 9, wherein the patch encoder is frozen during training of the image encoder.

11. One or more non-transitory computer readable media comprising program instructions executable by one or more processors to perform functions comprising: receiving training data comprising medical images and associated text reports; training an image encoder to associate image embeddings generated by the image encoder when the image encoder is applied to the medical images with text embeddings generated by a text encoder when the text encoder is applied to the associated text reports; training a vision-language model based on the image embeddings generated by the trained image encoder to represent the medical images and the associated text reports in a language-aligned embeddings space; training an adaptor which connects the vision-language model to a large language model (LLM) to generate LLM-aligned tokens which emphasize portions of medical image information to the LLM via an attention mechanism, wherein the LLM is configured to generate text based on the LLM-aligned tokens; and providing the trained image encoder, the trained vision-language model, and the trained adaptor to perform one or more inference tasks.

12. A system comprising one or more processors and one or more non-transitory computer readable media comprising program instructions executable by the one or more processors to perform functions comprising: receiving training data comprising medical images and associated text reports; training an image encoder to associate image embeddings generated by the image encoder when the image encoder is applied to the medical images with text embeddings generated by a text encoder when the text encoder is applied to the associated text reports;training a vision-language model based on the image embeddings generated by the trained image encoder to represent the medical images and the associated text reports in a language-aligned embeddings space; training an adaptor which connects the vision-language model to a large language model (LLM) to generate LLM-aligned tokens which emphasize portions of medical image information to the LLM via an attention mechanism, wherein the LLM is configured to generate text based on the LLM-aligned tokens; and providing the trained image encoder, the trained vision-language model, and the trained adaptor to perform one or more inference tasks.

13. A method comprising: receiving a medical image; applying an image encoder to the medical image to generate image embeddings; applying a vision-language model to the image embeddings generated by the image encoder to generate a representation of the medical image in a language-aligned embeddings space; and performing one or more inference tasks based on the representation of the medical image in the language-aligned embedding space.14 The method of claim 13, wherein the medical image comprises a chest X-ray, and wherein the one or more inference tasks comprise generating an impressions section of a radiology report.

15. The method of claim 13, wherein the one or more inference tasks comprise zero-shot classification, wherein the one or more prompts comprise one or more positive or negativeprompts, and wherein an output of the zero-shot classification is at least one score based on the one or more positive or negative prompts.

16. The method of claim 13, wherein the one or more inference tasks comprise data- efficient classification to generate a classification score based on application of a nonlinear classifier.

17. The method of claim 13, wherein the one or more inference tasks comprise semantic search based on a text description of a search term, and wherein an output of the semantic search is based on similarity between a query text embedding and image embeddings from a data pool.

18. The method of claim 13, further comprising: applying an adaptor to the representation of the medical image in the shared embeddings space to generate large language model (LLM)-aligned tokens which emphasize portions of the medical image to an LLM via an attention mechanism; and wherein the one or more inference tasks are performed based on application of the LLM to the LLM-aligned tokens.

19. The method of claim 18, wherein one or more prompts are input into the LLM.

20. The method of claim 18, wherein the one or more inference tasks comprise visual question answering (VQA), wherein an output of the VQA comprises answers given to questions based on the medical image.

21. The method of claim 18, wherein the one or more inference tasks comprise report quality assurance (QA), wherein the one or more prompts comprise a medical report, and wherein an output of the report QA comprises a description of the completeness and / or correctness of the medical report given the medical image, wherein the output of the report QA optionally further comprises rich text explanation of the decision.

22. The method of claim 13, wherein the medical image comprises a histopathology whole-slide image, wherein the histopathology whole-slide image comprises a plurality of smaller image patches.

23. The method of claim 22, further comprising: applying a patch encoder to the plurality of smaller image patches of the histopathology whole-slide image to determine a plurality of patch embeddings; and determining a sequence of the plurality of patch embeddings and associated positions for the histopathology whole-slide image, wherein the image encoder is applied to the sequence of patch embeddings and associated positions for the histopathology whole-slide image.

24. One or more non-transitory computer readable media comprising program instructions executable by one or more processors to perform functions comprising: receiving a medical image; applying an image encoder to the medical image to generate image embeddings; applying a vision-language model to the image embeddings generated by the image encoder to generate a representation of the medical image in a language-aligned embeddings space; andperforming one or more inference tasks based on the representation of the medical image in the language-aligned embedding space.

25. A system comprising one or more processors and one or more non-transitory computer readable media comprising program instructions executable by the one or more processors to perform functions comprising: receiving a medical image; applying an image encoder to the medical image to generate image embeddings; applying a vision-language model to the image embeddings generated by the image encoder to generate a representation of the medical image in a language-aligned embeddings space; and performing one or more inference tasks based on the representation of the medical image in the language-aligned embedding space.

Images