AI-driven natural language assistant for pathology

Abstract

Systems and methods are provided for providing natural language decision support for pathology. A low-dimensional representation is generated for each image in a received set of pathology images, and a first set of tokens is generated from the representations of the set of pathology images by projecting the low-dimensional representations of the received pathology images to the same dimensionality as the embedding space of a large language model for text tokens, or by a multi-modal module (e.g., cross-attention) added to the large language model. The large language model is trained on an instruction dataset compiled from a plurality of pathology-related sources. The large language model receives a second set of tokens associated with a natural language prompt. A response is determined by the large language model from the first set of tokens and the second set of tokens.

Description

[0001] Related applications This application claims priority to U.S. Provisional Application No. 63 / 608,671, filed December 11, 2023. The present invention relates to clinical decision support systems, and more particularly, to a system for providing analysis and interpretation of pathological images. Technical Field

[0002] This invention relates to clinical decision support systems, and more specifically, to a system for providing analysis and interpretation of pathological images. Background Technology

[0003] In recent years, the field of computational pathology has witnessed significant transformations, primarily due to the convergence of several key trends, including increased availability and institutional adoption of digital slide scanning, rapid advancements in artificial intelligence (AI) research, increasing accessibility to large datasets, and abundant high-performance computing resources. Researchers are leveraging deep learning to varying degrees to address a wide range of tasks, including cancer subtyping and grading, metastasis detection, survival and treatment response prediction, primary tumor site prediction, image search, mutation prediction, and biomarker screening. Meanwhile, general-purpose visual encoder models, trained on large datasets of unlabeled histopathological images and serving as versatile task-agnostic model backbones, are paving the way for further improvements in many tasks within computational pathology, both in terms of performance and labeling efficiency. Summary of the Invention

[0004] According to one example, a system provides natural language decision support for pathology. The system includes a processor and a non-transitory computer-readable medium storing instructions executable by the processor. The machine-executable instructions include an encoder that receives pathology images and generates a representation of those images. Additional encoders can be used to encode other patient-related data, such as genomics, electronic medical records, etc. A multimodal projector generates a first set of lexical units based on the representations of one or more pathology images. A large language model is trained on a dataset of instructions compiled from multiple pathology-related sources, where a given training sample includes a set of one or more pathology images and text describing or answering a specific query related to those images. The large language model receives a second set of lexical units associated with prompts and determines a response based on the first and second sets of lexical units.

[0005] According to another example, one approach provides natural language decision support for pathology. A low-dimensional representation of each received pathology image is generated, and a first set of lexical units is generated based on this representation by projecting the low-dimensional representation of the received pathology images to the same dimension as the embedding space of a large language model used for text lexical units. The large language model is trained on a dataset of instructions compiled from multiple pathology-related sources. A second set of lexical units is received by the large language model and associated with natural language prompts. The large language model then determines a response based on the first and second sets of lexical units.

[0006] According to another example, a system provides natural language decision support for pathology. The system includes a processor and a non-transitory computer-readable medium storing instructions executable by the processor. The machine-executable instructions include a visual encoder that receives a set of pathology images and generates a representation of each pathology image. A multimodal projector generates a first set of lexical units based on the representations of the pathology images. A user interface receives prompts from the user in the form of natural language text, and a word segmenter generates a second set of lexical units based on the prompts. A large language model is trained on a dataset of instructions compiled from multiple pathology-related sources. Each training sample within the instruction dataset includes one or more pathology images and text that describes or answers a specific query related to those images. The large language model receives the second set of lexical units and determines a response based on the first and second sets of lexical units. The user interface displays the response to the user on an associated display. Attached Figure Description

[0007] Figure 1 An example of a visual language artificial intelligence (AI) assisted system based on a multimodal large language model (MLLM) for human pathological analysis is shown. Figure 2 Another example of a MLLM-based visual-language AI-assisted system for human pathological analysis is shown; Figure 3 An example of a method for training a large multimodal language model is shown; Figure 4 An example of a method for providing natural language decision support for pathology is shown; and Figure 5 This is a schematic block diagram of an exemplary system showing hardware components capable of implementing the systems and methods disclosed herein.

[0008] definition As used herein, “pathology images” refer to images of human tissue used for disease diagnosis. Non-exclusive examples of pathology images include hematoxylin and eosin stain images, other images acquired using histological stains, immunohistochemistry images, electron microscopy images, cytology images, multiplex images, gross pathology images, radiology images, and any other supplementary images.

[0009] “Pathology-related sources” refer to blocks of text describing pathology findings and can include educational articles, image captioning text, pathology case reports, and regions of interest extracted from whole slide imaging analysis. DETAILED DESCRIPTION

[0010] While progress has been made in general-purpose automated image analysis and machine learning, the development of computational pathology has yet to reflect the important role of natural language in pathology, as a key to unlocking a rich variety of accumulated sources of human medical knowledge, as a potential signal for model supervision, and as a unifying medium to facilitate intuitive interaction between powerful artificial intelligence (AI) models and end users. Notably, in general-purpose machine learning, representative work has shown that large-scale visual-linguistic representation learning can add new capabilities to purely visual AI models, including zero-shot image recognition and text-to-image retrieval. Depending on the architecture design as well as the training data and objectives, visual-linguistic pre-training systems can also be fine-tuned for custom tasks, ranging from visual question answering and image captioning to object detection and semantic segmentation. In computational pathology, a few works have demonstrated promising zero-shot performance in selected diagnostic and retrieval tasks, while other works have also attempted to design specialized models for biomedical visual question answering or image captioning. However, these models are not yet sufficient to serve as interactive assistants for pathologists, researchers using pathology image data, or pathology trainees.

[0011] The systems and methods described herein provide a visual-linguistic interactive AI assistant powered by a customized, fine-tuned multimodal large language model (MLLM) for human pathology. The MLLM-based visual-linguistic AI assistant is capable of reasoning on both visual and natural language inputs. Specifically, in contrast to purely textual large language models (LLMs), multimodal large language models (MLLMs) are trained to understand and respond to user instructions in the form of natural language queries that can additionally incorporate inputs from other modalities (e.g., images). The support for multimodality is particularly valuable in the field of histopathology, as the examination and interpretation of visual information in high-resolution microscopic images (in conjunction with other clinical information) remains a cornerstone of the discipline and extends to many aspects of disease diagnosis and management in modern medicine.

[0012] Figure 1 An example of an MLLM-based visual-language AI-assisted system 100 for human pathological analysis is shown. System 100 includes a processor 102 and a non-transitory computer-readable medium 110 storing executable instructions that can be executed by the processor 102. It should be understood that the executable instructions can be distributed across multiple non-transitory computer-readable media, operatively linked by appropriate data connections, such that these executable instructions can be executed by multiple processors. Specifically, system 100 can be trained on multiple graphics processing units (GPUs), and the number of GPUs used for a given application and the amount of storage on the non-transitory computer-readable medium 110 can be scaled according to the application and the amount of available training data.

[0013] Executable instructions stored on the non-transitory computer-readable medium 110 include a visual encoder 112 that converts a received image into a low-dimensional representation of the image as a set of visual tokens. In one example, the visual encoder 112 may be implemented as an artificial neural network, such as a convolutional neural network or an autoencoder, trained on a training image corpus. Alternatively, one or more feature extraction algorithms may be used to reduce the image to a smaller set of numbers, such as ordered gradient histograms, scale-invariant feature transforms, local binary pattern extraction, frequency-based feature extraction, and similar algorithms. In one example, the visual encoder 112 may include a transformer encoder trained on a histological image corpus that operates directly on patches of the image to assign one or more classification parameters to the image. In one implementation, the visual encoder 112 may use an adapted version of the ViT-Large (ViT-L) architecture with twenty-four transformer multi-head attention blocks, each with sixteen attention heads, an embedding dimension of 1024, and a feedforward hidden dimension of 4096. In this implementation, the lexical size is 16x16, and the learned absolute position encoding is added to each lexical. An example of a suitable visual encoder 112 can be found in "A general foundational model for computational pathology" by Chen et al., published in Nature Medicine in March 2024.

[0014] A multimodal projection model 114 connects the output of the visual encoder 112 to the large language model 116 by projecting visual lexical units onto the same dimension as the embedding space of the large language model used for text lexical units. For example, the multimodal projection model 114 can be implemented as a linear projector or a visual abstractor, such as a Resampler or Q-Former. In one example, the multimodal projector model 114 includes an attention pooling layer followed by two layers of multilayer perceptrons. The attention pooling layer uses 128 learned latent queries and groups of multi-head cross-attention to reduce the final feature map of the visual encoder 112 to a fixed-length sequence of image lexical units to improve training and inference efficiency and prevent the total sequence length of lexical units from potentially exceeding the context window size of the large language model 116. In this example, the subsequent multilayer perceptron includes a single hidden layer and Gaussian Error Linear Unit activations that project the image lexical units onto the embedding dimension of the large language model. An example of a suitable multimodal projection model can be found in Lu et al., “Visual-Language-Based Models for Computational Pathology”, published in Nature Medicine in March 2024.

[0015] Large language model 116 receives a set of text tuples representing a cue and lexical units from multimodal projection model 114, and predicts an appropriate response, which is decoded into natural language, for example, in a tokenizer. Specifically, large language model 116 receives a set of tuples associated with the cue and lexical units from multimodal projection model 114, and determines the response based on these two sets of tuples. The resulting text can be provided to a user via a user interface (not shown) or stored on a non-transitory computer-readable medium, such as as part of a record in an electronic health record database. In one implementation, large language model 116 is implemented as a decoder-only transformer-based autoregressive language model with forty transformer layers, each with forty attention heads, an embedding dimension of 5120, a hidden dimension of 13824, and rotational position encoding, natively supporting a maximum context length of 4096. The cue can be provided by the user via a user interface or automatically generated, for example, for performing batch processing of stored images or real-time analysis of received images. In these cases, standard prompts or word tuples representing standard prompts can be provided to the large language model 116 without user input.

[0016] In one example, a large language model 116 is trained on an instruction dataset compiled from multiple pathology-related sources. Each training sample in the instruction dataset includes at least one pathology image and text describing or answering a specific query related to one or more images. To ensure that the large language model 116 can adapt to a variety of different instructions, the instruction data can include a variety of instruction formats, including open-ended multi-turn dialogues, detailed image descriptions, short-answer questions, multiple-choice questions, object detection and segmentation masks, function calls and tool usage for proxy workflows, and plain text questions. Instructions are derived from a variety of pathology-related sources, including image captions, educational articles, pathology case reports, and regions of interest extracted from internal whole-slice images. In one example, the instruction dataset may include approximately 400,000 instructions. However, it should be understood that the system 100 can be resized to utilize more or less training data, and the model's hyperparameters can be tuned depending on the amount of available data, available hardware, and the application.

[0017] Data filtering is performed separately for each source to ensure the quality and relevance of the visual language aids used to train the pathology-specific tool. For example, overly brief (e.g., less than twelve words) or insufficiently informative and overly general image descriptions (e.g., "H&E image of a tumor") can be omitted. Animal pathology-related descriptions or paragraphs, such as text containing animal-related keywords (e.g., "mouse" or "pig"), and experimental studies (identified by text containing appropriate keywords such as "experimental" or "positive control") can also be omitted. In one implementation, this filtering is performed using a regular expression pattern matching process. Furthermore, instructions can include examples of guardrail instructions, where image-specific instructions are provided to the system to avoid potential failures. For example, when no image is provided, the large language model 116 can be trained to respond to a prompt such as "Describe this histological image of a lung mass" with the response "Sorry, I cannot assist you since you have not uploaded any images." Similarly, when given an image unrelated to pathology, the large language model 116 is trained to output "Sorry, I can only assist you with pathology-related queries."

[0018] In one example, a large language model 116 and a multimodal projection model 114 are jointly trained through a two-stage process. In the first stage, the weights of the large language model 116 are kept frozen, and only the multimodal projection model 114 receives parameter updates to learn the appropriate projection from the image lexical space of the visual encoder 112 to the shared embedding space of the text lexical units used by the large language model. For this simple purpose, the system 100 is supervised to simply predict the caption corresponding to each image using sets of image-caption pairs, without curated instruction data. In one example, the first stage uses approximately 100,000 training samples. In the second stage, both the large language model 116 and the multimodal projection model 114 are trained end-to-end to generate responses to diverse instructions that include both natural language and visual input, as previously described. Specifically, given instruction X... instruct Reference Answer X ans and image X img Each of these is represented as a segmented input sequence, serving as parameters for the autoregressive language model, the multimodal projection model 114, and the large language model 116. and Under these conditions, the training process enables X ans Each word in (according to i=1 to L Likelihood of index) p Maximize, the objective function of which is expressed as:

[0019] The objective function can be easily extended to multi-round instruction data by conditionalizing it with instructions and reference answers from all previous rounds. For instructions without an image, X... img Undefined and removed from the conditional sequence. Similarly, if a given instruction is accompanied by multiple images, their respective image terms are interleaved with text terms according to their original positions in the unprocessed input sequence, and separator terms (such as newline characters) can be inserted between consecutive images. () , and the set of all image lexical units is considered as X img .

[0020] System 100 significantly outperforms existing multimodal large-scale language models in pathology queries. Further details demonstrating this superior performance, along with one implementation of the system, are available in Appendix A of this application. Theoretically, the ability to understand and respond to complex queries in natural language could enable System 100 to act as a helpful partner at various stages of human-involved clinical decision-making, education, and research. For example, in clinical settings, the AI ​​assistant could receive histopathological images, provide an initial assessment of morphological appearance, and identify potential malignant tumor features. Furthermore, prompts could be extended to include additional detection results and other relevant data to System 100, enabling the system to make final inferences to arrive at a diagnosis. In research, System 100 could summarize morphological features from large cohorts of histopathological images, potentially enabling automated quantification and interpretation of morphological markers in large data cohorts, or identifying disease locations in pathology images and generating detection bounding boxes or segmentation masks. In medical education, System 100 could help disseminate expert guidance and training in pathology, thereby bridging regional healthcare service gaps.

[0021] Figure 2 Another example of an MLLM-based visual-language AI-assisted system 200 for human pathological analysis is shown. System 200 includes a processor 202, a display 204, and a non-transitory computer-readable medium 210 storing executable instructions that can be executed by the processor 202. It should be understood that the executable instructions can be distributed across multiple non-transitory computer-readable media, operatively linked by appropriate data connections, such that these executable instructions can be executed by multiple processors.

[0022] Executable instructions include a user interface 212 that allows the user to input natural language prompts and optionally pathological images. For example, the prompt could be a request to evaluate the image and identify potential features (with localization) of malignant tumors or other diseases, or a request for a diagnosis. The user can then provide additional background information about the basic case (e.g., the patient's clinical parameters and tissue sites) in the natural language prompts and request the model to provide differential diagnostic suggestions, as well as suggestions for auxiliary tests and immunohistochemical staining to narrow down the differential diagnosis. Prompts from the user interface 212 are provided to a word segmenter 214 to reduce the text to tokens compatible with the embedding space of the large language model 216. The word segmenter 214 can use any suitable word segmentation technique, including one or more of word-based segmentation, sub-word segmentation, or character-level segmentation. The user interface 212 can be provided on a local terminal, a remote terminal, or a portable device (e.g., a laptop, tablet, or mobile device).

[0023] Any pathological image can be fed into the visual encoder 218, which generates a low-dimensional representation of each pathological image. In the illustrated implementation, the visual encoder 218 is implemented as an adapted version of the ViT-Large (ViT-L) architecture, featuring twenty-four transformer multi-head attention blocks, each with sixteen attention heads, an embedding dimension of 1024, and a feedforward hidden dimension of 4096. The token size in this implementation is sixteen by sixteen, and learned absolute position encodings are added to each token.

[0024] Multimodal projector model 220 connects the output of visual encoder 218 to large language model 216 by projecting visual lexical units to the same dimension as the embedding space for text lexical units in large language model 216. In the illustrated implementation, multimodal projector model 220 includes an attention pooling layer followed by two layers of multilayer perceptron. The attention pooling layer uses 128 learned latent queries and groups of multi-head cross-attention to reduce the final feature map of visual encoder 218 to a fixed-length sequence of image lexical units with an initial dimension of 768, improving training and inference efficiency and preventing the total sequence length of lexical units from potentially exceeding the context window size of the large language model. In this example, the subsequent multilayer perceptron includes a single hidden layer and Gaussian error linear unit activations, projecting image lexical units to the embedding dimension of large language model 216.

[0025] Large language model 216 generates appropriate responses to cues based on lexical representations provided from multimodal projector model 220 and word segmenter 214. This large language model is a transformer-based autoregressive language model, but may include additional multimodal modules (e.g., cross-attention) to enable it to incorporate lexicalized representations from additional modalities (e.g., images), in which case a separate multimodal projector is not required. In one example, large language model 216 is implemented as a decoder-only transformer-based autoregressive language model with forty transformer layers, each with forty attention heads, approximately five thousand embedding dimensions, approximately thirteen thousand hidden dimensions, and rotational positional encoding, natively supporting a maximum context length of 4096. However, it should be understood that these features of model 216 will vary in different implementations. In another example, large language model 216 is trained on a dataset of over 400,000 instructions to respond to pathology-specific queries. This dataset includes multiple instruction formats, including open-ended multi-turn dialogues, detailed image descriptions, short-answer questions, multiple-choice questions, object detection and segmentation, function calls and tool usage for proxy workflows, and plain text questions. The instruction dataset is generated using diverse pathology-related sources, covering image captions, educational articles, pathology case reports, and regions of interest extracted from internal whole-section images. Data filtering is performed separately for each source to ensure the quality and relevance of the visual language aids used to train pathology-specific tools, and various guardrails can be added during training to provide appropriate responses to irrelevant questions and potential error conditions.

[0026] In one example, a large language model 216 and a multimodal projector model 220 are jointly trained through a two-stage process. In the first stage, the weights of the large language model 216 are kept frozen, and only the multimodal projector model 220 receives parameter updates to learn the appropriate projection from the image lexical space of the visual encoder 218 to the shared embedding space of the text lexical units used by the large language model 216. For this simple purpose, the system 200 is supervised to simply predict the caption text corresponding to each image using sets of image-caption pairs, without curated instruction data. In one example, the first stage uses approximately 100,000 training samples.

[0027] In the second phase, using the instruction dataset described earlier, both the large language model 216 and the multimodal projector model 220 are trained end-to-end to generate responses to diverse instructions, including both natural language input and visual input. Additional training phases may also be performed, such as fine-tuning the model for a target task (e.g., classifying pathological images of a specific disease model) and adjusting its output to conform to human preferences via preference tuning to further guide the model's behavior.

[0028] The responses generated at the large language model 216 are provided to the user on the display 204 via the user interface 212. In the illustrated implementation, the user receives a prompt at the user interface 212 to evaluate the response provided by the large language model 216. In one example, the user is prompted to rate the response as helpful or not helpful. If the response is indicated as helpful, the response, the prompt or the tuple representing the prompt, and any pathological image or the tuple representing the pathological image are provided to the feedback component 222. The feedback component 222 constructs training instructions based on the response, the prompt or the tuple representing the prompt, and any pathological image or the tuple representing the pathological image. These training instructions can be used to provide further training to any or all of the visual encoder 218, the multimodal projector 220, and the large language model 216. In another example, the user receives multiple responses at the user interface 212 and is prompted to select from the provided responses. The response selected by the user can be passed to the feedback component 222 to generate new training instructions.

[0029] In another example, a user can tune a large language model 216 without training to perform a specific task using a small number of contextual examples. In the illustrated implementation, the user provides task cues and example data to the large language model 216, visual encoder 218, and multimodal projector 220 at user interface 212. The task cues can be a detailed set of instructions for the large language model 216 to follow when performing a specific task. The example data consists of example input pairs and example output pairs; the example input pairs are expected to be processed by the large language model 216 and visual encoder 218 to complete the user task, and the example output pairs are expected to be generated by the large language model 216 to follow the user task instructions. The example data can be pre-labeled by the user or labeled in-situ by the user in user interface 212. Once the task cues and example data are stored in the model, no further training of the large language model 216, visual encoder 218, and multimodal projector 220 to perform the specific task is required. Instead, the stored task cues and example data are used by the large language model 216 as context to perform the user's specific task without training.

[0030] In view of the structural and functional features described above, the methods according to various aspects of the present invention will be referred to Figure 3 and Figure 4 To gain a better understanding. However, for the purpose of simplifying the explanation, Figure 3 and Figure 4The methods are shown and described as being performed sequentially, but it should be understood and recognized that the invention is not limited to the order shown, as some aspects may occur in different orders and / or simultaneously with other aspects shown and described herein, according to the invention. Furthermore, not all features shown are necessary for implementing a method according to one aspect of the invention.

[0031] Figure 3 An example of method 300 for training a multimodal large language model is shown. At 302, a training instruction set is generated based on multiple pathology-related sources. The training dataset can be selected to include any or all of the following: open-ended multi-turn dialogues, detailed image descriptions, short-answer questions, multiple-choice questions, object detection and segmentation masks, and plain text questions. Pathology-related sources can include, for example, image captions, educational articles, pathology case reports, and regions of interest extracted from internal whole-section images. At 304, the training instruction set is filtered to ensure quality and relevance. For example, overly brief, inadequate, or overly general image captions can be omitted. Captions or paragraphs relevant to animal pathology or experimental studies can be identified by appropriate keywords and omitted, for example, using a regular expression pattern matching process. At 306, the multimodal large language model is trained on the filtered training instruction set. It should be understood that training can be performed in multiple phases during which parameters associated with one or more components of the multimodal large language model can be frozen to avoid being affected by that training phase.

[0032] Figure 4 An example of a method 400 for providing natural language decision support for pathology is shown. At 402, a low-dimensional representation of the received pathology image is generated, and at 404, a first set of lexical units is generated based on the representation of the pathology image by projecting the low-dimensional representation of the received pathology image to the same dimension as the embedding space of a large language model used for text lexical units. It should be understood that multiple images can be associated with a given cue, and when multiple images are received, a low-dimensional representation can be generated for each image, and a first set of lexical units can be generated based on the low-dimensional representations of all images.

[0033] In one example, the received pathology image is a whole-slice image. In this example, the whole-slice image is divided into multiple tiles, each of which has a location associated with the received pathology image. The whole-slice image may be additionally segmented to locate a region of interest, which is also divided into multiple tiles corresponding to the received pathology image. The low-dimensional representation of the received pathology image includes a low-dimensional representation of each of the multiple tiles, and a first set of terms includes at least one subset of a first set of terms generated based on the low-dimensional representations of the multiple tiles. This subset of the first set of terms represents both the content of each tile and the associated location of that tile.

[0034] At 406, the large language model receives a second set of lexical units associated with a natural language prompt. In one example, the natural language prompt is provided by the user via a user interface, and the second set of lexical units is generated by a tokenizer associated with the user interface based on the prompt. At 408, the large language model determines a response based on the first and second sets of lexical units. The large language model is trained on an instruction dataset compiled from multiple pathology-related sources. In one example, the multiple pathology-related sources include explanatory text for medical images, educational articles, pathology case reports, and each item from regions extracted from whole-slice images. In one example, the user can provide feedback via the user interface on the provided response, which is used to determine whether a new instruction can be generated for the instruction dataset based on the response, the prompt, and the pathology image.

[0035] Figure 5 This is a schematic block diagram illustrating an exemplary system 500 of hardware components capable of implementing the systems and methods disclosed herein. System 500 may include various systems and subsystems. System 500 may be a personal computer, laptop computer, workstation, computer system, device, application-specific integrated circuit (ASIC), server, BladeCenter server, server cluster, etc.

[0036] System 500 may include a system bus 502, a processing unit 504, a system memory 506, storage devices 508 and 510, a communication interface 512 (e.g., a network interface), a communication link 514, a display 516 (e.g., a video screen), and an input device 518 (e.g., a keyboard, a touchscreen, and / or a mouse). System bus 502 may communicate with processing unit 504 and system memory 506. Additional storage devices 508 and 510 (e.g., hard disk drives, servers, stand-alone databases, or other non-transitory storage) may also communicate with system bus 502. System bus 502 interconnects processing unit 504, storage devices 506-510, communication interface 512, display 516, and input device 518. In some examples, system bus 502 may also interconnect with additional ports (not shown) (e.g., a Universal Serial Bus (USB) port).

[0037] Processing unit 504 may be a computing device and may include an application-specific integrated circuit (ASIC). Processing unit 504 executes a set of instructions to implement the operations of the examples disclosed herein. The processing unit may include a processing core.

[0038] Additional storage devices 506, 508, and 510 can store data, programs, instructions, text, or compiled database queries, as well as any other information that may be needed to operate the computer. Memory devices 506, 508, and 510 can be implemented as computer-readable media (integrated or removable), such as memory cards, disk drives, optical discs (CDs), or servers accessible via a network. In some examples, memory devices 506, 508, and 510 may include text, images, video, and / or audio, some of which may be provided in a human-understandable format.

[0039] Alternatively, system 500 may access external data sources or query sources through communication interface 512, which may communicate with system bus 502 and communication link 514.

[0040] In operation, system 500 can be used to implement one or more parts of a natural language system for pathological image analysis. According to some examples, computer-executable logic for implementing the diagnostic system resides in system memory 506 and one or more storage devices 508, 510. Processing unit 504 executes one or more computer-executable instructions derived from system memory 506 and storage devices 508, 510. As used herein, the term "computer-readable medium" refers to a medium that participates in providing instructions to processing unit 504 for execution. This medium may be distributed across multiple discrete components, all of which are operatively connected to a common processor or associated processor group.

[0041] The aforementioned techniques, blocks, steps, and means can be implemented in various ways. For example, these techniques, blocks, steps, and means can be implemented in hardware, software, or a combination thereof. In a hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the aforementioned functions, and / or combinations thereof.

[0042] Additionally, it should be noted that an embodiment can be described as a process, which can be represented by a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. Although a flowchart can describe operations as a sequential process, many operations can be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. When an operation of a process is completed, it ends, but may have additional steps not included in the diagram. A process can correspond to a method, function, program, subroutine, subroutine, etc. When a process corresponds to a function, its end corresponds to the function returning to the calling function or the main function.

[0043] Furthermore, embodiments can be implemented using hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented in software, firmware, middleware, scripting languages, and / or microcode, program code or code segments that perform the necessary tasks can be stored in a machine-readable medium (e.g., storage medium). Code segments or machine-executable instructions can represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, scripts, classes, or instructions, data structures, and / or program statements. Code segments can be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, and / or memory contents. Information, arguments, parameters, data, etc., can be passed, forwarded, or transmitted by any suitable means, including memory sharing, message passing, ticket passing, network transmission, etc.

[0044] For firmware and / or software implementations, these methodologies can be implemented using modules (e.g., programs, functions, etc.) that perform the functions described herein. Any machine-readable medium that tangibly embodies instructions can be used to implement the methodologies described herein. For example, software code can be stored in memory. Memory can be implemented inside or outside the processor. As used herein, the term "memory" refers to any type of long-term, short-term, transient, non-transient, or other storage medium, and is not limited to any particular type of memory, the number of memories, or the type of medium storing memory.

[0045] Furthermore, as disclosed herein, the term "storage medium" can refer to one or more memories used for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, disk storage media, optical storage media, flash memory devices, and / or other machine-readable media used for storing information. The term "machine-readable medium" includes, but is not limited to, portable or fixed storage devices, optical storage devices, wireless channels, and / or various other storage media capable of storing, containing, or carrying instructions and / or data.

[0046] In the foregoing description, specific details have been set forth to provide a full understanding of exemplary implementations of the invention described in this disclosure. However, it will be apparent that various implementations can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the exemplary implementations with unnecessary details. In other cases, unnecessary details of well-known circuits, processes, algorithms, structures, and techniques may be omitted to avoid obscuring the examples. The description of the exemplary implementations will provide those skilled in the art with an enabling description for implementing examples of the invention, but it should be understood that various changes to functionality and elements can be made without departing from the spirit and scope of the invention. Therefore, the invention is intended to cover all such changes, modifications, and variations falling within the scope of the appended claims.

Claims

1. A system for providing natural language decision support for pathology, comprising: processor; and A non-transitory computer-readable medium storing instructions executable by the processor, the machine-executable instructions including: A visual encoder that receives pathological images and generates a representation of the pathological images; A multimodal projector that generates a first set of lexical units based on the representation of the pathological image; and A large language model is trained on a dataset of instructions compiled from multiple pathology-related sources. Given training samples in the instruction dataset include groups of pathology images and text, the text describing or answering a specific query related to the images. The large language model receives a second set of lexical units associated with the prompts and determines a response based on the first set of lexical units and the second set of lexical units.

2. The system according to claim 1, wherein the executable instructions further include: A user interface that receives the prompts from the user in the form of natural language text and displays the response to the user on an associated display; as well as The word segmenter generates the second set of tokens.

3. The system according to claim 2, wherein, The response is the first of a plurality of responses, and the user interface allows the user to select a response from the plurality of responses. The image and the selected response are added to the instruction dataset as training samples.

4. The system according to claim 2, wherein, The response is the first of a plurality of responses, and the user interface allows the user to rate the response as helpful or not helpful. When the response is rated as helpful, the image and the selected response are added as training samples to the instruction dataset.

5. The system according to claim 1, wherein, One lexical from the second set of lexical units and the prompt are stored on the non-transitory computer-readable medium, and the second set of lexical units is provided to the large language model without user input.

6. The system according to claim 1 further includes an image segmenter, the image segmenter selecting a region of interest within an image, the pathological image being a plurality of patches generated based on the region of interest, and the first set of morphemes representing the content and location of the pathological image.

7. The system according to claim 1, wherein, The multiple pathology-related sources include at least two of the following: captions on medical images, educational articles, pathology case reports, and regions extracted from whole-slice imaging.

8. The system according to claim 1, wherein, The instruction dataset was selected to exclude experimental studies, which were identified using a regular expression pattern matching process using key phrase groups.

9. The system according to claim 1, wherein, The instruction dataset was selected to exclude any text shorter than a threshold length.

10. A method for providing natural language decision support for pathology, the method comprising: Generate a low-dimensional representation of the received pathological images; A first set of lexical units is generated based on the representation of the pathological image by projecting the low-dimensional representation of the received pathological image to the same dimension as the embedding space of a large language model used for text lexical units. The large language model is trained on an instruction dataset compiled from multiple pathology-related sources. The large language model receives a second set of lexical units associated with natural language prompts; as well as The response is determined based on the first set of lexical units and the second set of lexical units.

11. The method according to claim 10, wherein, The received pathological image is a whole-section image, and the method further includes: Segmenting the received pathological images to locate the region of interest; and The region of interest is divided into multiple patches, each of which has an associated location within the received pathological image. The process of generating the low-dimensional representation of the first received pathological image includes generating a low-dimensional representation of a first patch among the plurality of patches; and the process of generating the first set of lexical terms based on the representation of the pathological image includes generating a subset of the first set of lexical terms based on the low-dimensional representation of the first patch; the subset of the first set of lexical terms represents both the content of the first patch and the associated position of the first patch.

12. The method according to claim 10, wherein, The received pathological image is the first pathological image among a plurality of received pathological images, and the group that generates the first set of lexical units based on the representation of the pathological image includes: generating the first set of lexical units based on the plurality of received pathological images.

13. The method according to claim 10, wherein, The natural language prompts are provided by the user via a user interface, and the method further includes generating the second set of lexical units in a word segmenter associated with the user interface.

14. The method according to claim 13, wherein, The user can provide feedback on the response via the user interface, and the method further includes generating instructions for the instruction dataset based on the feedback provided by the user.

15. The method according to claim 10, wherein, The multiple pathology-related sources include captions for medical images, educational articles, pathology case reports, and each of the regions extracted from whole-slice imaging.

16. A system for providing natural language decision support for pathology, the system comprising: processor; and A non-transitory computer-readable medium storing instructions executable by the processor, the machine-executable instructions including: A visual encoder that receives pathological images and generates a representation of the pathological images; A multimodal projector that generates a first group of lexical units based on the representation of the pathological image; A user interface that receives prompts from the user in the form of natural language text; A word segmenter that generates a second set of word units based on the prompt; and A large language model is trained on an instruction dataset compiled from multiple pathology-related sources. Each training sample in the instruction dataset includes a group of pathology images and text, the text describing or answering a specific query related to the images. The large language model receives a second set of lexical units and determines a response based on the first set of lexical units and the second set of lexical units. The user interface displays the response to the user on an associated display.

17. The system according to claim 16, wherein, The instruction dataset was selected to exclude text associated with animal pathology images, where the experimental study was identified using a regular expression pattern matching process using key phrase groups.

18. The system according to claim 16, wherein, The instruction dataset was selected to exclude any text that was identified as a generic description of an image.

19. The system according to claim 16, wherein, The response is the first of a plurality of responses, and the user interface allows the user to select a response from the plurality of responses, and the image and the selected response are added to the instruction dataset as training samples.

20. The system according to claim 16, wherein, The response is the first of a plurality of responses, and the user interface allows the user to rate the response as either helpful or not helpful. When the response is rated as helpful, the image and the selected response are added to the instruction dataset as training samples.

21. The system according to claim 16, wherein, The user interface allows the user to provide task prompts and example data, which are stored for use in tasks that can be adapted to the user without training.

Images