Diagnostic tools for review of digital pathology images

Abstract

In one embodiment, a method includes accessing a slide image related to tissue for medical analysis, segmenting the slide image into a plurality of tiles, selecting one or more tiles from the plurality of tiles using one or more machine learning models based on one or more criteria related to the medical analysis, displaying the selected one or more tiles for review by a user via a user interface, receiving one or more user inputs related to the one or more tiles via the user interface, and generating analysis results related to the medical analysis based on the one or more user inputs and the one or more tiles.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 394,928, filed August 3, 2022, the disclosure of which is incorporated herein by reference in its entirety. [Technical Field]

[0002] FIELD OF THE INVENTION SUMMARY The present disclosure relates to systems and methods for improving the efficiency of medical analysis based on tissue slide images. [Background technology]

[0003] Introduction Pathologists can aid in the medical analysis of patients by analyzing slide images taken from their tissues. These slide images are usually stained, and pathologists are familiar with reviewing specific color schemes for immunohistochemistry (IHC) and histological stains. Pathologists can examine these images, select specific areas of each slide image, perform detailed zoom-ins, and then analyze these areas. In cancer treatment, pathologists can analyze areas based on specific representative regions of tumor tissue in the medical analysis of cancer patients. For example, diffuse large B-cell lymphoma (DLBCL) is a cancer that begins in white blood cells called lymphocytes. It typically grows in lymph nodes, pea-sized glands in the neck, groin, armpits, and other locations that are part of the human immune system. It can also appear in other areas of the body. DLBCL exhibits a relatively uniform pattern, reflecting the clonal growth of the disease. In DLBCL, diagnosis largely relies on analysis of nuclei (e.g., size, density, and content). The analysis results from the pathologist may be provided to the physician to aid in treatment decisions. To perform the analysis, the pathologist must typically analyze thousands of image tiles, which is extremely time-consuming and inefficient. Summary of the Invention

[0004] Overview of Certain Embodiments SUMMARY OF THE INVENTION Systems and methods are provided herein for improving the efficiency of medical analysis based on tissue slide images.

[0005] In certain embodiments, a digital pathology imaging system can improve the efficiency of a patient's medical diagnostic workflow based on slide images captured from the patient's tissue. The digital pathology imaging system can present the entire tissue image to a pathologist via a user interface in a software tool associated with the system, along with a gallery of filtered tiles generated from the tissue image. These filtered tiles can be determined by an algorithm that can display only tiles of high interest that contain positive regions for a desired diagnosis (e.g., tumor). For these tiles, the digital pathology imaging system can further generate segmentations (e.g., nuclei for tumor diagnosis). The digital pathology imaging system can then provide the pathologist with the option to disable inappropriate tiles, such as tiles with artifacts or tiles with improper segmentation, via the software tool. If a tile is disabled by the pathologist, another tile can be suggested by the digital pathology imaging system. Once the pathologist reviews and marks (e.g., approves or disables) a set of tiles via the software tool, the digital pathology imaging system can further generate analytical results (e.g., inferring a risk score for the patient's health to predict disease recurrence or resistance to treatment). The analysis results may be further presented to a pathologist for review via a software tool. Once the analysis results have been approved by the pathologist, the digital pathology imaging system may further generate a report including the analysis results. The report may include a prompt or request for the pathologist's approval signature, after which the report may be sent to other parties (e.g., clinics, hospitals, physicians, etc.) to assist in determining an appropriate treatment plan for the patient.

[0006] In certain embodiments, the digital pathology imaging system can access slide images related to tissue for medical analysis. The digital pathology imaging system can then segment the slide images into a plurality of tiles. The digital pathology imaging system can then select one or more tiles from the plurality of tiles using one or more machine learning models based on one or more criteria related to the medical analysis. In certain embodiments, the digital pathology imaging system can display the selected one or more tiles for user review via a user interface. The digital pathology imaging system can then receive one or more user inputs related to the one or more tiles via the user interface. Furthermore, the digital pathology imaging system can generate analysis results related to the medical analysis based on the one or more user inputs and the one or more tiles. [Brief explanation of the drawings]

[0007] [Figure 1] 1 illustrates a network of interacting computer systems that can be used as described herein in accordance with some embodiments of the present disclosure.

[0008] [Figure 2] 1 illustrates an exemplary method for facilitating review of tiles for medical analysis.

[0009] [Figure 3] 1 illustrates an exemplary workflow for determining a treatment for a patient.

[0010] [Figure 4] 1 illustrates an exemplary initial interface of a software tool for reviewing tiles.

[0011] [Figure 5] 1 shows an example of zooming in on tiles.

[0012] [Figure 6] 1 shows an exemplary tile undergoing review by a pathologist.

[0013] [Figure 7] 10 shows another exemplary tile undergoing review by a pathologist.

[0014] [Figure 8] An example of a pathologist's approval of the tile is shown.

[0015] [Figure 9] Illustrates an example of a tile being rejected by a pathologist.

[0016] [Figure 10] 1 shows an example of reaching the minimum number of accepted tiles.

[0017] [Figure 11] 1 illustrates an exemplary report generated by a digital pathology imaging system.

[0018] [Figure 12] 1 illustrates an example computing system. DETAILED DESCRIPTION OF THE INVENTION

[0019] explanation FIG. 1 illustrates a network 100 of interacting computer systems that may be used as described herein in accordance with some embodiments of the present disclosure.

[0020] The digital pathology imaging system 120 can generate one or more whole slide images or other related digital pathology images corresponding to a particular sample. For example, an image generated by the digital pathology imaging system 120 can include a stained section of a biopsy sample. As another example, an image generated by the digital pathology imaging system 120 can include a slide image of a liquid sample (e.g., a blood film). As another example, an image generated by the digital pathology imaging system 120 can include fluorescence microscopy, such as a slide image showing fluorescence in situ hybridization (FISH) after a fluorescent probe binds to a target DNA or RNA sequence.

[0021] Several types of samples (e.g., biopsies, solid samples, and / or tissue-containing samples) can be processed by the sample preparation system 121 to fix and / or embed the sample. The sample preparation system 121 can facilitate the infiltration of a fixative (e.g., a liquid fixative such as a formaldehyde solution) and / or an embedding substance (e.g., histological wax) into the sample. For example, the sample fixation subsystem can fix the sample by exposing the sample to a fixative for at least a threshold time (e.g., at least 3 hours, at least 6 hours, or at least 13 hours). The dehydration subsystem can dehydrate the sample (e.g., by exposing the fixed sample and / or a portion of the fixed sample to one or more ethanol solutions) and possibly clear the dehydrated sample using a clearing intermediate (e.g., including ethanol and histological wax). The sample embedding subsystem can infiltrate the sample with heated (e.g., therefore liquid) histological wax (e.g., one or more times over a corresponding predetermined period of time). Histological wax can include paraffin wax and possibly one or more resins (e.g., styrene or polyethylene). The sample and wax can then be cooled, and the wax-infiltrated sample can then be blocked out.

[0022] A sample slicer 122 can receive the fixed and embedded sample and create a set of sections. The sample slicer 122 can expose the fixed and embedded sample to cold or cryogenic temperatures. The sample slicer 122 can then cut the cooled sample (or a trimmed version thereof) to create a set of sections. Each section can have a thickness of (e.g.) less than 100 μm, less than 50 μm, less than 10 μm, or less than 5 μm. Each section can have a thickness of (e.g.) greater than 0.1 μm, greater than 1 μm, greater than 2 μm, or greater than 4 μm. Cutting of the cooled sample can be performed in a warm water bath (e.g., at a temperature of at least 30° C., at least 35° C., or at least 40° C.).

[0023] The automated staining system 123 can facilitate staining of one or more of the sample sections by exposing each section to one or more stains. Each section can be exposed to a predetermined amount of stain for a predetermined period of time. In some cases, a single section is exposed to multiple stains simultaneously or sequentially.

[0024] Each of the one or more stained sections can be presented to an image scanner 124, which can capture a digital image of the section. The image scanner 124 can include a microscope camera. The image scanner 124 can capture digital images at multiple magnifications (e.g., using a 10x objective, a 20x objective, a 40x objective, etc.). Image manipulation can be used to capture selected portions of the sample at a desired magnification range. The image scanner 124 can further capture annotations and / or morphemes identified by a human operator. In some cases, the section can be returned to the automated staining system 123 after capturing one or more images, washed, exposed to one or more additional stains, and re-imaged. When multiple stains are used, the stains can be selected to have different color profiles so that a first region of the image corresponding to a first portion of the section that has absorbed a large amount of the first stain can be distinguished from a second region of the image (or another image) corresponding to a second portion of the section that has absorbed a large amount of the second stain.

[0025] It will be appreciated that one or more components of digital pathology imaging system 120 may, in some cases, operate in conjunction with a human operator. For example, a human operator may move samples between various subsystems (e.g., of sample preparation system 121 or digital pathology imaging system 120) and / or initiate or terminate the operation of one or more subsystems, systems, or components of digital pathology imaging system 120. As another example, some or all of one or more components of the digital pathology imaging system (e.g., one or more subsystems of sample preparation system 121) may be partially or completely replaced by the actions of a human operator.

[0026] Additionally, while the various functions and components described and illustrated for the digital pathology imaging system 120 relate to the processing of solid and / or biopsy samples, it will be understood that other embodiments may relate to liquid samples (e.g., blood samples). For example, the digital pathology imaging system 120 may receive a liquid sample (e.g., blood or urine) slide including a base slide, a smeared liquid sample, and a cover. The image scanner 124 may then capture an image of the sample slide. Further embodiments of the digital pathology imaging system 120 may relate to capturing images of the sample using advanced imaging techniques, such as FISH, as described herein. For example, once fluorescent probes are introduced into the sample and allowed to bind to target sequences, appropriate imaging may be used to capture an image of the sample for further analysis.

[0027] A given sample may be associated with one or more users (e.g., one or more physicians, laboratory technicians, and / or healthcare providers) during processing and imaging. Associated users may include, by way of non-limiting example, the person who ordered the test or biopsy that resulted in the imaged sample, the person authorized to receive the test or biopsy results, or the person who performed the analysis of the test or biopsy sample, among others. For example, a user may correspond to a physician, pathologist, clinician, or subject. A user may use one or more user devices 130 to submit one or more (e.g., subject-identifying) requests to process a sample through digital pathology image generation system 120 and process the resulting images through digital pathology image processing system 110.

[0028] The digital pathology image generation system 120 can transmit images generated by the image scanner 124 back to the user device 130. The user device 130 then communicates with the digital pathology image processing system 110 to initiate automated processing of the images. In some cases, the digital pathology image generation system 120 provides images generated by the image scanner 124 directly to the digital pathology image processing system 110, for example, at the direction of a user of the user device 130. Although not shown, other intermediate devices (e.g., a data store on a server connected to the digital pathology image generation system 120 or the digital pathology image processing system 110) can also be used. Furthermore, for simplicity, only one digital pathology image processing system 110, image generation system 120, and user device 130 are shown in the network 100. The present disclosure contemplates the use of one or more of each type of system and its components without necessarily departing from the teachings of the present disclosure.

[0029] The network 100 and associated systems shown in FIG. 1 can be used in a variety of contexts where scanning and evaluating digital pathology images, such as whole slide images, is an essential component of work. As an example, the network 100 may be associated with a clinical environment where a user is evaluating a sample for the purpose of a possible diagnosis. The user may use a user device 130 to review the image before providing it to the digital pathology imaging system 110. The user may provide additional information to the digital pathology imaging system 110 that can be used to guide or direct the analysis of the image by the digital pathology imaging system 110. For example, the user may provide a predicted diagnosis or preliminary evaluation of features within the scan. Additionally, the user may provide additional context, such as the type of tissue being reviewed. As another example, the network 100 may be associated with a laboratory environment where tissue is being examined, for example, to determine the effectiveness or potential side effects of a drug. In this context, it may be common for multiple types of tissue to be subject to review to determine the drug's effects on the whole body. This may present particular challenges for human scan reviewers, who may need to determine various contexts of the image, which may be highly dependent on the type of tissue being imaged. These contexts can optionally be provided to the digital pathology imaging system 110.

[0030] The digital pathology image processing system 110 can process digital pathology images, including whole slide images, to classify the digital pathology images and generate annotations and related outputs for the digital pathology images. As an example, the digital pathology image processing system 110 can process whole slide images of tissue samples or tiles of whole slide images of tissue samples generated by the digital pathology image processing system 110 to identify tumor regions. As another example, the digital pathology image processing system 110 can process tiles of whole slide images of tissue samples to identify tiles with high interest or high risk scores for medical analysis. The digital pathology image processing system 110 may crop a query image into multiple image tiles. A tile generation module 111 can define a set of tiles for each digital pathology image. To define the set of tiles, the tile generation module 111 can segment the digital pathology image into the set of tiles. As embodied herein, tiles may be non-overlapping (e.g., each tile includes image pixels not included in any other tile) or overlapping (e.g., each tile includes a portion of image pixels included in at least one other tile). Features such as whether tiles overlap, as well as the size of each tile and the window stride (e.g., the image distance or pixels between one tile and the next), can increase or decrease the dataset for analysis; more tiles (e.g., overlapping or smaller tiles) can increase or decrease the possible resolution of the final output and visualization. In some cases, the tile generation module 111 defines a set of tiles for an image, each tile being a predetermined size and / or with a predefined offset between tiles. Continuing with the example of detecting gene fusions, each slide image may be cropped into image tiles having a width and height of a specific number of pixels. Additionally, the tile generation module 111 can create multiple sets of tiles for each image with various sizes, overlaps, step sizes, etc.For example, the pixel width and height may be dynamically determined (i.e., not fixed) based on factors such as the assessment task, the query image itself, or any suitable factor. In some embodiments, the digital pathology image itself may include tile overlap, which may result from the imaging technique. Even segmentation without tile overlap may be a preferred solution to balance tile processing requirements and avoid impacting the embedding and weight value generation described herein. The tile size or tile offset may be determined, for example, by calculating one or more performance metrics (e.g., precision, recall, accuracy, and / or error) for each size / offset and selecting a tile size and / or offset associated with one or more performance metrics above a predetermined threshold and / or associated with one or more performance metrics (e.g., high precision, high recall, accuracy, and / or low error).

[0031] The tile generation module 111 can further define the tile size depending on the type of abnormality being detected. For example, the tile generation module 111 can be configured with knowledge of the type of tissue abnormality the digital pathology imaging system 110 is searching for and can customize the tile size depending on the tissue abnormality to improve detection. For example, the image generation module 111 can determine that if the tissue abnormality includes searching for inflammation or necrosis in lung tissue, the tile size should be reduced to increase scanning speed, but if the tissue abnormality includes an abnormality in Kupffer cells in liver tissue, the tile size should be increased to increase the opportunity for the digital pathology imaging system 110 to analyze the Kupffer cells holistically. In some cases, the tile generation module 111 defines sets of tiles such that for each image, the number of tiles in the set, the size of the tiles in the set, the resolution of the tiles in the set, or other related characteristics are defined and held constant for each of one or more images.

[0032] As embodied herein, the tile generation module 111 can further define a set of tiles for each digital pathology image along one or more color channels or color combinations. By way of example, digital pathology images received by the digital pathology image processing system 110 can include large-format, multi-color channel images with pixel color values ​​for each pixel of the image specified for one of several color channels. Examples of usable color specifications or color spaces include RGB, CMYK, HSL, HSV, or HSB color specifications. The set of tiles can be defined based on segmenting the color channels and / or generating a luminance map or grayscale equivalent for each tile. For example, for each segment of the image, the tile generation module 111 can provide a red tile, a blue tile, a green tile, and / or a luminance tile, or equivalents for the color specifications used. As described herein, segmenting the digital pathology image based on the image segments and / or their color values ​​can generate embeddings for the tiles and images, improving the accuracy and recognition rate of networks used to generate classifications for the images. Additionally, digital pathology imaging system 110 can convert between color specifications and / or create copies of tiles using multiple color specifications, for example, using tile generation module 111. Color specification conversions can be selected based on the desired type of image enhancement (e.g., emphasis or enhancement of particular color channels, saturation levels, brightness levels, etc.). Additionally, color specification conversions can be selected to improve compatibility between digital pathology imaging generation system 120 and digital pathology imaging system 110. For example, certain image scanning components can provide output in HSL color specifications, and as described herein, models used in digital pathology imaging system 110 can be trained using RGB images. By converting tiles to a compatible color specification, the tiles can still be reliably analyzed.Additionally, the digital pathology imaging system can upsample or downsample images provided at a particular color depth (e.g., 8-bit, 1-bit, etc.) to be usable by the digital pathology imaging system. Additionally, the digital pathology imaging system 110 can convert tiles depending on the type of image captured (e.g., a fluorescence image may contain more detail in color intensity or a wider range of colors).

[0033] As described herein, the tile embedding module 112 can generate an embedding of each tile in a corresponding feature embedding space. The embedding can be represented by the digital pathology image processing system 110 as a feature vector for the tile. The tile embedding module 112 can use a neural network (e.g., a convolutional neural network) to generate the feature vector representing each tile of the image. In particular embodiments, the tile embedding neural network can be based on a ResNet image network trained on a dataset based on natural (e.g., non-medical) images, such as the ImageNet dataset. By using an unspecialized tile embedding network, the tile embedding module 112 can leverage known advances in efficiently processing images to generate embeddings. Furthermore, by using a natural image dataset, the embedding neural network can learn to distinguish between tile segments at a holistic level.

[0034] In other embodiments, the tile embedding network used by the tile embedding module 112 can be an embedding network customized to process the multiple tiles of large-format images, such as digital pathology whole slide images. Additionally, the tile embedding network used by the tile embedding module 112 can be trained using a custom dataset. For example, the tile embedding network can be trained using a variety of samples of whole slide images, or even samples related to the subject for which the embedding network is generating embeddings (e.g., scans of a particular tissue type). Training the tile embedding network using a specialized or customized set of images allows the tile embedding network to identify finer differences between tiles, resulting in more detailed and accurate distances between tiles in feature embedding space, at the expense of additional time to acquire images and the computational and financial costs of training multiple tile generation networks for use by the tile embedding module 112. The tile embedding module 112 can select from a library of tile embedding networks based on the type of image being processed by the digital pathology image processing system 110.

[0035] As described herein, tile embeddings can be generated from a deep learning neural network using visual features of the tiles. Additionally, tile embeddings can be generated from contextual information associated with the tiles or from the content depicted in the tiles. For example, a tile embedding can include one or more features that indicate and / or correspond to the size of the depicted object (e.g., the size of the depicted cell or abnormality) and / or the density of the depicted object (e.g., the density of the depicted cell or abnormality). Size and density can be measured absolutely (e.g., expressed in pixels or width converted from pixels to nanometers) or measured relative to other tiles from the same digital pathology image, a class of digital pathology images (e.g., generated using similar technology or by a single digital pathology image generation system or scanner), or a related family of digital pathology images. Additionally, tiles can be classified prior to generation of the tile embedding by the tile embedding module 112, and the tile embedding module 112 can take the classification into account when creating the embedding.

[0036] For consistency, the tile embedding module 112 may generate embeddings of a predetermined size (e.g., a 512-element vector, a 2048-byte vector, etc.). The tile embedding module 112 can also generate embeddings of various arbitrary sizes. The tile embedding module 112 can adjust the size of the embedding based on user instructions, or it may be selected based on, for example, computational efficiency, accuracy, or other parameters. In particular embodiments, the embedding size can be based on the limitations or specifications of the deep learning neural network that generated the embedding. Using a larger embedding size can increase the amount of information captured in the embedding and improve the quality and accuracy of the results, while using a smaller embedding size can improve computational efficiency.

[0037] The digital pathology image processing system 110 can perform different inferences by applying one or more machine learning models to the embeddings, i.e., inputting the embeddings into the machine learning models. As an example, the digital pathology image processing system 110 can identify tumor regions based on a machine learning model trained to identify tumor regions. As another example, the digital pathology image processing system 110 can identify high-attention or high-risk tiles based on a machine learning model trained to identify high-attention or high-risk tiles. In some embodiments, it may not be necessary to crop the image into image tiles, generate embeddings for these tiles, and then perform inference based on such embeddings. Instead, the digital pathology image processing system 110 can apply machine learning models directly to the embeddings of the whole slide image to perform inference with sufficient GPU memory. The output of the machine learning model may be resized to the shape of the input image.

[0038] The whole slide image access module 113 can manage requests to access whole slide images from other modules in the digital pathology imaging system 110 and the user device 130. For example, the whole slide image access module 113 receives a request identifying a whole slide image based on a particular tile, tile identifier, or whole slide image identifier. The whole slide image access module 113 can perform the tasks of verifying that the whole slide image is available to the requesting user, identifying the appropriate database from which to retrieve the requested whole slide image, and retrieving any additional metadata that may be of interest to the requesting user or module. Additionally, the whole slide image access module 113 can handle efficient streaming of appropriate data to the requesting device. As described herein, whole slide images can be provided to the user device in chunks based on the likelihood that the user will want to view portions of the whole slide image. The whole slide image access module 113 can determine which regions of the whole slide image to provide and how to provide them. Additionally, the whole slide image access module 113 may be authorized within the digital pathology imaging system 110 to ensure that individual components do not lock or otherwise misuse the database or whole slide images to the detriment of other components or users.

[0039] The output generation module 114 of the digital pathology imaging system 110 can generate output corresponding to result tiles and result whole slide image datasets based on user requests. As described herein, the output can include various visualizations, interactive graphics, and reports based on the type of request and the type of data available. In many embodiments, the output is provided to the user device 130 for display, but in certain embodiments, the output can be accessed directly from the digital pathology imaging system 110. Because the output is based on the presence of and access to appropriate data, the output generation module is authorized to access necessary metadata and de-identified patient information as needed. Like other modules of the digital pathology imaging system 110, the output generation module 114 can be updated and improved in a modular manner, allowing new output features to be provided to users without requiring significant downtime.

[0040] The general techniques described herein can be integrated into a variety of tools and use cases. For example, as described above, a user (e.g., a pathologist or clinician) can access a user device 130 that communicates with the digital pathology imaging system 110 and provide a query image for analysis. The digital pathology imaging system 110, or a connection to the digital pathology imaging system, can be provided as a standalone software tool or package that searches for corresponding matches, identifies similar features, and generates appropriate output for the user upon request. As standalone tools or plug-ins that can be purchased or licensed on a streamlined basis, the tools can be used to enhance the capabilities of research or clinical laboratories. Additionally, the tools can be integrated into services made available to customers of the digital pathology imaging system. For example, the tools can be provided as a unified workflow in which a user performing or requesting the creation of a whole slide image automatically receives a report of notable features in previously indexed images and / or similar whole slide images. Thus, in addition to improving the analysis of whole slide images, these techniques can be integrated into existing systems to provide additional features not previously considered or possible.

[0041] Additionally, the digital pathology imaging system 110 may be trained and customized for use in a particular setting. For example, the digital pathology imaging system 110 may be specifically trained for use in providing insights into a particular type of tissue (e.g., lung, heart, blood, liver, etc.). As another example, the digital pathology imaging system 110 may be trained to assist in safety assessment, for example, in determining the level or degree of toxicity associated with a drug or other potential therapeutic treatment. Once trained for use in a particular subject or use case, the digital pathology imaging system 110 is not necessarily limited to that use case. Training may be performed in a particular context, such as toxicity assessment, with a relatively larger set of at least partially labeled or annotated images.

[0042] FIG. 2 illustrates an exemplary method 200 for facilitating review of tiles for medical analysis. The method may begin at step 210, where the digital pathology imaging system 110 may access slide images relating to tissue for medical analysis. In certain embodiments, such slide images may correspond to stained slides taken from patient tissue. The reason for staining the slides may be that object detection may require color patterns familiar to pathologists for greater efficiency. By way of example and not limitation, the slides may be stained with hematoxylin and eosin (H&E). The stained slides may then be digitized (e.g., scanned) to generate slide images.

[0043] In step 220, the digital pathology image processing system 110 can segment the slide image into multiple tiles. In certain embodiments, a tile generation module 111 can be used to generate the tiles. The tiles can be non-overlapping or overlapping. Features such as the size of each tile and the window stride, as well as whether the tiles overlap, can increase or decrease the data set for analysis; the more tiles there are, the higher the possible resolution of the final output and visualization. In certain embodiments, each tile can be a predetermined size, and / or the offset between tiles can be predefined. Furthermore, the tile generation module 111 can create multiple sets of tiles of various sizes, overlaps, step sizes, etc. for each image. The tile generation module 111 can generate tiles for each digital pathology image along one or more color channels or color combinations. The tiles can be generated based on segmenting the color channels and / or generating an intensity map or grayscale equivalent for each tile. Additionally, the digital pathology imaging system 110 can upsample or downsample images provided at a particular color depth usable by the digital pathology imaging system 110. Additionally, the digital pathology imaging system 110 can convert tiles depending on the type of image captured.

[0044] In step 230, the digital pathology imaging system 110 may select one or more tiles from the plurality of tiles using one or more machine learning models based on one or more criteria related to the medical analysis. In certain embodiments, the digital pathology imaging system 110 may select these tiles as follows: The digital pathology imaging system 110 may use a quality check (QC) algorithm to filter out artifact tiles. The digital pathology imaging system 110 may then use a condition detection algorithm to filter out tiles corresponding to normal regions (e.g., if the medical analysis includes tumor analysis, non-tumor tiles may be considered "not normal" and therefore removed). The digital pathology imaging system 110 may then use another algorithm to pre-select tiles of interest based on different criteria. In certain embodiments, the one or more criteria may include one or more of high salience or high representativeness of the disease targeted by the medical analysis.

[0045] At step 240, the digital pathology imaging system 110 may display the selected one or more tiles for user review via a user interface. In certain embodiments, the user interface may show individual tiles or clusters of tiles to the pathologist. The initial interface may provide a slide overview, a slide tile review, and a slide navigator. As an example, the tile review portion may include tiles, and the pathologist can scroll down to review other tiles. The digital pathology imaging system 110 may display the position of each selected tile relative to the tissue via the user interface. Additionally, the user interface may provide various visualization options. By way of example and not limitation, the user interface may allow the pathologist to review the tiles under different stains. The user interface may be operable to adjust the display of each of the selected one or more tiles based on one or more of an H&E stain or a virtual stain artificially generated by the digital pathology imaging system 110. For example, tiles may appear as if stained with DAB stain (i.e., 3,3'-diaminobenzidine oxidized by hydrogen peroxide in a reaction typically catalyzed by horseradish peroxidase (HRP)) rather than H&E. Additionally, the user interface may merge or overlap tiles. Additionally, the user interface may provide channels that the pathologist can enable / disable.

[0046] In step 250, the digital pathology imaging system 110 may receive one or more user inputs regarding one or more tiles via a user interface. In certain embodiments, the one or more user inputs may include one or more of tile acceptance, tile rejection, or tile scores. The pathologist can accept or reject each tile based on verified input (e.g., properly segmented tumor nuclei). For each rejected tile, the digital pathology imaging system 110 can algorithmically replace it with another preselected tile (e.g., with high attention value or high representativeness) to maintain an optimal number of tiles. In an alternative embodiment, the pathologist can provide a score for each tile instead of a simple acceptance or rejection. Specific instructions with strict criteria for when to accept or reject tiles can be provided to the pathologist prior to the start of the review process.

[0047] In step 260, the digital pathology imaging system 110 can generate an analysis result for the medical analysis based on one or more user inputs and one or more tiles. The generation of the analysis result may be automatically triggered in response to determining that the number of approved tiles reaches a predetermined number. In other words, when a minimum number of tiles is reached, a statistical analysis (e.g., calculation of a risk score) may be triggered. The digital pathology imaging system 110 can perform the statistical analysis based on the tiles approved by the pathologist. Alternatively, if the pathologist provides a score for the tile instead of approving / rejecting the tile, the digital pathology imaging system 110 can perform the statistical analysis using the tile score. In certain embodiments, the analysis result may include one or more of a risk score indicating the likelihood of disease recurrence, a risk score indicating the likelihood of resistance to treatment, or a probability indicating the risk of recurrence or incurability at a particular time point. Certain embodiments may repeat one or more steps of the method of FIG. 2 as appropriate. Although this disclosure describes and illustrates certain steps of the method of Figure 2 as occurring in a particular order, this disclosure contemplates any suitable steps of the method of Figure 2 occurring in any suitable order. Further, while this disclosure describes and illustrates exemplary methods for facilitating review of tiles for medical analysis that include certain steps of the method of Figure 2, this disclosure contemplates any suitable method for facilitating review of tiles for medical analysis that include any suitable steps, which may include all, some, or none of the steps of the method of Figure 2, where appropriate. Further, while this disclosure describes and illustrates certain components, devices, or systems that perform certain steps of the method of Figure 2, this disclosure contemplates any suitable combination of any suitable components, devices, or systems that perform any suitable steps of the method of Figure 2.

[0048] Medical analysis of slide images for tissue may depend on a combination of various factors, such as an attention score (i.e., whether a tile on the slide image is important) and a risk score (i.e., whether the tile is high-risk or low-risk). By way of example and not limitation, in DLBCL, detecting tumor regions by an algorithm may be difficult. Selection may need to be supervised by a pathologist. Furthermore, selection of relevant regions (e.g., tiles) may require zoom and pan functionality to enable visualization of the histological context. Even though the algorithm may be perceived as a black box, a pathologist may need to sign off on the analysis results, indicating that keeping a human (e.g., a pathologist) in the loop may be important to increase the reliability of the analysis. In consideration of the above-mentioned factors essential for effective analysis of slide images, embodiments disclosed herein have developed a digital pathology image processing system 110 that integrates machine learning and human expertise to select tiles and then generate analysis results based on the selected tiles.

[0049] FIG. 3 illustrates an exemplary workflow 300 for determining treatment for a patient. After a patient first sees a physician, the patient may proceed to clinical trial screening in step 310. In certain embodiments, the medical analysis may include determining one or more of disease recurrence or resistance to treatment. Then, in step 320, a request for a specific test related to the patient's disease (e.g., tumor) may be sent to a laboratory for sending stained slides taken from the patient's tissue to a clinical research organization (CRO). The reason for staining the slides may be that object detection may require a color pattern familiar to pathologists for greater efficiency. By way of example and not limitation, the slides may be stained with H&E. In step 330, the pathologist may select a reference slide. In step 340, the slides, along with the reference slide, may be shipped to the CRO. In step 350, a technician may push or scan the non-digital slides to digitize them, i.e., generate slide images.

[0050] In step 360, the digital pathology imaging system 110 can store the slide images in a storage device and then analyze them using artificial intelligence (AI) and machine learning while keeping the pathologist in the loop based on the following substeps: In certain embodiments, the machine learning model can be based on a neural network. In substep 360a, the pathologist can begin the analysis on one selected slide that is representative (e.g., exhibits a dominant histological pattern) of the medical case (e.g., tumor) under analysis. The digital pathology imaging system 110 can prepare the slide images accordingly.

[0051] In substep 360b, the digital pathology image processing system 110 may verify the input. In certain embodiments, the digital pathology image processing system 110 may generate a first subset from the plurality of tiles by removing one or more first tiles from the plurality of tiles. Each of the one or more first tiles may include an artifact, and one or more selected tiles may be selected from the first subset. Specifically, the digital pathology image processing system 110 may use a quality check (QC) algorithm to remove artifacts from the slide image. In certain embodiments, the digital pathology image processing system 110 may generate a second subset from the first subset by removing one or more second tiles from the first subset. Each of the one or more second tiles may correspond to an area for medical analysis, and one or more selected tiles may be selected from the second subset. Specifically, the digital pathology image processing system 110 may use a condition detection algorithm to remove normal tiles. By way of example and not limitation, if the medical analysis includes a tumor analysis, the generation of the second subset can be based on a tumor detection algorithm, and each of the tiles in the second subset can include a tumor.

[0052] The digital pathology image processing system 110 may then pre-select tiles of interest based on different criteria using another algorithm. In certain embodiments, the one or more criteria may include one or more of high attention value or high representativeness of the disease targeted by the medical analysis. By way of example and not limitation, such an algorithm may be based on the attention value for each tile. In certain embodiments, such an algorithm may determine that tiles with high attention value are most influential and their patterns are most relevant. For example, the pre-selected tiles may all have high attention value. Further information regarding high attention learning can be found in U.S. Patent Application No. 63 / 108,659, filed November 2, 2020, which is incorporated by reference in its entirety. By way of another example and not limitation, tile pre-selection may be based on the representativeness of the disease of interest (e.g., tumor). For example, the pre-selected tiles may all be representative. If the tissue is associated with a patient with a tumor, the digital pathology image processing system 110 may further generate a segmentation for each selected tile that includes the nucleus. In certain embodiments, the algorithm for pre-selection may be trained based on expert experience. By way of example and not limitation, an expert can be asked to annotate multiple images and indicate what the most representative regions (e.g., tiles) of the patient are, and then an algorithm can be trained to find those exact regions. A pathologist can then perform a quality check on these tiles pre-selected by the algorithm. In certain embodiments, the digital pathology image processing system 110 can generate a user interface via a software tool to allow the pathologist to easily perform the quality check. In addition to the pathologist, the user interface may be viewable / accessible by any party under a clinical disclosure agreement.

[0053] In certain embodiments, the user interface may present individual tiles or clusters of tiles to the pathologist. The pathologist may review each tile and provide user input. In certain embodiments, the one or more user inputs may include one or more of tile acceptance, tile rejection, or tile score. The pathologist may accept or reject each tile based on verified input (e.g., properly segmented tumor nuclei). In an alternative embodiment, the pathologist may provide a score for each tile instead of simple acceptance or rejection. Specific instructions with strict criteria for when to accept or reject a tile may be provided to the pathologist prior to the start of the review process. In an alternative embodiment, the digital pathology imaging system 110 may further provide annotations for each tile to the pathologist that can aid in the pathologist's review of the tile. The annotations may be generated by an algorithm or another pathologist.

[0054] For each approved tile, the digital pathology imaging system 110 can provide a visualization that can help the pathologist better review it. By way of example and not limitation, if a tile is tumor-related, the digital pathology imaging system 110 can indicate the nuclei with orange (e.g., DAB staining), red staining, or blue staining, and a flag based on the pathologist's familiarity. If one or more user inputs include one or more rejections of one or more tiles, the digital pathology imaging system 110 can further include selecting one or more additional tiles from the plurality of tiles for review by the user using one or more machine learning models based on one or more criteria related to medical analysis. In other words, for each rejected tile, the digital pathology imaging system 110 can replace it with another tile (e.g., having high attention value or high representativeness) preselected by an algorithm to maintain an optimal number of tiles (e.g., 50). In an alternative embodiment, the digital pathology imaging system may not replace rejected tiles. Instead, there may be an excess amount of tiles available for preselection. The digital pathology imaging system can continue pre-selecting tiles from this excess amount for the pathologist to review until there are enough tiles approved by the pathologist to generate an analysis result. In certain embodiments, if one or more user inputs include one or more approvals of one or more tiles, the digital pathology imaging system 110 may further determine whether the number of approved one or more tiles reaches a predetermined number. Generation of the analysis result may be automatically triggered in response to determining that the number of approved one or more tiles reaches the predetermined number. In other words, reaching a minimum number of tiles may trigger a statistical analysis (e.g., calculation of a risk score). In alternative embodiments, generation of the analysis result may be automatically triggered in response to determining that a particular region of the pre-selected tiles has been reached. By way of example and not limitation, there may be certain regions that are more important. The digital pathology imaging system can first pre-select other regions for the pathologist to review.However, the process can continue until the digital pathology imaging system pre-selects this specific region for review by the pathologist and the pre-selection is approved. The digital pathology imaging system 110 can perform statistical analysis based on the tiles approved by the pathologist. Alternatively, if the pathologist provides a score for the tile instead of approving / rejecting the tile, the digital pathology imaging system 110 can perform statistical analysis using the tile's score. In certain embodiments, the approve / reject workflow can increase the speed of the pathologist's review by minimizing clicks by the pathologist.

[0055] The digital pathology imaging system 110 can then output the statistical analysis via a software tool and present it to a pathologist for further review. In certain embodiments, the analysis results may include one or more of a risk score indicating the likelihood of disease recurrence, a risk score indicating the likelihood of resistance to treatment, or a probability indicating the risk of recurrence or refractory disease at a particular time point.

[0056] In certain embodiments, the digital pathology imaging system 110 can receive one or more additional user inputs via a user interface, including one or more of approval of the analysis results, adjustment of the analysis results, or override of the analysis results. Specifically, the pathologist can approve the statistical analysis (e.g., risk score), which can trigger the generation of a report. Alternatively, the pathologist can adjust the final output, for example, to adjust the risk score, and in some cases even override the final output. The digital pathology imaging system 110 can then generate a medical report based on the one or more additional user inputs. In substep 360c, the digital pathology imaging system 110 can present the generated report to the pathologist for review. By way of example and not limitation, the pathologist can review the report by checking each element (checkbox) and approve the final report. Thereafter, the digital pathology imaging system 110 can receive an approval signature for the medical report. In substep 360d, the digital pathology imaging system 110 can issue an electronic report to the pathologist so that the pathologist can provide an approval signature.

[0057] In step 370, the report including the analysis results may be stored in a test database for easy access by interested parties. In step 380, the patient may visit the doctor again. In step 390, the patient may receive treatment, which may be determined based on the report including the analysis results.

[0058] FIG. 4 illustrates an exemplary initial interface of a software tool for reviewing tiles. The initial interface may provide a slide overview 410, a slide tile review portion 420, and a slide navigator 430. Additionally, the initial interface may provide various view options 440. By way of example and not limitation, these view options 440 may include pan, zoom, ICC profile, intensity, photo mode, and case information. The initial interface may further provide various annotation options 450. By way of example and not limitation, these annotation options 440 may include flexpoly, rectangle, and counter. Flexpoly may allow a user to draw freehand annotations to create any desired shape. Rectangle may allow a user to draw annotations in the form of rectangles of various sizes. Counter may allow a user to annotate any object by placing a mark (e.g., a colored dot) in its center, and each dot may be labeled as desired by the user. Additionally, the initial interface may provide various editing options 460. By way of example and not limitation, these editing options 460 may include selection, shape editing, rotation, and translation. A user may use these editing options 460 if they need to modify or change an existing annotation. This modification or change may include changing the size, proportion, and position of an existing annotation. In certain embodiments, the initial interface may display a slide identifier (ID) 470 and annotations 480. In certain embodiments, the slide 490 may be stained. By way of example and not limitation, the slide 490 in FIG. 4 may be H&E stained. As shown in FIG. 4, the tile review portion 420 may include tiles, and the pathologist may scroll down to review tiles other than tile 1 420a, tile 2 420b, and tile 3 420c. The pathologist may reject or approve the tiles.The minimum number of tiles to be accepted may be 50, and currently no tiles have been accepted yet (ie, 0 / 50).

[0059] FIG. 5 shows an example of zooming in on tiles. As shown in FIG. 5, a pathologist may be reviewing tile 1 420a. In certain embodiments, the digital pathology imaging system 110 may display one or more positions of each of the selected one or more tiles relative to the tissue via a user interface. As shown in FIG. 5, a slide navigator 430 may indicate the position of this tile 420a relative to the entire slide image. Additionally, software tools may provide various visualization options. In certain embodiments, the digital pathology imaging system 110 may artificially generate a virtual stain (e.g., a virtual DAB stain) for the tissue using one or more machine learning models based on information derived from the underlying segmentation results. If the tissue is originally stained based on H&E stain, the user interface may be operable to adjust the display of each of the selected one or more tiles based on one or more of the H&E stain or the virtual DAB stain. By way of example and not limitation, if all tumor nuclei are detected and segmented by a machine learning model, the segmentation results may be displayed as a virtual DAB stain using a slider function within the user interface. It may not require registration or overlay of the true DAB stain. As shown in FIG. 5, the left portion of the tile 420a under review can display the virtual DAB stain, while the right portion of the tile 420a under review can display the original H&E stain. The pathologist can easily perform left-right slides using the separator 510 to review the same portion of the tile 420a under different stains. Additionally, the software tool may merge or overlay tiles 420a. Additionally, the software tool may provide channels that the pathologist can enable / disable. By way of example and not limitation, using channels, the software tool may create three different channel positions.

[0060] Figure 6 shows exemplary tiles under review by a pathologist. As shown in Figure 6, the pathologist may be reviewing tile 2 420b. The slide navigator 430 may indicate the location of this tile 420b relative to the overall slide image.

[0061] 7 shows another example tile under review by a pathologist. As shown in FIG. 7, the pathologist may be reviewing tile 3 420c. The slide navigator 430 may indicate the location of this tile 420c relative to the overall slide image.

[0062] FIG. 8 shows an example of a pathologist approving tiles. As shown in FIG. 8, the pathologist may have approved tile 1 420a. As a result, the currently approved tile may be 1 of 50. The pathologist can revoke the approval. The pathologist may currently be reviewing tile 2 420b. The slide navigator 430 can indicate the location of tile 2 420b relative to the overall slide image.

[0063] FIG. 9 illustrates an example of a tile rejection by a pathologist. As shown in FIG. 9, the pathologist may have rejected tile 2 420b. The pathologist can undo the rejection. The pathologist may now be reviewing tile 3 420c. The slide navigator 430 can indicate the location of tile 3 420c relative to the overall slide image. Due to the rejection of tile 2 420b, the digital pathology imaging system 110 can replace the rejected tile 2 420b with another tile for the pathologist to review.

[0064] Figure 10 shows an example of reaching the minimum number of approved tiles. As shown in Figure 10, 50 tiles may now be approved of the required 50 tiles. Approval of the minimum number of tiles can trigger statistical analysis by the digital pathology imaging system 110. As a result, the pathologist can select "View Results" 1010 to view the analysis results.

[0065] FIG. 11 illustrates an exemplary report 1100 generated by the digital pathology imaging system 110. By way of example and not limitation, the report 1100 may include patient information 1110, such as patient name, gender, date of birth, report date, specimen source / identifier (ID), and physician. Additionally, the report may include a risk score 1120 (e.g., 60) and a risk of relapse / refractory disease at 24 months (e.g., 80%) 1130. The software tool may allow the pathologist to confirm the histology of DLBCL 1140. For example, the pathologist may reject or approve that the histology has been confirmed. Additionally, the software tool may allow the pathologist to reject or approve that the image represents the patient 1150. Additionally, the software tool may allow the pathologist to approve the calculated risk score. Approving the calculated risk score automatically allows the pathologist to sign off on the report.

[0066] In certain embodiments, analysis of slide images based on the above-described workflow, i.e., pre-selection of tiles based on a machine learning model and review of the pre-selected tiles by a pathologist, can include predicting the cell of origin of the slide image. In certain embodiments, to predict the cell of origin, the pre-selection of tiles can be based on a machine learning model and can be considered as region proposals. The presently disclosed embodiments have conducted experiments on cell of origin prediction, with the following results. The results show that by integrating region proposals by a machine learning model and review by a pathologist, the digital pathology image processing system 110 can improve the accuracy of predicting the cell of origin compared to using only a machine learning model for region proposals. The region proposal results are improved after a manual quality check review, in which the pathologist rejected some of the regions. The experiments are based on multiple (e.g., 97) Goya test set slides with manual tumor annotations. The comparison of results, evaluated by AUC (area under the receiver operating characteristic curve), is as follows: The AUC of the baseline model using manual tumor annotations is 74.3%. The AUC of the region proposals without pathologist review is 72.6%. The AUC for region suggestions with pathologist review is 74.2%. Because the embodiments disclosed herein are not limited to manual tumor annotation, results can also be reported for a larger cohort of 129 slides, including slides without manual tumor annotation. The AUC for region suggestions without pathologist review is 75.3%. The AUC for region suggestions with pathologist review is 76.7%.

[0067] 12 illustrates an exemplary computer system 1200. In particular embodiments, one or more computer systems 1200 perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems 1200 provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems 1200 performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems 1200. As used herein, references to a computer system can encompass computing devices, and vice versa, where appropriate. Furthermore, references to a computer system can encompass one or more computer systems, where appropriate.

[0068] The present disclosure contemplates any suitable number of computer systems 1200. The present disclosure contemplates computer system 1200 taking any suitable physical form. By way of example and not limitation, computer system 1200 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system 1200 may include one or more computer systems 1200 and may be unitary, distributed, spanning multiple locations, multiple machines, multiple data centers, or in a cloud which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1200 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example and not limitation, one or more computer systems 1200 may perform one or more steps of one or more methods described or illustrated herein in real time or batch mode. One or more computer systems 1200 may, where appropriate, perform one or more steps of one or more methods described or illustrated herein at different times or in different locations.

[0069] In a particular embodiment, computer system 1200 includes a processor 1202, a memory 1204, a storage device 1206, an input / output (I / O) interface 1208, a communication interface 1210, and a bus 1212. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0070] In particular embodiments, processor 1202 includes hardware for executing instructions, such as those comprising a computer program. By way of example and not limitation, to execute instructions, processor 1202 may retrieve (or fetch) instructions from an internal register, an internal cache, memory 1204, or storage device 1206, decode and execute them, and then write one or more results to an internal register, an internal cache, memory 1204, or storage device 1206. In particular embodiments, processor 1202 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor 1202 including any suitable number of any suitable internal caches, where appropriate. By way of example and not limitation, processor 1202 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in an instruction cache may be copies of instructions in memory 1204 or storage device 1206, and the instruction cache may speed up retrieval of those instructions by processor 1202. Data in the data cache may be a copy of data in memory 1204 or storage device 1206 for the operation of an instruction executed in processor 1202, the results of a previous instruction executed in processor 1202 for access by a subsequent instruction executed in processor 1202 or for writing to memory 1204 or storage device 1206, or other suitable data. The data cache may speed up read or write operations by processor 1202. The TLB may speed up virtual address translation for processor 1202. In particular embodiments, processor 1202 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor 1202 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor 1202 may include one or more arithmetic logic units (ALUs), may be a multi-core processor, or may include one or more processors 1202.Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

[0071] In particular embodiments, memory 1204 includes main memory for storing instructions to be executed by processor 1202 or data for operation of processor 1202. By way of example and not limitation, computer system 1200 may load instructions into memory 1204 from storage device 1206 or another source (e.g., another computer system 1200, etc.). Processor 1202 can then load the instructions from memory 1204 into an internal register or internal cache. To execute instructions, processor 1202 may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of an instruction, processor 1202 may write one or more results (which may be intermediate or final results) to an internal register or internal cache. Processor 1202 may then write one or more of those results to memory 1204. In particular embodiments, processor 1202 executes only instructions in one or more internal registers or internal caches or in memory 1204 (as opposed to storage device 1206 or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory 1204 (as opposed to storage device 1206 or elsewhere). One or more memory buses (each of which may include an address bus and a data bus) may connect processor 1202 to memory 1204. Bus 1212 may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor 1202 and memory 1204 to facilitate accesses to memory 1204 requested by processor 1202. In particular embodiments, memory 1204 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. This RAM may be dynamic RAM (DRAM) or static RAM (SRAM), where appropriate. Further, this RAM may be single-ported RAM or multi-ported RAM, where appropriate. This disclosure contemplates any suitable RAM. Memory 1204 may include one or more memories 1204, where appropriate.Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

[0072] In particular embodiments, storage 1206 includes mass storage for data or instructions. By way of example and not limitation, storage 1206 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Storage 1206 may include removable or non-removable (i.e., fixed) media, where appropriate. Storage 1206 may be internal or external to computer system 1200, where appropriate. In particular embodiments, storage 1206 is non-volatile solid-state memory. In particular embodiments, storage 1206 includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these. This disclosure contemplates mass storage 1206 taking any suitable physical form. Storage 1206 may include, where appropriate, one or more storage control units that facilitate communications between processor 1202 and storage 1206. Where appropriate, storage 1206 may include one or more storage devices 1206. Although this disclosure describes and illustrates particular storage devices, this disclosure contemplates any suitable storage device.

[0073] In particular embodiments, I / O interface 1208 includes hardware, software, or both that provide one or more interfaces for communication between computer system 1200 and one or more I / O devices. Computer system 1200 may include one or more of these I / O devices, where appropriate. One or more of these I / O devices may enable communication between a person and computer system 1200. By way of example and not limitation, the I / O devices may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, other suitable I / O device, or a combination of two or more thereof. The I / O devices may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interface 1208 therefor. Where appropriate, I / O interface 1208 may include one or more device or software drivers that enable processor 1202 to drive one or more of these I / O devices. I / O interface 1208 may include, where appropriate, one or more I / O interfaces 1208. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0074] In particular embodiments, communication interface 1210 includes hardware, software, or both that provide one or more interfaces for communications (e.g., packet-based communications, etc.) between computer system 1200 and one or more other computer systems 1200 or one or more networks. By way of example and not limitation, communication interface 1210 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a Wi-Fi network. This disclosure contemplates any suitable network and any suitable communication interface 1210 for that network. By way of example and not limitation, computer system 1200 may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet, or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. By way of example, computer system 1200 may communicate with a wireless PAN (WPAN) (e.g., a BLUETOOTH WPAN, etc.), a Wi-Fi network, a Wi-MAX network, a cellular network (e.g., a Global System for Mobile Communications (GSM) network), or any other suitable wireless network, or a combination of two or more of these. Computer system 1200 may include any suitable communication interface 1210 for any of these networks, where appropriate. Communication interface 1210 may include one or more communication interfaces 1210, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

[0075] In particular embodiments, bus 1212 includes hardware, software, or both that connects components of computer system 1200 to one another. By way of example and not limitation, bus 1212 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or any other suitable bus, or a combination of two or more thereof. Bus 1212 may include one or more buses 1212, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0076] As used herein, a non-transitory computer-readable storage medium may include, where appropriate, one or more semiconductor-based or other integrated circuits (ICs) (e.g., field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable non-transitory computer-readable storage media, or any suitable combination of two or more of these. A non-transitory computer-readable storage medium may, where appropriate, be volatile, non-volatile, or a combination of volatile and non-volatile.

[0077] As used herein, "or" is inclusive and not exclusive, unless expressly stated otherwise or clear from the context. Thus, as used herein, "A or B" means "A, B, or both," unless expressly stated otherwise or clear from the context. Furthermore, "and" is both jointly and severally, unless expressly stated otherwise or clear from the context. Thus, as used herein, "A and B" means "A and B jointly or severally," unless expressly stated otherwise or clear from the context.

[0078] The scope of the present disclosure encompasses all changes, substitutions, variations, changes, and modifications to the exemplary embodiments described or illustrated herein that would be understood by one skilled in the art. The scope of the present disclosure is not limited to the exemplary embodiments described or illustrated herein. Furthermore, although the present disclosure describes and illustrates each embodiment herein as including particular components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that would be understood by one skilled in the art. Furthermore, references in the appended claims to an apparatus or system or a component of an apparatus or system that is arranged, arranged, enabled, configured, enabled, operable, or operates to perform a particular function encompass that apparatus, system, or component, so long as it is so arranged, arranged, enabled, configured, enabled, operable, or operates, regardless of whether it or that particular function is activated, turned on, or released. Furthermore, although the present disclosure describes or illustrates particular embodiments as providing certain advantages, a particular embodiment may provide none, some, or all of these advantages.

[0079] Illustrative Embodiments Embodiments disclosed herein may include the following. 1. A method comprising: accessing, by a digital pathology imaging system, slide images relating to tissue for medical analysis; segmenting the slide images into a plurality of tiles; selecting one or more tiles from the plurality of tiles using one or more machine learning models based on one or more criteria relating to the medical analysis; displaying, via a user interface, the selected one or more tiles for review by a user; receiving, via the user interface, one or more user inputs relating to the one or more tiles; and generating analysis results relating to the medical analysis based on the one or more user inputs and the one or more tiles. 2. The method of embodiment 1, further comprising: generating a first subset from the plurality of tiles by removing one or more first tiles from the plurality of tiles, wherein each of the one or more first tiles includes the artifact, and the one or more tiles are selected from the first subset. 3. The method of embodiment 2, further comprising: generating a second subset from the first subset by removing one or more second tiles from the first subset, each of the one or more second tiles corresponding to an area for medical analysis, and one or more tiles being selected from the second subset. 4. The method of embodiment 3, wherein the medical analysis includes tumor analysis, and generating the second subset is based on a tumor detection algorithm, and each tile in the second subset includes a tumor. 5. The method of any one of embodiments 1-4, wherein the one or more criteria include one or more of high salience or high representativeness of the disease for which the medical analysis is aimed. 6. The method of any one of embodiments 1 to 5, wherein the tissue relates to a patient having a tumor, and the method further comprises generating a segmentation including a nucleus for each of the one or more selected tiles. 7. The method of any one of embodiments 1-6, wherein the medical analysis includes determining one or more of disease recurrence or resistance to treatment. 8. The method of any one of embodiments 1-7, wherein the one or more user inputs include one or more of: approving the tile, rejecting the tile, or scoring the tile. 9. The method of any one of embodiments 1-8, wherein the one or more user inputs include one or more approvals of one or more tiles, and the method further includes determining that the number of the one or more approved tiles reaches a predetermined number, and the analysis result is automatically generated based on the number of the one or more approved tiles reaching the predetermined number. 10. The method of any one of embodiments 1-9, wherein the one or more user inputs include one or more rejections of one or more tiles, and the method further includes selecting one or more additional tiles from the plurality of tiles for review by the user by one or more machine learning models based on one or more criteria related to medical analysis. 11. The method of any one of embodiments 1-10, wherein the analysis results include one or more of a risk score indicating the likelihood of disease recurrence, a risk score indicating the likelihood of resistance to treatment, or a probability indicating the risk of recurrence or refractory disease at a particular time point. 12. The method of any one of embodiments 1-11, further comprising displaying, via a user interface, one or more positions of the selected one or more tiles relative to the tissue, respectively. 13. The method of any one of embodiments 1 to 12, wherein the tissue is stained based on H&E staining, and the method further includes generating a virtual DAB stain for the tissue by one or more machine learning models, and the user interface is operable to adjust the display of each of the one or more selected tiles based on one or more of the H&E staining or the virtual DAB staining. 14. The method of any one of embodiments 1 to 13, further comprising receiving, via a user interface, one or more additional user inputs including one or more of approving the analysis results, adjusting the analysis results, or overwriting the analysis results; generating a medical report based on the one or more additional user inputs; and receiving an approval signature for the medical report. 15. One or more non-transitory computer-readable storage media embodying software operable, when executed, to perform the method of any one of embodiments 1-14. 16. A system comprising one or more processors and a non-transitory memory coupled to the processors and containing instructions executable by the processors, the processors operable, upon execution of the instructions, to perform the method of any one of embodiments 1 to 14.

Claims

1. With the digital pathology image processing system, accessing slide images of tissue for medical analysis; segmenting the slide image into a plurality of tiles; selecting one or more tiles from the plurality of tiles by one or more machine learning models based on one or more criteria related to the medical analysis; displaying the one or more selected tiles for review by a user via a user interface; receiving one or more user inputs regarding the one or more tiles via the user interface; generating an analysis result for the medical analysis based on the one or more user inputs and the one or more tiles; A method comprising:

2. 2. The method of claim 1, further comprising generating a first subset from the plurality of tiles by removing one or more first tiles from the plurality of tiles, each of the one or more first tiles including an artifact, and the one or more tiles selected from the first subset.

3. 3. The method of claim 2, further comprising generating a second subset from the first subset by removing one or more second tiles from the first subset, each of the one or more second tiles corresponding to an area for the medical analysis, and the one or more tiles selected from the second subset.

4. 4. The method of claim 3, wherein the medical analysis includes a tumor analysis, and generating the second subset is based on a tumor detection algorithm, and each tile in the second subset includes a tumor.

5. The method of any one of claims 1 to 4, wherein the one or more criteria include one or more of high salience or high representativeness of a disease targeted by the medical analysis.

6. the tissue is from a patient having a tumor; The method comprises: The method of any one of claims 1 to 5, further comprising generating a segmentation including a nucleus for each of the one or more selected tiles.

7. The method of any one of claims 1 to 6, wherein the medical analysis includes determining one or more of disease recurrence or resistance to treatment.

8. The method of any preceding claim, wherein the one or more user inputs include one or more of: accepting a tile, rejecting a tile, or scoring a tile.

9. The one or more user inputs include one or more approvals of one or more tiles, and the method further comprises: determining whether a number of the one or more tiles that have been accepted reaches a predetermined number; The method of any one of claims 1 to 8, wherein the analysis result is generated automatically based on the number of the one or more tiles that are approved reaching the predetermined number.

10. The one or more user inputs include one or more rejections of one or more tiles, and the method further comprises:

10. The method of claim 1, further comprising selecting one or more additional tiles from the plurality of tiles for review by the user by the one or more machine learning models based on the one or more criteria related to the medical analysis.

11. The method of any one of claims 1 to 10, wherein the analysis results include one or more of a risk score indicating the likelihood of disease recurrence, a risk score indicating the likelihood of resistance to treatment, or a probability indicating the risk of recurrence or refractory disease at a particular time point.

12. The method of any one of claims 1 to 11, further comprising displaying, via the user interface, one or more positions of the one or more selected tiles relative to the organization, respectively.

13. The tissue is stained according to H&E staining; The method comprises:

13. The method of claim 1, further comprising generating a virtual DAB stain for the tissue by the one or more machine learning models, wherein the user interface is operable to adjust the display of each of the one or more selected tiles based on one or more of the H&E stain or the virtual DAB stain.

14. receiving one or more additional user inputs via the user interface, including one or more of: approving the analysis results, adjusting the analysis results, or overriding the analysis results; generating a medical report based on the one or more additional user inputs; receiving a signature approving said medical report; The method of any one of claims 1 to 13, further comprising:

15. One or more non-transitory computer-readable storage media embodying software that, when executed, Access slide images of tissues for medical analysis, Segmenting the slide image into a plurality of tiles; selecting one or more tiles from the plurality of tiles using one or more machine learning models based on one or more criteria related to the medical analysis; displaying the one or more selected tiles for review by a user via a user interface; receiving one or more user inputs regarding the one or more tiles via the user interface; generating an analysis result for the medical analysis based on the one or more user inputs and the one or more tiles; One or more non-transitory computer-readable storage media capable of operating in such a manner.

16. The software, when executed, 16. The medium of claim 15, further operable to generate a first subset from the plurality of tiles by removing one or more first tiles from the plurality of tiles, each of the one or more first tiles including an artifact, and the selected one or more tiles being selected from the first subset.

17. The software, when executed, 17. The medium of claim 16, further operable to generate a second subset from the first subset by removing one or more second tiles from the first subset, each of the one or more second tiles corresponding to an area for the medical analysis, and the selected one or more tiles being selected from the second subset.

18. The medium of any one of claims 15 to 17, wherein the one or more criteria include one or more of high salience or high representativeness of a disease targeted by the medical analysis.

19. one or more processors; and a non-transitory memory coupled to the processors and containing instructions executable by the processors, the processors, upon execution of the instructions, Access slide images of tissues for medical analysis, Segmenting the slide image into a plurality of tiles; selecting one or more tiles from the plurality of tiles using one or more machine learning models based on one or more criteria related to the medical analysis; displaying the one or more selected tiles for review by a user via a user interface; receiving one or more user inputs regarding the one or more tiles via the user interface; generating an analysis result for the medical analysis based on the one or more user inputs and the one or more tiles; A system that can operate like this.

20. 20. The system of claim 19, wherein the one or more criteria include one or more of high salience or high representativeness of a disease targeted by the medical analysis.

Images