Diagnostic tool for review of digital pathological images

Abstract

In one embodiment, the invention provides a method comprising: accessing a slide image associated with tissue for medical analysis; segmenting the glass slide image into a plurality of image blocks; selecting, by one or more machine learning models, one or more tiles from the plurality of tiles based on one or more criteria associated with the medical analysis; displaying, via a user interface, the one or more selected tiles for user review; receiving, via the user interface, one or more user inputs associated with the one or more tiles; and generating analysis results for the medical analysis based on the one or more user inputs and the one or more tiles.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 394,928, filed on August 3, 2022, the disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates to systems and methods for improving the efficiency of medical analysis of tissue-based slide images. Background Art

[0004] Pathologists can help perform medical analysis on patients by analyzing slide images of tissues taken from patients. These slide images are usually stained, and pathologists are familiar with the review of specific color schemes for immunohistochemistry (IHC) and histological staining. Pathologists can examine these images, select specific areas of each slide image, perform detailed amplification, and then analyze these areas. In cancer treatment, pathologists may analyze areas based on specific representative areas of tumor tissue during medical analysis of patients with cancer. For example, diffuse large B-cell lymphoma or DLBCL is a cancer that begins in white blood cells called lymphocytes. It usually grows in lymph nodes, i.e., pea-sized glands in the neck, groin, armpits, and other parts that are part of the human immune system. It can also appear in other parts of the body. DLBCL shows a relatively homogeneous pattern that reflects the clonal growth of the disease. In DLBCL, a large part of the diagnosis relies on the analysis of the nucleus (e.g., size, density, content). The analysis results from the pathologist can be provided to the doctor to help determine the treatment. To perform analysis, pathologists typically have to analyze thousands of image tiles, which is time-consuming and inefficient. Summary of the invention

[0005] A system and method are provided herein for improving the efficiency of medical analysis of tissue-based slide images.

[0006] In a particular embodiment, the digital pathology image processing system can improve the efficiency of the patient's medical diagnosis workflow based on a slide image taken from the patient's tissue. The digital pathology image processing system can show the entire tissue image and a gallery of filtered tiles generated from the tissue image to the pathologist via the user interface of the software tool associated with the system. These filtered tiles can be determined by an algorithm, where only high-interest tiles including positive areas for target diagnosis (e.g., tumors) can be displayed. On these tiles, the digital pathology image processing system can further generate segmentation (e.g., a core for tumor diagnosis). The digital pathology image processing system can then provide the pathologist with an option via a software tool to invalidate inappropriate tiles (e.g., with artifacts or with incorrect segmentation). If the pathologist invalidates the tile, the digital pathology image processing system can propose another tile. Once the pathologist reviews and marks (e.g., for approval or invalidation) a set of tiles via a software tool, the digital pathology image processing system can further generate analysis results (e.g., inferring a risk score associated with the patient's health to predict the recurrence of the disease or resistance to treatment). The analysis results can be further presented to the pathologist via the software tool for review. Once the pathologist's approval of the analysis results is obtained, the digital pathology image processing system can further generate a report including the analysis results. The report can include a prompt or request for the pathologist to sign, after which the report can be sent to other interested parties (e.g., clinics, hospitals, doctors, etc.) to help determine a treatment plan suitable for the patient.

[0007] In a particular embodiment, the digital pathology image processing system may access a slide image associated with the tissue for medical analysis. The digital pathology image processing system may then segment the slide image into a plurality of tiles. The digital pathology image processing system may then select one or more tiles from the plurality of tiles based on one or more criteria associated with the medical analysis through one or more machine learning models. In a particular embodiment, the digital pathology image processing system may display one or more selected tiles via a user interface for user review. The digital pathology image processing system may then receive one or more user inputs associated with the one or more tiles via the user interface. The digital pathology image processing system may further generate analysis results for medical analysis based on the one or more user inputs and the one or more tiles. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1

[0046] Figure 2 shows a network of interacting computer systems that may be used in accordance with some embodiments of the present disclosure, as described herein.

[0009] Figure 2 An example method for facilitating tile review for medical analysis is shown.

[0010] Figure 3 An example workflow for determining treatment for a patient is shown.

[0011] Figure 4 An example initial interface of a software tool for reviewing tiles is shown.

[0012] Figure 5 An example zoom-in of a tile is shown.

[0013] Figure 6 An example tile being reviewed by a pathologist is shown.

[0014] Figure 7 Another example tile is shown being reviewed by a pathologist.

[0015] Figure 8 An example approval of a tile by a pathologist is shown.

[0016] Fig. 9 An example rejection of a tile by a pathologist is shown.

[0017] Fig.10 An example of reaching the minimum number of approved tiles is shown.

[0018] Fig.11 An example report generated by a digital pathology image processing system is shown.

[0019] Fig.12 An example of a computing system is shown. DETAILED DESCRIPTION

[0020] Figure 1 A network 100 of interacting computer systems is shown that may be used in accordance with some embodiments of the present disclosure, as described herein.

[0021] The digital pathology image generation system 120 can generate one or more whole slide images or other related digital pathology images corresponding to a particular sample. For example, the image generated by the digital pathology image generation system 120 can include a stained section of a biopsy sample. For another example, the image generated by the digital pathology image generation system 120 can include a slide image of a liquid sample (e.g., a blood smear). For another example, the image generated by the digital pathology image generation system 120 can include a fluorescent micrograph, such as a slide image depicting fluorescent in situ hybridization (FISH) after a fluorescent probe has bound to a target DNA or RNA sequence.

[0022] Some types of samples (e.g., biopsy tissue, solid sample and / or sample including tissue) can be processed by sample preparation system 121 to fix and / or embed the sample. Sample preparation system 121 can facilitate infiltration of sample with fixative (e.g., liquid fixative, such as formaldehyde solution) and / or embedding material (e.g., histological wax). For example, the sample fixing subsystem can fix the sample by exposing the sample to the fixative for at least a threshold amount of time (e.g., at least 3 hours, at least 6 hours or at least 13 hours). The dehydration subsystem can dehydrate the sample (e.g., by exposing a portion of the fixed sample and / or the fixed sample to one or more ethanol solutions) and may use a clearing intermediate (e.g., which includes ethanol and histological wax) to remove the dehydrated sample. The sample embedding subsystem can infiltrate the sample with heated (e.g., so that it is liquid) histological paraffin (e.g., once or multiple times for a corresponding predefined time period). Histological wax can include paraffin and possible one or more resins (e.g., styrene or polyethylene). The sample and wax can then be cooled, and the sample infiltrated with wax can then be sealed.

[0023] The sample slicer 122 can receive a fixed and embedded sample and can produce a set of slices. The sample slicer 122 can expose the fixed and embedded sample to a cool or cold temperature. The sample slicer 122 can then cut the cooled sample (or its trimmed version) to produce a set of slices. Each slice can have a thickness of (for example) less than 100 μm, less than 50 μm, less than 10 μm or less than 5 μm. Each slice can have a thickness of (for example) greater than 0.1 μm, greater than 1 μm, greater than 2 μm or greater than 4 μm. The cutting of the cooled sample can be carried out in a warm water bath (for example, at a temperature of at least 30° C., at least 35° C. or at least 40° C.).

[0024] Automated staining system 123 can promote the staining of one or more sample sections by exposing each section to one or more staining agents. Each section can be exposed to a predefined volume of staining agent for a predefined time period. In some cases, a single section is exposed to a variety of staining agents simultaneously or sequentially.

[0025] Each of one or more stained sections can be presented to an image scanner 124, which can capture a digital image of the section. The image scanner 124 can include a microscope camera. The image scanner 124 can capture digital images at multiple magnification levels (e.g., using a 10x objective lens, a 20x objective lens, a 40x objective lens, etc.). The manipulation of the image can be used to capture the selected portion of the sample within the desired magnification range. The image scanner 124 can further capture annotations and / or morphological measurements identified by a human operator. In some cases, after capturing one or more images, the section is returned to the automated staining system 123 so that the section can be cleaned, exposed to one or more other stains and imaged again. When using a variety of stains, stains can be selected to have different color profiles, so that the first region of the image corresponding to the first section portion absorbing a large amount of the first stain can be distinguished from the second region of the image (or different images) corresponding to the second section portion absorbing a large amount of the second stain.

[0026] It should be understood that in some cases, one or more components of the digital pathology image generation system 120 may be operated in conjunction with a human operator. For example, a human operator may move samples across various subsystems (e.g., the sample preparation system 121 or subsystems of the digital pathology image generation system 120) and / or initiate or terminate the operation of one or more subsystems, systems, or components of the digital pathology image generation system 120. As another example, a portion or all of one or more components of the digital pathology image generation system (e.g., one or more subsystems of the sample preparation system 121) may be partially or completely replaced with actions of a human operator.

[0027] In addition, it should be understood that while the various described and depicted functions and components of the digital pathology image generation system 120 relate to the processing of solid and / or biopsy samples, other embodiments may relate to liquid samples (e.g., blood samples). For example, the digital pathology image generation system 120 can receive a liquid sample (e.g., blood or urine) slide that includes a base slide, a smeared liquid sample, and a coverslip. The image scanner 124 can then capture an image of the sample slide. Other embodiments of the digital pathology image generation system 120 may involve capturing images of the sample using advanced imaging techniques such as FISH described herein. For example, once a fluorescent probe has been introduced into the sample and bound to a target sequence, an image of the sample can be captured using appropriate imaging for further analysis.

[0028] A given sample may be associated with one or more users (e.g., one or more physicians, laboratory technicians, and / or medical providers) during processing and imaging. The associated providers may include, for example, but not limited to, a person who orders a test or biopsy that produces an imaged sample, a person who is entitled to receive the results of a test or biopsy, or a person who performs an analysis of a test or biopsy sample, etc. For example, a user may correspond to a physician, a pathologist, a clinician, or a subject. A user may use one or more user devices 130 to submit one or more requests (e.g., identifying a subject) to process a sample by a digital pathology image generation system 120 and to process the resulting image by a digital pathology image processing system 110.

[0029] The digital pathology image generation system 120 can transmit the image generated by the image scanner 124 back to the user device 130. The user device 130 then communicates with the digital pathology image processing system 110 to initiate automated processing of the image. In some cases, the digital pathology image generation system 120 provides the image generated by the image scanner 124 directly to the digital pathology image processing system 110, for example, at the instruction of the user of the user device 130. Although not shown, other intermediate devices (e.g., data storage areas connected to the digital pathology image generation system 120 or a server of the digital pathology image processing system 110) may also be used. In addition, for simplicity, only one digital pathology image processing system 110, image generation system 120, and user device 130 are shown in the network 100. The present disclosure contemplates the use of one or more of each type of system and its components without departing from the teachings of the present disclosure.

[0030] Figure 1The network 100 and associated systems shown in the can be used in various environments where scanning and evaluating digital pathology images (such as full slide images) are important components of the work. As an example, the network 100 can be associated with a clinical environment in which a user evaluates a sample for possible diagnostic purposes. The user can use a user device 130 to review the image, and then provide the image to the digital pathology image processing system 110. The user can provide additional information to the digital pathology image processing system 110, which can be used to guide or instruct the digital pathology image processing system 110 to analyze the image. For example, the user can provide an expected diagnosis or preliminary assessment of the features within the scan. The user can also provide additional background, such as the type of tissue being reviewed. As another example, the network 100 can be associated with a laboratory environment in which the tissue is being examined, for example, to determine the efficacy or potential side effects of a drug. In this case, it may be common for multiple types of tissues to be submitted for review to determine the effects of the drug on the whole body. This may pose a special challenge to human scan reviewers, who may need to determine the various backgrounds of the image, which may be highly dependent on the type of tissue being imaged. These scenarios can be optionally provided to the digital pathology image processing system 110.

[0031] The digital pathology image processing system 110 can process digital pathology images, including whole slide images, to classify digital pathology images and generate annotations for digital pathology images and related outputs. As an example, the digital pathology image processing system 110 can process a whole slide image of a tissue sample or a tile of a whole slide image of a tissue sample generated by the digital pathology image processing system 110 to identify a tumor region. As another example, the digital pathology image processing system 110 can process tiles of a whole slide image of a tissue sample to identify tiles with a high concern value or a high risk score associated with a medical analysis. The digital pathology image processing system 110 can crop a query image into a plurality of image tiles. The tile generation module 111 can define a set of tiles for each digital pathology image. In order to define a set of tiles, the tile generation module 111 can segment the digital pathology image into a set of tiles. As embodied herein, tiles may be non-overlapping (e.g., each tile includes pixels of an image that are not included in any other tile) or overlapping (e.g., each tile includes a portion of pixels of an image that are included in at least one other tile). In addition to the size of each tile and the stride of the window (e.g., the image distance or pixels between a tile and a subsequent tile), features such as whether tiles overlap may also increase or decrease the data set for analysis, where more tiles (e.g., through overlapping or smaller tiles) increase the potential resolution of the final output and visualization. In some cases, the tile generation module 111 defines a set of tiles for an image, where each tile has a predefined size and / or the offset between tiles is predefined. Continuing with the example of detecting gene fusions, each slide image may be cropped into image tiles with a width and height of a certain number of pixels. In addition, the tile generation module 111 may create multiple sets of tiles with different sizes, overlaps, strides, etc. for each image. As an example, the width and height in pixels may be dynamically determined (i.e., not fixed) based on factors such as the assessment task, the query image itself, or any suitable factors. In some embodiments, the digital pathology image itself may contain block overlap, which may be caused by the imaging technology. Average segmentation without block overlap may be a preferred solution to balance block processing requirements and avoid affecting the embedding generation and weighted value generation discussed herein. The block size or block offset may be determined, for example, by calculating one or more performance metrics (e.g., precision, recall, accuracy, and / or error) for each size / offset, and by selecting a block size and / or offset associated with one or more performance metrics above a predetermined threshold and / or associated with one or more performance metrics (e.g., high precision, high recall, high accuracy, and / or low error).

[0032] The block generation module 111 may also define the block size depending on the type of abnormality detected. For example, the block generation module 111 may be configured to understand the type of tissue abnormality that the digital pathology image processing system 110 will search for and may customize the block size based on the tissue abnormality to improve detection. For example, the image generation module 111 may determine that when the tissue abnormality includes searching for inflammation or necrosis in lung tissue, the block size should be reduced to increase the scan rate, and when the tissue abnormality includes an abnormality of Kupffer cells in liver tissue, the block size should be increased to increase the chance of the digital pathology image processing system 110 analyzing Kupffer cells as a whole. In some cases, the tile generation module 111 defines a set of tiles, wherein the number of tiles in the set for each image, the size of the tiles in the set, the resolution of the tiles in the set, or other related attributes are defined and kept constant for each of one or more images.

[0033] As embodied herein, the tile generation module 111 may further define a set of tiles for each digital pathology image along one or more color channels or color combinations. As an example, the digital pathology images received by the digital pathology image processing system 110 may include large-format multi-color channel images having pixel color values ​​for each pixel of the image specified in one of several color channels. Exemplary color specifications or color spaces that may be used include RGB, CMYK, HSL, HSV, or HSB color specifications. A set of tiles may be defined based on segmenting color channels and / or generating a brightness map or grayscale map for each tile. For example, for each fragment of an image, the block generation module 111 may provide a red block, a blue block, a green block, and / or a brightness block, or an equivalent of the color specification used. As explained herein, segmenting digital pathology images based on fragments of an image and / or color values ​​of a fragment may improve the accuracy and recognition rate of a network used to generate embeddings of tiles and images and produce image classification. In addition, the digital pathology image processing system 110, for example, using the tile generation module 111, may convert between color specifications and / or prepare copies of tiles using multiple color specifications. The color specification conversion can be selected based on the desired type of image enhancement (e.g., strengthening or enhancing a specific color channel, saturation, brightness level, etc.). The color specification conversion can also be selected to improve the compatibility between the digital pathology image generation system 120 and the digital pathology image processing system 110. For example, a specific image scanning component can provide an output of HSL color specifications, and as described herein, the model used in the digital pathology image processing system 110 can be trained using RGB images. Converting the tiles to compatible color specifications can ensure that the tiles can still be analyzed. In addition, the digital pathology image processing system can upsample or downsample images provided at a specific color depth (e.g., 8 bits, 1 bit, etc.) for use by the digital pathology image processing system. In addition, the digital pathology image processing system 110 can convert tiles according to the type of image that has been captured (e.g., a fluorescent image can include more details about color intensity or a wider range of colors).

[0034] As described herein, the tile embedding module 112 can generate an embedding for each tile in a corresponding feature embedding space. The embedding can be represented by the digital pathology image processing system 110 as a feature vector of the tile. The tile embedding module 112 can use a neural network (e.g., a convolutional neural network) to generate a feature vector for each tile representing the image. In a particular embodiment, the block embedding neural network can be based on a ResNet image network trained on a dataset based on natural (e.g., non-medical) images (such as the ImageNet dataset). By using a non-specialized block embedding network, the block embedding module 112 can take advantage of known advances in efficiently processing images to generate embeddings. In addition, using a natural image dataset allows the embedding neural network to learn to discern differences between tile fragments at a holistic level.

[0035] In other embodiments, the tile embedding network used by the tile embedding module 112 can be an embedding network customized to handle large format images, such as a large number of tiles of a digital pathology whole slide image. In addition, a custom data set can be used to train the block embedding network used by the block embedding module 112. For example, the block embedding network can be trained using various samples of whole slide images, or even trained using samples related to the subject matter (e.g., scans of a specific tissue type) for which the embedding network will generate embeddings. Using a collection of specialized or customized images to train the tile embedding network can allow the tile embedding network to recognize more subtle differences between tiles, which can make the distances between tiles in the feature embedding space more detailed and accurate, but at the cost of the computational and economic costs of additional time required to acquire images and train multiple tile generation networks for use by the tile embedding module 112. The block embedding module 112 can select from the block embedding network library based on the type of image being processed by the digital pathology image processing system 110.

[0036] As described herein, visual features of a tile can be used to generate a tile embedding from a deep learning neural network. The tile embedding can be further generated based on contextual information associated with the tile or based on the content displayed in the tile. For example, the tile embedding can include one or more features that indicate and / or correspond to the size of the depicted object (e.g., the size of the depicted cell or aberration) and / or the density of the depicted object (e.g., the density of the depicted cell or aberration). The size and density can be measured absolutely (e.g., width expressed in pixels or converted from pixels to nanometers) or relative to other tiles from the same digital pathology image, from a class of digital pathology images (e.g., generated using similar technology or by a single digital pathology image generation system or scanner), or from a related series of digital pathology images. In addition, the tile can be classified before the tile embedding module 112 generates an embedding for the tile, so that the tile embedding module 112 takes the classification into account when preparing the embedding.

[0037] For consistency, the block embedding module 112 can generate embeddings of predefined sizes (e.g., vectors of 512 elements, vectors of 2048 bytes, etc.). The block embedding module 112 can also generate embeddings of various and arbitrary sizes. The block embedding module 112 can adjust the size of the embedding based on user direction, or can make a selection based on, for example, computational efficiency, accuracy, or other parameters. In certain embodiments, the embedding size can be based on the limitations or specifications of the deep learning neural network that generates the embedding. Larger embedding sizes can be used to increase the amount of information captured in the embedding and improve the quality and accuracy of the results, while smaller embedding sizes can be used to improve computational efficiency.

[0038] The digital pathology image processing system 110 can perform different inferences by applying one or more machine learning models to the embedding, that is, inputting the embedding to the machine learning model. As an example, the digital pathology image processing system 110 can identify tumor areas based on a machine learning model trained to identify tumor areas. As another example, the digital pathology image processing system 110 can identify high-interest or high-risk tiles based on a machine learning model trained to identify high-interest or high-risk tiles. In some embodiments, it may not be necessary to crop the image into image tiles, generate embeddings for these tiles, and then perform inference based on such embeddings. Instead, the digital pathology image processing system 110 can apply the machine learning model directly to the embedding of the whole slide image to perform inference with sufficient GPU memory. The output of the machine learning model can be adjusted to the shape of the input image.

[0039] The whole slide image access module 113 can manage requests for access to whole slide images from other modules of the digital pathology image processing system 110 and the user device 130. For example, the whole slide image access module 113 receives a request to identify a whole slide image based on a specific block, an identifier of a block, or an identifier of a whole slide image. The whole slide image access module 113 can perform the following tasks: confirm that the whole slide image is available to the requesting user, identify the appropriate database from which to retrieve the requested whole slide image, and retrieve any additional metadata that may be of interest to the requesting user or module. In addition, the whole slide image access module 113 can efficiently handle streaming the appropriate data to the requesting device. As described herein, the whole slide image can be provided to the user device in the form of blocks based on the likelihood that the user will want to see portions of the whole slide image. The whole slide image access module 113 can determine which areas of the whole slide image to provide and how to provide these areas. Furthermore, the whole slide image access module 113 may be authorized within the digital pathology image processing system 110 to ensure that no individual component locks or otherwise misuses the database or whole slide images to the detriment of other components or users.

[0040] The output generation module 114 of the digital pathology image processing system 110 can generate outputs corresponding to result blocks and result full slide image data sets based on user requests. As described herein, the output can include various visualizations, interactive graphics, and reports based on the type of request and the type of data available. In many embodiments, the output will be provided to the user device 130 for display, but in some embodiments, the output can be accessed directly from the digital pathology image processing system 110. The output will be based on the presence and access of appropriate data, so the output generation module will be authorized to access necessary metadata and anonymous patient information as needed. Like other modules of the digital pathology image processing system 110, the output generation module 114 can be updated and improved in a modular manner so that new output features can be provided to users without a lot of downtime.

[0041] The general techniques described herein can be integrated into various tools and use cases. For example, as described, a user (e.g., a pathologist or clinician) can access a user device 130 communicating with a digital pathology image processing system 110 and provide a query image for analysis. The digital pathology image processing system 110 or a connection to the digital pathology image processing system can be provided as an independent software tool or package that searches for corresponding matches, identifies similar features, and generates appropriate outputs for users upon request. As an independent tool or plug-in that can be purchased or licensed on a streamlined basis, the tool can be used to enhance the capabilities of a research or clinical laboratory. In addition, the tool can be integrated into the services available to customers of a digital pathology image generation system. For example, the tool can be provided as a unified workflow, wherein a user who performs or requests to automatically create a full slide image receives a report of noteworthy features in an image and / or a similar full slide image that has been previously indexed. Therefore, in addition to improving full slide image analysis, these techniques can also be integrated into existing systems to provide additional functionality that was not previously considered or possible.

[0042] In addition, the digital pathology image processing system 110 can be trained and customized for specific settings. For example, the digital pathology image processing system 110 can be specifically trained to provide insights related to specific types of tissues (e.g., lungs, heart, blood, liver, etc.). As another example, the digital pathology image processing system 110 can be trained to assist in safety assessments, such as determining the level or degree of toxicity associated with a drug or other potential treatment. Once trained for a specific subject or use case, the digital pathology image processing system 110 is not necessarily limited to that use case. Due to the relatively large collection of at least partially labeled or annotated images, training can be performed in specific scenarios, such as toxicity assessments.

[0043] Figure 2An example method 200 for facilitating tile review for medical analysis is shown. The method may begin at step 210, where a digital pathology image processing system 110 may access a slide image associated with a tissue for medical analysis. In a particular embodiment, such a slide image may correspond to a stained slide of tissue taken from a patient. The reason for staining the slide may be that the detection of an object may require a color pattern familiar to a pathologist to improve efficiency. As an example and not by way of limitation, the slide may be stained with hematoxylin and eosin (H&E). The stained slide may then be digitized (e.g., scanned) to generate a slide image.

[0044] At step 220, the digital pathology image processing system 110 can segment the slide image into a plurality of tiles. In a particular embodiment, a tile generation module 111 can be used to generate tiles. Tiles can be non-overlapping or overlapping. In addition to the size of each tile and the stride of the window, features such as whether the tiles overlap can also increase or decrease the data set for analysis, where more tiles increase the potential resolution of the final output and visualization. In a particular embodiment, each tile can have a predefined size and / or the offset between tiles can be predefined. In addition, the tile generation module 111 can create multiple sets of tiles with different sizes, overlaps, strides, etc. for each image. The tile generation module 111 can generate tiles for each digital pathology image along one or more color channels or color combinations. Tiles can be generated based on segmenting color channels and / or generating brightness maps or grayscale equivalents for each tile. In addition, the digital pathology image processing system 110 can upsample or downsample images provided at a specific color depth to be available for use by the digital pathology image processing system 110. Additionally, the digital pathology image processing system 110 may cause the tiles to be converted depending on the type of image that has been captured.

[0045] At step 230, the digital pathology image processing system 110 may select one or more tiles from a plurality of tiles based on one or more criteria associated with the medical analysis through one or more machine learning models. In a particular embodiment, the digital pathology image processing system 110 may select these tiles as follows. The digital pathology image processing system 110 may filter out tiles of artifacts using a quality check (QC) algorithm. The digital pathology image processing system 110 may then use a conditional detection algorithm to filter out tiles corresponding to normal areas (e.g., if the medical analysis includes tumor analysis, non-tumor tiles may be considered "abnormal" and therefore filtered). The digital pathology image processing system 110 may then use another algorithm to pre-select tiles of interest based on different criteria. In a particular embodiment, one or more criteria may include one or more of a high concern value or a high representativeness of a disease targeted by the medical analysis.

[0046] At step 240, the digital pathology image processing system 110 may display one or more selected tiles for user review via a user interface. In a particular embodiment, the user interface may display a separate tile or tile cluster to a pathologist. The initial interface may provide an overview of the slide, a tile review of the slide, and a slide navigator. As an example, the tile review portion may include tiles and the pathologist may scroll down to review other tiles. The digital pathology image processing system 110 may display the position of the selected tiles relative to the tissue via a user interface, respectively. In addition, the user interface may provide different visualization options. As an example and not by way of limitation, the user interface may enable a pathologist to review tiles under different stains. The user interface may be operable to adjust the display of each tile in one or more selected tiles based on one or more of H&E staining or virtual staining artificially generated by the digital pathology image processing system 110. For example, a tile may appear as if it is not stained with H&E, but with DAB stain (i.e., 3,3'-diaminobenzidine is oxidized by hydrogen peroxide in a reaction usually catalyzed by horseradish peroxidase (HRP)). The user interface may also blend or overlay tiles. Additionally, the user interface may provide channels that can be activated / deactivated by the pathologist.

[0047] At step 250, the digital pathology image processing system 110 may receive one or more user inputs associated with one or more tiles via a user interface. In a particular embodiment, the one or more user inputs include one or more of approval of a tile, rejection of a tile, or a rating of a tile. The pathologist may approve or reject each tile based on a verified input (e.g., a correctly segmented tumor nucleus). For each rejected tile, the digital pathology image processing system 110 may replace it with another pre-selected tile (e.g., with a high attention value or high representativeness) by an algorithm in order to maintain an optimal number of tiles. In an alternative embodiment, the pathologist may provide a rating for each tile instead of simply approving or rejecting it. Before the review process begins, specific instructions with strict criteria for when to approve or reject a tile may be provided to the pathologist.

[0048] At step 260, the digital pathology image processing system 110 may generate analysis results for medical analysis based on one or more user inputs and one or more tiles. Generation of analysis results may be automatically triggered in response to determining that the number of approved tiles has reached a predetermined number. In other words, once a minimum number of tiles is reached, a statistical analysis (e.g., calculation of a risk score) may be triggered. The digital pathology image processing system 110 may perform statistical analysis based on the tiles approved by the pathologist. Alternatively, if the pathologist provides a score to the tile instead of approving / rejecting it, the digital pathology image processing system 110 may perform statistical analysis using the score of the tile. In particular embodiments, the analysis results may include one or more of: a risk score indicating the likelihood of recurrence of the disease, a risk score indicating the likelihood of resistance to treatment, or a probability indicating the risk of recurrence or refractory disease at a particular time point. Particular embodiments may be repeated where appropriate. Figure 2 Although the present disclosure will Figure 2 The specific steps of the method are described and shown as occurring in a particular order, but the present disclosure contemplates Figure 2 Any suitable steps of the method may occur in any suitable order. In addition, although the present disclosure describes and illustrates example methods for facilitating image tile review for medical analysis, including Figure 2 The present disclosure contemplates any suitable method for facilitating tile review for medical analysis, including any suitable steps, which may include Figure 2 In addition, although the present disclosure describes and illustrates the method of performing Figure 2 specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing Figure 2 Any suitable combination of any suitable components, devices or systems of any suitable steps of the method.

[0049] Medical analysis of slide images associated with tissues may rely on a combination of different factors, such as interest (i.e., is the tile of the slide image important?) and risk score (i.e., is the tile risk high or low?). As an example and not by way of limitation, in DLBCL, it may be challenging to detect tumor regions by an algorithm. The selection may need to be supervised by a pathologist. In addition, the selection of relevant regions (e.g., tiles) may need to allow visualization of the histological context through zoom and pan functions. Although the algorithm can be viewed as a black box, the pathologist may need to sign on the analysis results, indicating that it may be important to keep humans (e.g., pathologists) informed to increase the confidence of the analysis. In view of the aforementioned factors that are critical to the effective analysis of slide images, the embodiments disclosed herein have developed a digital pathology image processing system 110 that integrates machine learning and human expertise for selecting tiles and then generating analysis results based on the selected tiles.

[0050] Figure 3 An example workflow 300 for determining treatment for a patient is shown. After the patient's first visit, the patient can enter the clinical trial screening at step 310. In a particular embodiment, the medical analysis can include determining one or more of the recurrence of the disease or resistance to treatment. At step 320, a request for certain tests associated with the patient's disease (e.g., a tumor) can then be sent to the laboratory to send the stained slides taken from the patient's tissue to a clinical research organization (CRO). The reason for staining the slide may be because the detection of the object may require a color pattern familiar to the pathologist to improve efficiency. As an example and not by way of limitation, the slide can be H&E stained. At step 330, the pathologist can select a reference slide. At step 340, the slide can be transported to the CRO together with the reference slide. At step 350, the technician can push or scan the non-digital slide to digitize it, that is, generate a slide image.

[0051] At step 360, the digital pathology image processing system 110 may store the slide image in memory and then analyze it using artificial intelligence (AI) and machine learning while keeping the pathologist informed based on the following sub-steps. In certain embodiments, the machine learning model may be based on a neural network. At sub-step 360a, the pathologist may begin analysis on a selected slide that is representative (e.g., displays a major histological pattern) for the medical case (e.g., tumor) being analyzed. The digital pathology image processing system 110 may prepare the slide image accordingly.

[0052] At sub-step 360b, the digital pathology image processing system 110 may verify the input. In a particular embodiment, the digital pathology image processing system 110 may generate a first subset from a plurality of tiles by filtering out one or more first tiles from the plurality of tiles. Each of the one or more first tiles may include an artifact and the selected one or more tiles may be selected from the first subset. Specifically, the digital pathology image processing system 110 may use a quality check (QC) algorithm to filter out artifacts of the slide image. In a particular embodiment, the digital pathology image processing system 110 may generate a second subset from the first subset by filtering out one or more second tiles from the first subset. Each of the one or more second tiles may correspond to an area for medical analysis and the selected one or more tiles may be selected from the second subset. Specifically, the digital pathology image processing system 110 may use a conditional detection algorithm to filter out normal tiles. As an example and not by way of limitation, if the medical analysis includes tumor analysis, generating the second subset may be based on a tumor detection algorithm and each of the tiles in the second subset may include a tumor.

[0053] The digital pathology image processing system 110 may then use another algorithm to preselect tiles of interest based on different criteria. In a particular embodiment, one or more criteria may include one or more of a high attention value or a high representativeness of a disease targeted by medical analysis. As an example and not by way of limitation, such an algorithm may be based on attention values ​​associated with tiles, respectively. In a particular embodiment, such an algorithm may determine that tiles with high attention values ​​are the most influential and their patterns are the most relevant. For example, the preselected tiles may all have high attention values. More information about high attention learning can be found in U.S. Patent Application No. 63 / 108659 filed on November 2, 2020, the entire contents of which are incorporated by reference. As another example and not by way of limitation, the preselection of tiles may be based on the representativeness of the disease of interest (e.g., a tumor). For example, the preselected tiles may all be representative. If the tissue is associated with a patient with a tumor, the digital pathology image processing system 110 may further generate a segmentation including a nucleus for each selected tile in the selected tiles. In a particular embodiment, the algorithm for preselection may be trained based on the experience of an expert. As an example and not by way of limitation, an expert may be asked to annotate multiple images indicating what are the most representative regions (e.g., tiles) for a patient, and then the algorithm may be trained to find those exact regions. The pathologist may then perform a quality check on these tiles pre-selected by the algorithm. In a particular embodiment, the digital pathology image processing system 110 may generate a user interface via a software tool to allow the pathologist to easily perform the quality check. In addition to the pathologist, any interested party may view / access the user interface in accordance with the clinical disclosure agreement.

[0054] In a particular embodiment, the user interface can display a separate tile or a cluster of tiles to the pathologist. The pathologist can review each tile and provide user input. In a particular embodiment, one or more user inputs include one or more of approval of the tile, rejection of the tile, or scoring of the tile. The pathologist can approve or reject each tile based on verified input (e.g., a correctly segmented tumor nucleus). In an alternative embodiment, the pathologist can provide a score for each tile instead of simply approving or rejecting it. Before the review process begins, specific instructions with strict criteria for when to approve or reject the tile can be provided to the pathologist. In an alternative embodiment, the digital pathology image processing system 110 can further provide the pathologist with annotations for each tile, which can help the pathologist's review of the tile. The annotations can be generated by an algorithm or by another pathologist.

[0055] For each approved tile, the digital pathology image processing system 110 can provide a visualization that can help the pathologist better review it. As an example and not by way of limitation, if the tile is associated with a tumor, the digital pathology image processing system 110 can display the nucleus in orange (e.g., DAB stain), or red stain, or blue stain, and a marker based on the habit of the pathologist. If one or more user inputs include one or more rejections of one or more tiles, the digital pathology image processing system 110 can further select one or more additional tiles from a plurality of tiles for user review based on one or more criteria associated with medical analysis through one or more machine learning models. In other words, for each rejected tile, the digital pathology image processing system 110 can replace it with another pre-selected tile (e.g., with a high attention value or high representativeness) by an algorithm in order to maintain an optimal number (e.g., 50) of tiles. In an alternative embodiment, the digital pathology image processing system may not replace the rejected tiles. On the contrary, there may be an excess of tiles available for pre-selection. The digital pathology image processing system can continue to pre-select from the excess tiles for pathologist review until the tiles approved by the pathologist are sufficient to generate analysis results. In a specific embodiment, if one or more user inputs include one or more approvals for one or more tiles, the digital pathology image processing system 110 can further determine that the number of one or more approved tiles reaches a predetermined number. The generation of analysis results can be automatically triggered in response to determining that the number of one or more approved tiles reaches a predetermined number. In other words, once the minimum number of tiles is reached, statistical analysis (e.g., calculation of risk scores) can be triggered. In an alternative embodiment, the generation of analysis results can be automatically triggered in response to determining that a specific area of ​​the pre-selected tiles has been reached. As an example and not by way of limitation, there may be more important specific areas. Digital pathology image processing can pre-select other areas for pathologist review at the beginning. But the process can continue until the digital pathology image processing system pre-selects the specific area for pathologist review and the pre-selection is approved. The digital pathology image processing system 110 can perform statistical analysis based on the tiles approved by the pathologist. Alternatively, if the pathologist provides a score to the tile instead of approving / rejecting it, the digital pathology image processing system 110 can use the score of the tile to perform statistical analysis. In certain embodiments, the approve / reject workflow can minimize clicks by the pathologist, thereby increasing the speed of the pathologist's review.

[0056] The digital pathology image processing system 110 can then output the statistical analysis via a software tool to present it to a pathologist for additional review. In certain embodiments, the analysis results can include one or more of the following: a risk score indicating the likelihood of recurrence of the disease, a risk score indicating the likelihood of resistance to treatment, or a probability indicating the risk of recurrence or refractory at a specific time point.

[0057] In a particular embodiment, the digital pathology image processing system 110 can receive one or more additional user inputs via a user interface, and these additional user inputs include one or more of the approval of the analysis result, the adjustment of the analysis result, or the overwriting of the analysis result. Specifically, the pathologist can approve the statistical analysis (e.g., risk score), which can trigger the generation of the report. Or, the pathologist can adjust the final output, such as adjusting the risk score, and may even overwrite the final output. Then, the digital pathology image processing system 110 can generate a medical report based on one or more additional user inputs. At substep 360c, the digital pathology image processing system 110 can present the generated report to the pathologist for review. As an example and not by way of limitation, the pathologist can review the report and approve the final report by checking each element (check box). Subsequently, the digital pathology image processing system 110 can receive the signing of the medical report. At substep 360d, the digital pathology image processing system 110 can issue an electronic report to the pathologist so that the pathologist can sign the report.

[0058] At step 370, the report including the analysis results can be stored in the trial database for easy access by interested parties. At step 380, the patient can be re-visited. At step 390, the patient can be treated, which can be determined based on the report including the analysis results.

[0059] Figure 4An example initial interface of a software tool for reviewing a block is shown. The initial interface can provide an overview 410 of a slide, a block review portion 420 of a slide, and a slide navigator 430. The initial interface can also provide different viewing options 440. As an example and not by way of limitation, these viewing options 440 can include panning, zooming, ICC configuration, intensity, photo mode, and case information (info). The initial interface can additionally provide different annotation options 450. As an example and not by way of limitation, these annotation options 440 can include flexpoly, rectangle, and counter. Flexpoly can allow freehand drawing of annotations to create any shape required by the user. Rectangles can allow annotations to be drawn in the form of rectangles of different sizes. Counters can allow any object to be annotated by placing a mark (e.g., a colored point) in its center, wherein each point can be marked according to the user's desired needs. The initial interface can further provide different editing options 460. As an example and not by way of limitation, these editing options 460 can include selection, editing shape, rotation, and transformation. If it is necessary to correct or change an existing annotation, the user can use these editing options 460. The modification or change may include modification of the size, scale, and position of the existing annotations. In certain embodiments, the initial interface may display a slide identifier (ID) 470 and annotations 480. In certain embodiments, the slide 490 may be stained. As an example and not by way of limitation, Figure 4 The slide 490 in the embodiment may be stained with H&E. Figure 4 As shown, the tile review section 420 may include tiles and the pathologist may scroll down to review other tiles except tile 1 420a, tile 2 420b, and tile 3 420c. The pathologist may reject or approve the tiles. The minimum number of tiles to be approved may be 50 and no tiles have been approved at this time (i.e., 0 / 50).

[0060] Figure 5 An example zoom-in of a tile is shown. Figure 5 As shown, the pathologist may be reviewing tile 1 420a. In certain embodiments, the digital pathology image processing system 110 may display one or more selected tiles relative to one or more locations of the tissue via the user interface. Figure 5As shown, the slide navigator 430 can indicate the position of the tile 420a relative to the whole slide image. In addition, the software tool can provide different visualization options. In a particular embodiment, the digital pathology image processing system 110 can artificially generate a virtual stain (e.g., a virtual DAB stain) associated with the tissue through one or more machine learning models based on information derived from the underlying segmentation results. If the tissue was originally stained based on H&E staining, the user interface can be operable to adjust the display of each selected tile in one or more selected tiles based on one or more of H&E staining or virtual DAB staining. As an example and not by way of limitation, if all tumor nuclei are detected and segmented by a machine learning model, the results of the segmentation can be displayed in the form of a virtual DAB stain using a slider function in the user interface. It may not require registration or overlap of a real DAB stain. As Figure 5 As shown, the left portion of the tile 420a being reviewed can display a virtual DAB stain, while the right portion of the tile 420a being reviewed can display the original H&E stain. Using separator 510, the lesion can be easily slid left and right to review the same portion of the tile 420a under different stains. The software tool can also merge or overlay the tile 420a. In addition, the software tool can also provide channels that can be activated / deactivated by the pathologist. As an example and not by way of limitation, using channels, the software tool can create three different channel positions.

[0061] Figure 6 An example tile is shown being reviewed by a pathologist. Figure 6 As shown, the pathologist may be reviewing tile 2 420b. The slide navigator 430 may indicate the location of this tile 420b relative to the full slide image.

[0062] Figure 7 Another example tile is shown being reviewed by a pathologist. Figure 7 As shown, the pathologist may be reviewing tile 3 420c. The slide navigator 430 may indicate the location of this tile 420c relative to the full slide image.

[0063] Figure 8 An example approval of a block by a pathologist is shown. Figure 8 As shown, the pathologist may have approved tile 1 420a. Therefore, 1 tile out of 50 may now be approved. The pathologist may revoke the approval. The pathologist may now be reviewing tile 2 420b. The slide navigator 430 may indicate the location of tile 2 420b relative to the full slide image.

[0064] Fig. 9An example rejection of a tile by a pathologist is shown. Fig. 9 As shown, the pathologist may have rejected Tile 2 420b. The pathologist may revoke the rejection. The pathologist may now be reviewing Tile 3 420c. The slide navigator 430 may indicate the location of Tile 3 420c relative to the full slide image. Due to the rejection of Tile 2 420b, the digital pathology image processing system 110 may replace the rejected Tile 2 420b with another tile for the pathologist to review.

[0065] Fig.10 An example of reaching the minimum number of approved tiles is shown. Fig.10 As shown, 50 of the required 50 tiles may now be in the process of being approved. Approval of the minimum number of tiles may trigger a statistical analysis by the digital pathology image processing system 110. As a result, the pathologist may select "Show Results" 1010 to view the results of the analysis.

[0066] Fig.11 An example report 1100 generated by a digital pathology image processing system 110 is shown. As an example and not by way of limitation, the report 1100 may include patient information 1110 such as patient name, gender, date of birth, report date, sample source / identifier (ID), and physician. The report may also include a risk score 1120 (e.g., 60) and a risk of relapse / refractory at 24 months (e.g., 80%) 1130. The software tool may enable a pathologist to confirm the histology of DLBCL 1140. For example, the pathologist may reject or approve that the histology is confirmed. The software tool may also enable the pathologist to reject or approve that the image represents the patient 1150. The software tool may additionally enable the pathologist to approve the calculated risk score. By approving the calculated risk score, the pathologist may automatically sign the report.

[0067] In a particular embodiment, the analysis of the slide image based on the aforementioned workflow, that is, pre-selecting tiles based on a machine learning model and having a pathologist review the pre-selected tiles, can also include predicting the cells of origin of the slide image. In a particular embodiment, in order to predict the cells of origin, the tiles can be pre-selected by a machine learning model, which can be regarded as a region proposal. The embodiments disclosed herein conducted experiments on predicting cells of origin and obtained the following results. The results show that by integrating the region proposal and pathologist review performed by the machine learning model, the digital pathology image processing system 110 can improve the accuracy of predicting cells of origin, which is better than using the machine learning model alone for region proposal. After the manual quality inspection review of some regions in which the pathologist rejected the region, the region proposal results improved. The experiment is based on multiple (e.g., 97) Goya test set slides with manual tumor annotations. The comparison of the results measured by AUC (area under the ROC curve) is as follows. The AUC for the baseline model using manual tumor annotations is 74.3%. The AUC for the region proposal that has not been reviewed by a pathologist is 72.6%. The AUC for the region proposal reviewed by a pathologist is 74.2%. Since the embodiments disclosed herein are not limited to manual tumor annotation, we can also report results on a larger set of 129 slides, including slides without manual tumor annotation. The AUC for region proposals that were not reviewed by a pathologist was 75.3%. The AUC for region proposals that were reviewed by a pathologist was 76.7%.

[0068] Fig.12 An exemplary computer system 1200 is shown. In particular embodiments, one or more computer systems 1200 perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems 1200 provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems 1200 performs one or more steps of one or more methods described or illustrated herein, or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems 1200. Herein, references to computer systems may include computing devices and vice versa, where appropriate. Furthermore, references to computer systems may include one or more computer systems, where appropriate.

[0069] The present disclosure contemplates any suitable number of computer systems 1200. The present disclosure contemplates computer systems 1200 in any suitable physical form. As an example and not limitation, computer system 1200 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (such as, for example, a computer on a module (COM) or a system on a module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a computer system grid, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more thereof. Where appropriate, computer system 1200 may include one or more computer systems 1200; may be integrated or distributed; may span multiple locations; may span multiple machines; may span multiple data centers; or may reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1200 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. As an example and not by way of limitation, one or more computer systems 1200 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. Where appropriate, one or more computer systems 1200 may perform one or more steps of one or more methods described or illustrated herein at different times or in different locations.

[0070] In a particular embodiment, computer system 1200 includes a processor 1202, memory 1204, storage 1206, input / output (I / O) interface 1208, communication interface 1210, and bus 1212. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

[0071] In a particular embodiment, the processor 1202 includes hardware for executing instructions, such as those that constitute a computer program. As an example and not limitation, to execute instructions, the processor 1202 may retrieve (or fetch) instructions from an internal register, an internal cache, memory 1204, or storage 1206; these instructions may be decoded and executed; and one or more results may then be written to the internal register, internal cache, memory 1204, or storage 1206. In a particular embodiment, the processor 1202 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1202 that includes any suitable number of any suitable internal caches. As an example and not limitation, the processor 1202 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of the instructions in the memory 1204 or storage 1206, and the instruction cache may speed up the retrieval of those instructions by the processor 1202. The data in the data cache may be: a copy of the data in the memory 1204 or storage device 1206 for operation by instructions executed at the processor 1202; the result of the previous instruction executed at the processor 1202 for access or writing to the memory 1204 or storage device 1206 by subsequent instructions executed at the processor 1202; or other suitable data. The data cache may speed up the read or write operation of the processor 1202. The TLB may speed up the virtual address translation of the processor 1202. In a particular embodiment, the processor 1202 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1202 including any suitable number of any suitable internal registers. Where appropriate, the processor 1202 may include one or more arithmetic logic units (ALUs); may be a multi-core processor; or may include one or more processors 1202. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0072] In a particular embodiment, the memory 1204 includes a main memory for storing instructions for the processor 1202 to execute or data for the processor 1202 to operate on. As an example and not limitation, the computer system 1200 may load instructions from the storage device 1206 or another source (such as, for example, another computer system 1200) to the memory 1204. The processor 1202 may then load the instructions from the memory 1204 to an internal register or an internal cache. To execute the instructions, the processor 1202 may retrieve the instructions from the internal register or the internal cache and decode them. During or after the execution of the instructions, the processor 1202 may write one or more results (which may be intermediate results or final results) to the internal register or the internal cache. The processor 1202 may then write one or more of those results to the memory 1204. In a particular embodiment, the processor 1202 executes only instructions in one or more internal registers or internal caches or in the memory 1204 (rather than in the storage device 1206 or elsewhere), and operates only on data in one or more internal registers or internal caches or in the memory 1204 (rather than in the storage device 1206 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 1202 to the memory 1204. The bus 1212 may include one or more memory buses, as described below. In a particular embodiment, one or more memory management units (MMUs) reside between the processor 1202 and the memory 1204 and facilitate access to the memory 1204 requested by the processor 1202. In a particular embodiment, the memory 1204 includes a random access memory (RAM). Where appropriate, the RAM may be a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port or multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, memory 1204 may include one or more memories 1204. Although this disclosure describes and illustrates particular memories, this disclosure contemplates any suitable memories.

[0073] In a particular embodiment, the storage device 1206 includes a mass storage device for data or instructions. By way of example and not limitation, the storage device 1206 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more thereof. Where appropriate, the storage device 1206 may include a removable or non-removable (or fixed) medium. Where appropriate, the storage device 1206 may be inside or outside the computer system 1200. In a particular embodiment, the storage device 1206 is a non-volatile solid-state memory. In a particular embodiment, the storage device 1206 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more thereof. The present disclosure contemplates a mass storage device 1206 in any suitable physical form. Where appropriate, storage 1206 may include one or more memory control units that facilitate communications between processor 1202 and storage 1206. Where appropriate, storage 1206 may include one or more storage devices 1206. Although this disclosure describes and illustrates particular storage devices, this disclosure contemplates any suitable storage devices.

[0074] In a particular embodiment, the I / O interface 1208 includes hardware, software, or both, which provides one or more interfaces for communicating between the computer system 1200 and one or more I / O devices. Where appropriate, the computer system 1200 may include one or more of these I / O devices. One or more of these I / O devices can enable communication between a person and the computer system 1200. As an example and not limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a trackball, a camera, another suitable I / O device, or a combination of two or more thereof. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 1208 for them. Where appropriate, the I / O interface 1208 may include one or more devices or software drivers so that the processor 1202 can drive one or more of these I / O devices. Where appropriate, the I / O interface 1208 may include one or more I / O interfaces 1208. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0075] In a particular embodiment, the communication interface 1210 includes hardware, software, or both that provides one or more interfaces for communication (such as, for example, packet-based communication) between the computer system 1200 and one or more other computer systems 1200 or one or more networks. As an example and not limitation, the communication interface 1210 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network (such as a WI-FI network). The present disclosure contemplates any suitable network and any suitable communication interface 1210 for it. As an example and not limitation, the computer system 1200 may communicate with one or more parts of an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more thereof. One or more parts of one or more of these networks may be wired or wireless. As an example, the computer system 1200 may be connected to a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more thereof. Where appropriate, the computer system 1200 may include any suitable communication interface 1210 for any of these networks. Where appropriate, the communication interface 1210 may include one or more communication interfaces 1210. Although the present disclosure describes and illustrates particular communication interfaces, the present disclosure encompasses any suitable communication interface.

[0076] In certain embodiments, bus 1212 includes hardware, software, or both that couples components of computer system 1200 to each other. By way of example and not limitation, bus 1212 may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front side bus (FSB), a hypertransport (HT) interconnect, an industry standard architecture (ISA) bus, an INFINIBAND interconnect, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus, or another suitable bus or a combination of two or more thereof. Where appropriate, bus 1212 may include one or more buses 1212. Although the present disclosure describes and illustrates a particular bus, the present disclosure contemplates any suitable bus.

[0077] Herein, one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more thereof. Where appropriate, the computer-readable non-transitory storage medium may be a volatile storage medium, a non-volatile storage medium, or a combination of a volatile storage medium and a non-volatile storage medium.

[0078] As used herein, "or" is inclusive and not exclusive, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A or B" means "A, B, or both," unless expressly stated otherwise or the context indicates otherwise. Furthermore, as used herein, "and" means both jointly and severally, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly stated otherwise or the context indicates otherwise.

[0079] The scope of the present disclosure covers all changes, substitutions, variations, alterations and modifications to the exemplary embodiments described or shown herein that will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the exemplary embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the corresponding embodiments herein as including specific parts, elements, features, functions, operations or steps, any of these embodiments may include any combination or arrangement of any parts, elements, features, functions, operations or steps described or shown anywhere herein that will be understood by those of ordinary skill in the art. In addition, in the appended claims, references to devices or systems or parts of devices or systems adapted to, arranged to, capable of, configured to, enabled to, operable to or operated to perform specific functions, cover the device, system, component, whether or not it or the specific function is activated, turned on or unlocked, as long as the device, system or component is so adapted, arranged, capable of, configured to, enabled to, operable or operated. In addition, although the present disclosure describes or illustrates specific embodiments as providing specific advantages, specific embodiments may not provide these advantages, some advantages or all advantages.

[0080] Exemplary Embodiments

[0081] Embodiments disclosed herein may include:

[0082] 1. A method comprising: accessing, by a digital pathology image processing system, a slide image associated with a tissue for medical analysis; segmenting the slide image into a plurality of tiles; selecting, by one or more machine learning models, one or more tiles from the plurality of tiles based on one or more criteria associated with the medical analysis; displaying the one or more selected tiles via a user interface for user review; receiving, via the user interface, one or more user inputs associated with the one or more tiles; and generating analysis results for the medical analysis based on the one or more user inputs and the one or more tiles.

[0083] 2. The method of embodiment 1, further comprising: generating a first subset from the plurality of tiles by filtering out one or more first tiles from the plurality of tiles, wherein each of the one or more first tiles comprises an artifact, and wherein the one or more tiles are selected from the first subset.

[0084] 3. The method according to embodiment 2 further includes: generating a second subset from the first subset by filtering out one or more second tiles from the first subset, wherein each of the one or more second tiles corresponds to an area used for the medical analysis, and wherein the one or more tiles are selected from the second subset.

[0085] 4. The method of embodiment 3, wherein the medical analysis comprises a tumor analysis, and wherein generating the second subset is based on a tumor detection algorithm, and wherein each tile in the second subset comprises a tumor.

[0086] 5. The method of any one of embodiments 1 to 4, wherein the one or more criteria include one or more of a high interest value or a high representativeness of the disease targeted by the medical analysis.

[0087] 6. The method of any one of embodiments 1 to 5, wherein the tissue is associated with a patient having a tumor, and wherein the method further comprises: generating a segmentation including a nucleus for each of the one or more selected tiles.

[0088] 7. The method of any one of embodiments 1 to 6, wherein the medical analysis comprises determining one or more of recurrence of disease or resistance to treatment.

[0089] 8. The method of any one of embodiments 1 to 7, wherein the one or more user inputs include one or more of an approval of a tile, a rejection of a tile, or a rating of a tile.

[0090] 9. A method according to any one of embodiments 1 to 8, wherein the one or more user inputs include one or more approvals for one or more tiles, wherein the method further includes: determining that the number of one or more approved tiles reaches a predetermined number, wherein the analysis result is automatically generated based on the number of the one or more approved tiles reaching the predetermined number.

[0091] 10. A method according to any one of embodiments 1 to 9, wherein the one or more user inputs include one or more rejections of one or more tiles, and wherein the method further comprises: selecting one or more additional tiles from the multiple tiles for review by the user based on the one or more criteria associated with the medical analysis through the one or more machine learning models.

[0092] 11. The method of any one of embodiments 1 to 10, wherein the analysis results comprise one or more of: a risk score indicating the likelihood of recurrence of the disease, a risk score indicating the likelihood of resistance to treatment, or a probability indicating the risk of relapse or refractory disease at a specific time point.

[0093] 12. The method according to any one of embodiments 1 to 11 further comprises: displaying one or more positions of the one or more selected tiles relative to the tissue respectively via the user interface.

[0094] 13. A method according to any one of embodiments 1 to 12, wherein the tissue is stained based on H&E staining, wherein the method further comprises: generating a virtual DAB staining associated with the tissue through the one or more machine learning models, wherein the user interface is operable to adjust the display of each of the one or more selected tiles based on one or more of the H and E staining or the virtual DAB staining.

[0095] 14. The method according to any one of embodiments 1 to 13 further comprises: receiving one or more additional user inputs via the user interface, including one or more of the following items: approval of the analysis results, adjustment of the analysis results, or overwriting of the analysis results; generating a medical report based on the one or more additional user inputs; and receiving a signature for the medical report.

[0096] 15. One or more computer-readable non-transitory storage media embodying software that, when executed, is operable to perform the method of any one of embodiments 1 to 14.

[0097] 16. A system comprising: one or more processors; and a non-volatile memory coupled to the processor, the non-volatile memory comprising instructions executable by the processor, the processor being operable to perform the method according to any one of claims 1 to 14 when executing the instructions.

Claims

1. A method comprising, by a digital pathology image processing system: accessing slide images associated with tissue for medical analysis; dividing the slide image into a plurality of image blocks; selecting, by one or more machine learning models, one or more tiles from the plurality of tiles based on one or more criteria associated with the medical analysis; displaying the one or more selected tiles via a user interface for user review; receiving, via the user interface, one or more user inputs associated with the one or more tiles; as well as An analysis result for the medical analysis is generated based on the one or more user inputs and the one or more tiles.

2. The method according to claim 1, further comprising: A first subset is generated from the plurality of tiles by filtering out one or more first tiles from the plurality of tiles, wherein each of the one or more first tiles includes an artifact, and wherein the one or more tiles are selected from the first subset.

3. The method according to claim 2, further comprising: A second subset is generated from the first subset by filtering out one or more second tiles from the first subset, wherein each of the one or more second tiles corresponds to a region for the medical analysis, and wherein the one or more tiles are selected from the second subset. 4 . The method of claim 3 , wherein the medical analysis comprises a tumor analysis, and wherein generating the second subset is based on a tumor detection algorithm, and wherein each tile in the second subset comprises a tumor.

5. The method according to any one of claims 1 to 4, wherein the one or more criteria include one or more of a high interest value or a high representativeness of the disease targeted by the medical analysis.

6. The method according to any one of claims 1 to 5, wherein the tissue is associated with a patient suffering from a tumor, and wherein the method further comprises: A segmentation including a core is generated for each selected tile of the one or more selected tiles.

7. The method of any one of claims 1 to 6, wherein the medical analysis comprises determining one or more of recurrence of disease or resistance to treatment.

8. The method of any one of claims 1 to 7, wherein the one or more user inputs include one or more of an approval of a tile, a rejection of a tile, or a rating of a tile.

9. The method of any one of claims 1 to 8, wherein the one or more user inputs include one or more approvals of one or more tiles, wherein the method further comprises: It is determined that the number of one or more approved tiles reaches a predetermined number, wherein the analysis result is automatically generated based on the number of the one or more approved tiles reaching the predetermined number.

10. The method of any one of claims 1 to 9, wherein the one or more user inputs include one or more rejections of one or more tiles, wherein the method further comprises: One or more additional tiles are selected from the plurality of tiles for the user review based on the one or more criteria associated with the medical analysis by the one or more machine learning models.

11. The method according to any one of claims 1 to 10, wherein the analysis results include one or more of the following: a risk score indicating the likelihood for recurrence of the disease, a risk score indicating the likelihood for resistance to treatment, or a probability indicating the risk of relapse or refractoryness at a specific time point.

12. The method according to any one of claims 1 to 11, further comprising: One or more locations of the one or more selected tiles relative to the tissue are respectively displayed via the user interface.

13. The method according to any one of claims 1 to 12, wherein the tissue is stained based on H&E staining, wherein the method further comprises: A virtual DAB stain associated with the tissue is generated by the one or more machine learning models, wherein the user interface is operable to adjust the display of each of the one or more selected tiles based on one or more of the H&E stain or the virtual DAB stain.

14. The method according to any one of claims 1 to 13, further comprising: receiving, via the user interface, one or more additional user inputs including one or more of: approval of the analysis result, adjustment of the analysis result, or overwriting of the analysis result; generating a medical report based on the one or more additional user inputs; as well as A signature on the medical report is received.

15. One or more computer-readable non-transitory storage media embodying software that, when executed, is operable to: accessing slide images associated with tissue for medical analysis; dividing the slide image into a plurality of image blocks; selecting, by one or more machine learning models, one or more tiles from the plurality of tiles based on one or more criteria associated with the medical analysis; displaying the one or more selected tiles via a user interface for user review; receiving, via the user interface, one or more user inputs associated with the one or more tiles; as well as An analysis result for the medical analysis is generated based on the one or more user inputs and the one or more tiles.

16. The medium of claim 15, wherein the software when executed is further operable to: A first subset is generated from the plurality of tiles by filtering out one or more first tiles from the plurality of tiles, wherein each of the one or more first tiles includes an artifact, and wherein the selected one or more tiles are selected from the first subset.

17. The medium of claim 16, wherein the software when executed is further operable to: A second subset is generated from the first subset by filtering out one or more second tiles from the first subset, wherein each of the one or more second tiles corresponds to a region for the medical analysis, and wherein the selected one or more tiles are selected from the second subset.

18. The medium of any one of claims 15 to 17, wherein the one or more criteria include one or more of a high interest value or a high representativeness of a disease targeted by the medical analysis.

19. A system comprising: one or more processors; and a non-transitory memory coupled to the processor, the non-transitory memory comprising instructions executable by the processor, the processor, when executing the instructions, being operable to: accessing slide images associated with tissue for medical analysis; dividing the slide image into a plurality of image blocks; selecting, by one or more machine learning models, one or more tiles from the plurality of tiles based on one or more criteria associated with the medical analysis; displaying the one or more selected tiles via a user interface for user review; receiving, via the user interface, one or more user inputs associated with the one or more tiles; as well as An analysis result for the medical analysis is generated based on the one or more user inputs and the one or more tiles.

20. The system of claim 19, wherein the one or more criteria include one or more of a high interest value or a high representativeness of a disease targeted by the medical analysis.

Images