Predicting tile level category tags for histopathologic images

Abstract

A method implemented by one or more computer devices includes providing a weakly supervised neural network for analyzing histopathological images. The method includes accessing a histopathologic image that includes a slide level category tag. The method includes extracting a plurality of pixel regions of the histopathologic image at a plurality of magnifications. For each of the extracted plurality of pixel regions, the method further comprises: inputting the pixel region into a machine learning model, the machine learning model trained to generate a prediction of a category tag for the pixel region based on the pixel region and the slide level category tag; and outputting the prediction of the category tag of the pixel region. The method includes generating a prediction of one or more tile-level category tags of the histopathologic image based on the prediction of a category tag for each of the extracted plurality of pixel regions.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims the priority of U.S. Provisional Patent Application No. 63 / 418,425, filed on October 21, 2022, with the title "Predicting Tile - Level Class Labels for Histopathology Images", the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] This application generally relates to histopathology images and, more particularly, to techniques for predicting tile - level class labels for histopathology images. Background Art

[0004] Histopathology images can generally enable a histopathologist to visualize and analyze tissues or cells to determine, for example, whether variations occurring in the tissues or cells are caused by diseases, toxicities, and / or natural processes. For example, histopathology images can include very large and high - resolution images, in some cases including up to 100K×100K pixels. Thus, achieving efficient analysis of histopathology images can generally rely on one or more image - analysis tasks or machine - learning - model - based tasks to convert raw image data into a qualitative determination of the tissues or cells included within the histopathology image. Specifically, one or more image - analysis tasks or machine - learning - model - based tasks can generally include image enhancement, image segmentation, image feature extraction, and finally image classification.

[0005] For example, some specific cases of one or more image analysis tasks or machine learning model-based tasks can include identifying certain regions of tissue or cells that appear normal, diseased, or correspond to one or more other similar classes of clinical interest. For example, by identifying regions of tissue or cells that appear normal or diseased and quantifying the area, shape, or texture of these regions of the tissue or cells, one or more image analysis tasks or machine learning model-based tasks can be performed in a few minutes that would otherwise require many hours of arduous effort by a histopathologist. However, for example, training a machine learning model to identify regions of tissue or cells that appear normal or diseased and quantifying the area, shape, or texture of these regions of the tissue or cells typically may require a large dataset of histopathology images, each dataset including a large number of annotations. Specifically, in order to accumulate sufficient training data to accurately train a machine learning model, a histopathologist or multiple other professional human annotators would have to manually annotate a large number of histopathology image datasets, each dataset potentially including thousands or millions of individual features of clinical interest. In practice, such manual annotation can be time-consuming, expensive, and prone to significant human error. Summary of the Invention

[0006] Embodiments of the present disclosure relate to one or more computing devices, methods, and non-transitory computer-readable media that can be used to train one or more machine learning models to generate predictions of one or more tile-level class labels for a histopathology image based on a slide-level class label pre-assigned to the histopathology image. For example, in some embodiments, one or more computing devices can access a histopathology image that includes, for example, a slide-level class label (e.g., a high-level or sparse image label or a hand-drawn boundary geometry covering a large area of different tissues, cells, or other features) pre-assigned to the histopathology image by a histopathologist or another professional human annotator. Then, one or more computing devices can extract and downsample multiple image tiles of the histopathology image to multiple different magnifications (e.g., ranging from low magnification, medium magnification up to high magnification).

[0007] In some embodiments, one or more computing devices may then input image tiles at multiple different magnifications into corresponding machine learning models (e.g., a machine learning model ensemble), where each machine learning model is trained to generate predictions of tile-level class labels for the corresponding image tiles at different magnifications using the image tiles and slide-level class labels pre-assigned to the histopathology image. In some embodiments, the predictions of the tile-level class labels for the corresponding image tiles at different magnifications may then be averaged to generate one or more tile-level class label predictions for the histopathology image. In this way, the presently disclosed embodiments can provide predicted tile-level class labels (e.g., class labels or boundary geometries identifying specific regions of cancer cells, immune cell regions, one or more biomarkers corresponding to specific regions, or other clinically interesting feature regions) for low-level features within a histopathology image using only the corresponding extracted and downsampled pixel regions (e.g., one or more pixel tiles) and the sparse slide-level class labels pre-assigned to the histopathology image. In fact, without the presently disclosed embodiments, such tile-level annotation tasks would otherwise require time-consuming, expensive, and potentially error-prone manual annotation. Additionally, by training machine learning models (e.g., a machine learning model ensemble) to predict tile-level class labels at different magnifications, once trained, the machine learning models (e.g., a machine learning model ensemble) may be more suitable for predicting tile-level class labels for histopathology image features at different magnifications and / or resolutions (e.g., in a manner similar to how a histopathologist would analyze and classify histopathology image features).

[0008] In some embodiments, one or more computing devices may access a histopathology image, where the histopathology image includes slide-level class labels. For example, in some embodiments, the histopathology image may include at least one of a histological stain image, a fluorescence in situ hybridization (FISH) image, an immunofluorescence (IF) image, or a hematoxylin and eosin (H&E) image. In some embodiments, one or more computing devices may then extract multiple pixel regions of the histopathology image at multiple magnifications. For example, in some embodiments, each magnification of the multiple magnifications is different from each other magnification of the multiple magnifications. In some embodiments, extracting multiple pixel regions of the histopathology image at multiple magnifications may include downsampling the multiple pixel regions to the multiple magnifications.

[0009] In some embodiments, for each of the plurality of pixel regions extracted, one or more computing devices may input the pixel region into a machine learning model that is trained to generate a prediction of the class label of the pixel region based on the pixel region and a slide-level class label, and output from the machine learning model a prediction of the class label of the pixel region. For example, in one embodiment, the machine learning model may include an artificial neural network (ANN), a convolutional neural network (CNN), or a deep neural network (DNN). In another embodiment, the machine learning model may include one CNN in a CNN ensemble. In some embodiments, one or more computing devices may generate a prediction of one or more tile-level class labels of the histopathology image based on the prediction of the class label of each of the plurality of pixel regions extracted. For example, in some embodiments, the prediction of one or more tile-level class labels may include the identification of one or more biomarkers associated with the tissue or cells included within the histopathology image.

[0010] In some embodiments, prior to generating a prediction of one or more tile-level class labels, one or more computing devices may normalize the prediction of the class label of each of the plurality of pixel regions extracted. For example, in some embodiments, normalizing the prediction of the class label of each of the plurality of pixel regions extracted may include normalizing the prediction of the class label of each of the plurality of pixel regions extracted to the scale of the pixel region at the maximum magnification. In some embodiments, one or more computing devices may generate a prediction of one or more tile-level class labels by calculating the average of the predictions of the class label of each of the plurality of pixel regions extracted. In some embodiments, training one or more machine learning models by one or more computing devices to generate a prediction of one or more tile-level class labels of the histopathology image may include training one or more machine learning models according to a weak supervision learning process.

[0011] For example, in some embodiments, after training one or more machine learning models to generate predictions of one or more tile-level class labels for a whole-slide histopathology image, one or more computing devices can then access a second histopathology image, input the second histopathology image into the trained one or more machine learning models to generate predictions of one or more tile-level class labels for the second histopathology image, and output, by the one or more machine learning models, the predictions of one or more tile-level class labels for the second histopathology image. In certain embodiments, the predictions of one or more tile-level class labels for the second histopathology image can include the identification of one or more biomarkers associated with the tissue or cells included within the second histopathology image. In certain embodiments, one or more computing devices can generate a report based on the predictions of one or more tile-level class labels for the histopathology image. In one embodiment, one or more computing devices can cause a human-machine interface (HMI) associated with a pathologist or clinician to display the report. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] One or more of the figures included herein are in color in accordance with 37 CFR § 1.84. The color figures are necessary for illustrating the invention. More specifically, Figure 3 、 Figure 6 、 Figure 7A 、 Figure 7B and Figure 8 are one or more high-resolution histopathology images of tissue or cells, all of these color figures play a major role in enabling those skilled in the art to understand the invention and such color figures are the only practical medium for disclosing the patent subject matter.

[0013] Figure 1 Shows an exemplary network of an interactive computer system.

[0014] Figure 2 Shows a system workflow diagram for training one or more machine learning models to generate predictions of one or more tile-level class labels for a histopathology image based on slide-level class labels pre-assigned to the histopathology image.

[0015] Figure 3 Shows an illustrative workflow diagram for training one or more machine learning models to generate predictions of one or more tile-level class labels for a histopathology image based on slide-level class labels pre-assigned to the histopathology image.

[0016] Figure 4Disclosed is a flowchart of a method for training one or more machine learning models to generate predictions of one or more tile-level class labels for a histopathology image based on slide-level class labels pre-assigned to the histopathology image.

[0017] Figure 5 Disclosed is a flowchart of a method for using one or more machine learning models to generate predictions of one or more tile-level class labels for a histopathology image based on slide-level class labels pre-assigned to the histopathology image.

[0018] Figure 6 Disclosed is a running example of using one or more machine learning models to generate predictions of one or more tile-level class labels for a histopathology image based on slide-level class labels pre-assigned to the histopathology image.

[0019] Figure 7A and Figure 7B Disclosed are one or more graphs or implementation examples of predictions of tile-level class labels for histopathology images at different magnifications.

[0020] Figure 8 Disclosed is an example machine learning model evaluation graph.

[0021] Figure 9 Shows a diagram of an example artificial intelligence (AI) architecture included as part of a network of an interactive computing system. Detailed Description

[0022] Embodiments of the present disclosure relate to one or more computing devices, methods, and non-transitory computer-readable media that can be used to train one or more machine learning models to generate predictions of one or more tile-level class labels for a histopathology image based on slide-level class labels pre-assigned to the histopathology image. For example, in some embodiments, one or more computing devices can access a histopathology image that includes, for example, slide-level class labels (e.g., high-level or sparse image labels or hand-drawn boundary geometries covering large areas of different tissues, cells, or other features) pre-assigned to the histopathology image by a histopathologist or another professional human annotator. Then, one or more computing devices can extract and downsample multiple image tiles of the histopathology image to multiple different magnifications (e.g., ranging from low magnification, medium magnification up to high magnification).

[0023] In some embodiments, one or more computing devices may then input image tiles at multiple different magnifications into corresponding machine learning models (e.g., a machine learning model ensemble), where each machine learning model is trained to generate predictions of tile-level class labels for the image tiles at different magnifications using the image tiles and slide-level class labels pre-assigned to the histopathology image. In some embodiments, the predictions of the tile-level class labels for the corresponding image tiles at different magnifications may then be averaged to generate one or more tile-level class label predictions for the histopathology image.

[0024] In this way, the presently disclosed embodiments can provide predicted tile-level class labels (e.g., class labels or boundary geometries identifying specific regions of cancer cells, immune cell regions, one or more biomarkers corresponding to specific regions, or other clinically interesting feature regions) for low-level features within a histopathology image using only the corresponding extracted and downsampled pixel regions (e.g., one or more pixel tiles) and sparse slide-level class labels pre-assigned to the histopathology image. In fact, without the presently disclosed embodiments, such tile-level annotation tasks would otherwise require time-consuming, expensive, and potentially error-prone manual annotation. Additionally, by training machine learning models (e.g., a machine learning model ensemble) to predict tile-level class labels at different magnifications, once trained, the machine learning models (e.g., a machine learning model ensemble) may be better suited to predict tile-level class labels for histopathology image features at different magnifications and / or resolutions (e.g., in a manner similar to how a histopathologist would analyze and classify histopathology image features).

[0025] Figure 1Network 100 of an interactive computer system is shown in accordance with presently disclosed embodiments, and one or more of such interactive computer systems can be used to train one or more machine learning models to generate predictions of one or more tile-level class labels for a histopathology image based on slide-level class labels pre-assigned to the histopathology image. In some embodiments, a whole slide image generation system 110 can generate, for example, one or more whole slide images (WSIs) or other relevant histopathology images corresponding to a particular sample. In one embodiment, one or more images generated by the whole slide image generation system 110 can include stained sections of a biopsy sample. In another embodiment, one or more images generated by the whole slide image generation system 110 can include a slide image of a liquid sample (e.g., a blood smear). In accordance with presently disclosed embodiments, one or more images generated by the whole slide image generation system 110 can include, for example, any one of a variety of histopathology images, such as fluorescence in situ hybridization (FISH) images, immunofluorescence (IF) images, multiplex immunofluorescence (mxIF) images, hematoxylin and eosin (H&E) images, immunohistochemistry (IHC) images, multiplex immunohistochemistry (mxIHC) images, imaging mass cytometry (IMC) images, etc.

[0026] In some embodiments, some types of samples (e.g., biopsy tissue, solid samples, and / or samples including tissue) can be processed by a sample preparation system 121 to fix and / or embed the sample. The sample preparation system 121 can facilitate infiltration of the sample with a fixative (e.g., a liquid fixative such as a formaldehyde solution) and / or an embedding substance (e.g., histological wax). For example, a sample fixation subsystem can fix the sample by exposing the sample to the fixative for at least a threshold amount of time (e.g., at least 3 hours, at least 6 hours, or at least 13 hours). A dehydration subsystem can dehydrate the sample (e.g., by exposing the fixed sample and / or a portion of the fixed sample to one or more ethanol solutions) and potentially use a clearing intermediate (e.g., which includes ethanol and histological wax) to clear the dehydrated sample. A sample embedding subsystem can infiltrate the sample with heated (e.g., thus liquid) histological wax (e.g., one or more times for a corresponding predefined period of time). The histological wax can comprise paraffin and potentially one or more resins (e.g., styrene or polyethylene). The sample and wax can then be cooled, and then the wax-infiltrated sample can be enclosed.

[0027] In some embodiments, the sample slicer 122 can receive fixed and embedded samples and can produce a set of sections. The sample slicer 122 can expose the fixed and embedded samples to cool or cold temperatures. The sample slicer 122 can then cut the frozen sample (or a trimmed version thereof) to produce a set of sections. Each section can have a thickness of (for example) less than 100 μm, less than 50 μm, less than 10 μm, or less than 5 μm. Each section can have a thickness of (for example) greater than 0.1 μm, greater than 1 μm, greater than 2 μm, or greater than 4 μm. The cutting of the frozen sample can be performed in a warm water bath (for example, at a temperature of at least 30 °C, at least 35 °C, or at least 40 °C). In some embodiments, the automated staining system 123 can facilitate the staining of one or more of the sample sections by exposing each section to one or more stains.

[0028] In some embodiments, each of one or more stained sections can be presented to an image scanner 124, which can capture a digital image of the section. In some embodiments, the image scanner 124 can include a microscope camera. The image scanner 124 can capture digital images at multiple magnifications (for example, using 2x objective, 5x objective, 10x objective, 20x objective, 40x objective, 100x objective, 200x objective, 500x objective, etc.). Manipulation of the image can be used to capture a selected portion of the sample within a desired magnification range. The image scanner 124 can further capture annotations and / or morphometric measurements identified by a human operator. In some embodiments, after one or more images are captured, the section can be returned to the automated staining system 123 such that the section can be washed, exposed to one or more other stains, and imaged again.

[0029] In some embodiments, a given sample can be associated with one or more users (e.g., one or more physicians, laboratory technicians, and / or healthcare providers) during processing and imaging. The associated users can include, for example but not limited to, the person who orders the test or biopsy that generates the imaged sample, the person who is authorized to receive the results of the test or biopsy, or the person who analyzes the test or biopsy sample, etc. For example, the user can correspond to a physician, a histopathologist, a clinician, or a subject. The user can use one or a user device 130 to submit one or more of the following requests (e.g., which identify the subject): to process the sample by the whole slide image generation system 110 and to process the resulting image by the whole slide image processing system 110. In some embodiments, the whole slide image generation system 110 can transmit the image generated by the image scanner 124 back to the user device 130. The user device 130 can then communicate with the whole slide image processing system 110 to initiate automated processing of the image. In some embodiments, the whole slide image generation system 110 can provide the image generated by the image scanner 124 directly to the whole slide image processing system 110, for example, under the instruction of the user of the user device 130.

[0030] In some embodiments, the whole slide image processing system 110 can process histopathology images to classify the histopathology images and generate annotations for the histopathology images and related outputs. For example, the tile generation module 111 can define a set of tiles (e.g., pixel regions or pixel sub-regions) for each histopathology image. To define the tile set, the tile generation module 111 can segment the histopathology image into the tile set. In some embodiments, the tiles can be non-overlapping (e.g., each tile includes pixels of the image that are not included in any other tile) or overlapping (e.g., each tile includes a portion of the pixels of the image that are included in at least one other tile). In addition to the size of each tile and the stride of the window (e.g., the image distance or pixels between a tile and a subsequent tile), features such as whether the tiles are overlapping can also increase or decrease the data set used for analysis, where more tiles (e.g., by overlapping or smaller tiles) increase the potential resolution of the final output and visualization.

[0031] In some embodiments, the tile generation module 111 may define a set of tiles for an image, where each tile has a predefined size and / or the offset between tiles is predefined. Additionally, the tile generation module 111 may create multiple sets of tiles for each image with different sizes, overlaps, strides, etc. In some embodiments, the histopathology image itself may contain tile overlaps, which may be generated by the imaging technique. An average segmentation without tile overlap may be a preferred solution to balance tile processing requirements and avoid affecting the embedding generation and weight value generation. The tile size or tile offset may be determined, for example, by calculating one or more performance metrics (e.g., precision, recall, accuracy, and / or error) for each size / offset, and by selecting the tile size and / or offset associated with one or more performance metrics above a predetermined threshold and / or associated with one or more optimal (e.g., high precision, highest recall, highest accuracy, and / or lowest error) performance metrics.

[0032] In certain embodiments, the tile generation module 111 may also define the tile size depending on the type of detected abnormality. For example, the tile generation module 111 may be configured to know the type of tissue abnormality that the whole slide image processing system 110 will search for and may customize the tile size according to the tissue abnormality to optimize detection. For example, the image generation module 111 may determine that when the tissue abnormality includes searching for inflammation or necrosis in lung tissue, the tile size should be reduced to increase the scanning rate, while when the tissue abnormality includes an abnormality of Kupffer cells in liver tissue, the tile size should be increased to increase the opportunity for the whole slide image processing system 110 to analyze Kupffer cells overall.

[0033] In certain embodiments, the tile generation module 111 may further define a set of tiles for each histopathology image along one or more color channels or color combinations. For example, the histopathology images received by the whole slide image processing system 110 may include large format multi-color channel images that have pixel color values for each pixel of the image specified for one of several color channels. Exemplary color specifications or color spaces that may be used include RGB, CMYK, HSL, HSV, or HSB color specifications. The set of tiles may be defined based on segmenting the color channels and / or generating a luminance map or grayscale equivalent for each tile. For example, for each segment of the image, the tile generation module 111 may provide red tiles, blue tiles, green tiles, and / or luminance tiles, or equivalents of the color specification used.

[0034] In some embodiments, the tile embedding module 112 may generate an embedding for each tile in a corresponding feature embedding space. The embedding may be represented as a feature vector of the tile by the whole slide image processing system 110. The tile embedding module 112 may utilize a neural network (e.g., a convolutional neural network (CNN)) or other similar image classification neural network) to generate a feature vector representing each tile of the image. In some embodiments, the tile embedding neural network may be based on, for example, a residual neural network (ResNet) image classification neural network trained on a dataset of natural (e.g., non-medical) images such as the ImageNet dataset. By using a non-specialized tile embedding network, the tile embedding module 112 may leverage known advancements in efficiently processing images to generate embeddings. Additionally, using a natural image dataset allows the embedding neural network to learn to discriminate differences between patch segments at an overall level.

[0035] In other embodiments, the tile embedding network used by the tile embedding module 112 may be an embedding network customized to process a large number of tiles of large format images such as histopathology images. Additionally, a custom dataset may be used to train the tile embedding network used by the tile embedding module 112. For example, the tile embedding network may be trained using various samples of the WSI, or even samples related to the subject (e.g., scans of a specific tissue type) for which the embedding network will generate embeddings. Training the tile embedding network using a specialized or custom image set may allow the tile embedding network to identify more subtle differences between tiles, which may result in a more detailed and accurate distance between tiles in the feature embedding space, but at the cost of additional time to acquire images and the computational and economic costs of training multiple tile generation networks for use by the tile embedding module 112. The tile embedding module 112 may select from a library of tile embedding networks based on the type of image being processed by the whole slide image processing system 110.

[0036] In some embodiments, the whole slide image access module 113 may manage requests to access the WSI from other modules of the whole slide image processing system 110 as well as the user device 130. For example, the whole slide image access module 113 may receive requests to identify the WSI based on a specific tile, an identifier of the tile, or an identifier of the whole slide image. The whole slide image access module 113 may perform the following tasks: confirm the availability of the WSI to the requesting user, identify the appropriate database from which to retrieve the requested WSI, and retrieve any additional metadata that the requesting user or module may be interested in.

[0037] In some embodiments, the output generation module 114 of the whole slide image processing system 110 can generate outputs corresponding to the result tiles and the result WSI dataset based on a user request. As described herein, the outputs can include various visualizations, interactive graphics, and reports based on the type of request and the type of data available. In many embodiments, the outputs will be provided to the user device 130 for display, but in some embodiments, the outputs can be accessed directly from the whole slide image processing system 110. The outputs can be based on the presence and access to appropriate data, so the output generation module 116 can be authorized to access metadata and anonymized patient information as needed. Like other modules of the whole slide image processing system 110, the output generation module 114 can be updated and improved in a modular fashion such that new output features can be provided to the user without significant downtime.

[0038] The general techniques described herein can be integrated into a variety of tools and use cases. For example, as described, a user (e.g., a histopathologist or a clinician) can access a user device 130 that communicates with the whole slide image processing system 110 and provide a query image for analysis. The whole slide image processing system 110 or the connection to the whole slide image processing system 110 can be provided as a stand-alone software tool or package that searches for corresponding matches, identifies similar features, and generates an appropriate output for the user upon request. As a stand-alone tool or plug-in that can be purchased or licensed on a simplified basis, the tool can be used to enhance the capabilities of a research or clinical laboratory. Additionally, the tool can be integrated into services available to customers of the whole slide image generation system.

[0039] Figure 2 Shown is a system workflow diagram 200 according to presently disclosed embodiments for training one or more machine learning models to generate predictions of one or more tile-level class labels for a histopathology image based on a slide-level class label pre-assigned to the histopathology image. As depicted, in some embodiments, a histopathology image 202 can be accessed. In one embodiment, the histopathology image 202 may have been in accordance with the above with respect to Figure 1One or more of the described techniques are for generating and processing images. In some embodiments, the histopathology image 202 can include, for example, any one of a variety of WSIs, such as fluorescence in situ hybridization (FISH) images, immunofluorescence (IF) images, multiplexed immunofluorescence (mxIF) images, hematoxylin and eosin (H&E) images, immunohistochemistry (IHC) images, multiplexed immunohistochemistry (mxIHC) images, imaging mass cytometry (IMC) images, and the like. In one embodiment, the histopathology image 202 can include one of a histopathology image dataset (e.g., hundreds or thousands of histopathology images), and each histopathology image can include very large and high-resolution images (e.g., 1.5K×2K pixels, 2K×4K pixels, 6K×8K pixels, 7.5K×10K pixels, 9K×12K pixels, 15K×20K pixels, 20K×24K pixels, 20K×30K pixels, 24K×30K pixels, 20K×40K pixels, 40K×60K pixels, 20K×80K pixels, 60K×80K pixels, 70K×100K pixels, 80K×100K pixels, 100K×100K pixels).

[0040] As further shown by Figure 2 In some embodiments, the histopathology image 202 can be assigned a slide-level class label. For example, as previously described, the histopathology image 202 can include very large and high-resolution images depicting, for example, a large amount of tissue or cells. In one embodiment, a histopathologist or other professional human annotator can assign a slide-level annotation (e.g., a high-level or sparse image label such as "neoplasm" or a hand-drawn boundary geometry covering different tissues, cells, or other features over a large area) to the histopathology image 202. While it may be clinically beneficial for a histopathologist or other professional human annotator to further annotate low-level features within the histopathology image 202 (e.g., assign class labels or boundary geometries identifying specific regions of cancer cells, immune cell regions, one or more biomarkers corresponding to specific regions, or other regions of clinical interest), performing such annotation tasks by human annotators can be time-consuming, expensive, and prone to significant human error.

[0041] Thus, as will be further understood below, it can be useful to train one or more machine learning models to generate predictions of one or more tile-level annotations of a histopathology image 202 based on slide-level annotations pre-assigned to the histopathology image 202. In fact, as will be described in more detail below, one or more machine learning models (e.g., machine learning models 206A, 206B, 206C, and 206D) can be trained according to a weakly-supervised learning process, where the one or more machine learning models (e.g., machine learning models 206A, 206B, 206C, and 206D) can be trained to generate predictions of one or more tile-level class labels of the histopathology image 202 based only on the following slide-level class labels, in which there is no prior knowledge about which pixel tiles of the histopathology image 202 are associated with the slide-level class labels pre-assigned to the histopathology image 202.

[0042] In certain embodiments, as depicted by Figure 2 further, multiple pixel regions (e.g., one or more pixel tiles) of the histopathology image 202 can be extracted (e.g., sampled) and downsampled (e.g., by corresponding downsampling functions 202A, 202B, and 202C) to multiple magnifications (e.g., low magnification "Nx", low magnification "Mx", low magnification "Yx" up to high magnification "Zx", where "N", "M", "Y", and "Z" each include an integer greater than 1). In one embodiment, a first image tile (e.g., a 256×256 pixel image tile) can be extracted and downsampled to 2x magnification, a second image tile (e.g., a 256×256 pixel image tile) can be extracted and downsampled to 5x magnification, a third image tile (e.g., a 256×256 pixel image tile) can be extracted and downsampled to 10x magnification, and a fourth image tile (e.g., a 256×256 pixel image tile) can be extracted and downsampled to 20x magnification. However, it should be understood that according to the presently disclosed embodiments, multiple pixel regions (e.g., one or more pixel tiles) can be extracted and downsampled to any number of magnifications (e.g., 2x magnification, 5x magnification, 10x magnification, 20x magnification, 50x magnification, 100x magnification, 200x magnification, 500x magnification, etc.).

[0043] In certain embodiments, as depicted by Figure 2Further depicted, multiple pixel regions (e.g., one or more pixel tiles) can then be input into corresponding machine learning models 206A, 206B, 206C, and 206D. For example, in some embodiments, machine learning models 206A, 206B, 206C, and 206D can each include a convolutional neural network (CNN) or a deep neural network (DNN). In one embodiment, machine learning models 206A, 206B, 206C, and 206D can include a machine learning model ensemble (e.g., a CNN ensemble trained in parallel or end-to-end, or other similar image classification neural network ensemble), where each of machine learning models 206A, 206B, 206C, and 206D can be based on a residual neural network (ResNet) image classification neural network or a deep ResNet image classification neural network (e.g., ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet-152).

[0044] In certain embodiments, as depicted by Figure 2 Further depicted, machine learning model 206A can be trained to utilize the slide-level class label associated with the histopathology image 202 to generate a tile-level class label prediction 208A for the image tile that is extracted and downsampled to an Nx magnification. Similarly, in certain embodiments, machine learning model 206B can be trained to utilize the slide-level class label associated with the histopathology image 202 to generate a tile-level class label prediction 208B for the image tile that is extracted and downsampled to an Mx magnification. Machine learning model 206C can be trained to utilize the slide-level class label associated with the histopathology image 202 to generate a tile-level class label prediction 208C for the image tile that is extracted and downsampled to a Yx magnification. Finally, machine learning model 206D can be trained to utilize the slide-level class label of the histopathology image 202 to generate a tile-level class label prediction 208D for the image tile that is extracted and downsampled to a Zx magnification.

[0045] In some embodiments, one or more of the generated patch-level class label predictions 208A, 208B, 208C, and 208D can then be upsampled (e.g., by upsampling functions 210A, 210B, 210C, and 210D) to normalize or scale each of the patch-level class label predictions 208A, 208B, 208C, and 208D to the same resolution. For example, in one embodiment, each of the patch-level class label predictions 208A, 208B, 208C, and 208D can be normalized or scaled to the magnification or resolution of a pixel region (e.g., one or more pixel patches) that was previously downsampled to a maximum magnification (e.g., high magnification “Zx”).

[0046] In some embodiments, the normalized patch-level class label predictions 208A, 208B, 208C, and 208D can then be averaged (e.g., by averaging function 212) to generate one or more patch-level class label predictions 214 for the histopathology image 202. For example, as previously described, the machine learning models 206A, 206B, 206C, and 206D can include a machine learning model ensemble (e.g., a CNN ensemble trained in parallel or end-to-end, or other similar image classification neural network ensemble), and thus the normalized patch-level class label predictions 208A, 208B, 208C, and 208D can be averaged to, for example, significantly improve the accuracy of one or more patch-level class label predictions 214 as compared to any one of the patch-level class label predictions 208A, 208B, 208C, and 208D.

[0047] In this way, the presently disclosed embodiments can provide predicted patch-level class labels (e.g., class labels or boundary geometries identifying specific regions of cancer cells, immune cell regions, one or more biomarkers corresponding to specific regions, or other clinically interesting feature regions) for low-level features within the histopathology image 202 using only the corresponding extracted and downsampled pixel regions (e.g., one or more pixel patches) and sparse slide-level class labels pre-assigned to the histopathology image 202. In fact, without the presently disclosed embodiments, such patch-level annotation tasks would otherwise require time-consuming, expensive, and potentially error-prone manual annotation. Additionally, by training a machine learning model (e.g., a machine learning model ensemble) to predict patch-level class labels at different magnifications, once trained, the machine learning model (e.g., a machine learning model ensemble) may be better suited to predict patch-level class labels for histopathology image features at different magnifications and / or resolutions (e.g., in a manner similar to how a histopathologist would analyze and classify histopathology image features).

[0048] In some embodiments, once computed, one or more tile-level class label predictions 214 of the histopathology image 202 can be used in one or more downstream tasks. For example, in some embodiments, a report can be generated based on one or more tile-level class label predictions 214 of the histopathology image 202. For example, in one embodiment, the report can include a clinical report that can be associated with one or more cancer patients for providing and displaying to, for example, a histopathologist or a clinician (e.g., an oncologist) for research and / or diagnosis, prognosis, and treatment of one or more patients. In another embodiment, the report can include an interpretability and / or explainability report that can be associated with the machine learning models 206A, 206B, 206C, and 206D for providing and displaying to, for example, one or more data scientists or developers for determining and elucidating the prediction and decision-making behaviors of the machine learning models 206A, 206B, 206C, and 206D.

[0049] Figure 3 Illustrative workflow diagram 300 in accordance with presently disclosed embodiments is shown, which is for training one or more machine learning models to generate predictions of one or more tile-level class labels of a histopathology image based on a slide-level class label pre-assigned to the histopathology image. Specifically, illustrative workflow diagram 300 can generally correspond to system workflow diagram 200 discussed above with respect to Figure 2 and shows the presently disclosed techniques as applied to, for example, an H&E histopathology image 302 that includes slide-level annotations (e.g., as shown by bounding geometry 350).

[0050] As depicted, in some embodiments, an H&E histopathology image 302 can be accessed. As further shown by Figure 3 In some embodiments, the H&E histopathology image 302 can be assigned a slide-level class label. For example, as previously described, the histopathology image 202 can include very large and high-resolution images depicting, for example, a large amount of tissue or cells. In one embodiment, a histopathologist or other professional human annotator can assign a slide-level annotation to the H&E histopathology image 302, for example, by drawing a bounding geometry (e.g., as shown by bounding geometry 350) around the tissue or cells depicted by the H&E histopathology image 302.

[0051] In some embodiments, as shown by Figure 3Further depicted, multiple pixel regions 304A, 304B, 304C, and 304D (e.g., one or more pixel tiles) of the H&E histopathology image 302 can be extracted and downsampled (e.g., by respective downsampling functions 202A, 202B, and 202C) to multiple magnifications (e.g., low magnification "Nx", low magnification "Mx", low magnification "Yx" up to high magnification "Zx", where "N", "M", "Y", and "Z" each include an integer greater than 1). In one embodiment, the first image tile can be extracted and downsampled to a 2x magnification, the second image tile can be extracted and downsampled to a 5x magnification, the third image tile can be extracted and downsampled to a 10x magnification, and the fourth image tile can be extracted and downsampled to a 20x magnification.

[0052] In certain embodiments, as depicted by Figure 3 Further depicted, multiple pixel regions 304A, 304B, 304C, and 304D (e.g., one or more pixel tiles) can then be input into respective machine learning models 206A, 206B, 206C, and 206D. In certain embodiments, as depicted by Figure 3 Further depicted, the machine learning model 206A can be trained to generate a tile-level class label prediction 308A for the pixel region 304A (e.g., image tile) using the slide-level class label associated with the H&E histopathology image 302. Similarly, in certain embodiments, the machine learning model 206B can be trained to generate a tile-level class label prediction 308B for the pixel region 304B (e.g., image tile) using the slide-level class label associated with the H&E histopathology image 302. The machine learning model 206C can be trained to generate a tile-level class label prediction 308C for the pixel region 304C (e.g., image tile) using the slide-level class label associated with the H&E histopathology image 302. Finally, the machine learning model 206D can be trained to generate a tile-level class label prediction 308D for the pixel region 304D (e.g., image tile) using the slide-level class label associated with the H&E histopathology image 302.

[0053] In some embodiments, one or more of the generated tile - level class label predictions 308A, 308B, 308C, and 308D can then be upsampled (e.g., by upsampling functions 210A, 210B, 210C, and 210D) to normalize or scale each of the tile - level class label predictions 308A, 308B, 308C, and 308D to the same resolution. For example, in one embodiment, each of the tile - level class label predictions 308A, 308B, 308C, and 308D can be normalized or scaled to the magnification or resolution of a pixel region (e.g., one or more pixel tiles) that was previously downsampled to the maximum magnification (e.g., high magnification “Zx”). In some embodiments, the normalized tile - level class label predictions 308A, 308B, 308C, and 308D can then be averaged (e.g., by averaging function 212) to generate one or more tile - level class label predictions 306 for the H&E histopathology image 302.

[0054] For example, as previously described, the machine - learning models 206A, 206B, 206C, and 206D can include a machine - learning model ensemble (e.g., a CNN ensemble trained in parallel or end - to - end, or other image - classification neural - network ensemble), and thus can average the normalized tile - level class label predictions 308A, 308B, 308C, and 308D to, for example, significantly improve the accuracy of one or more tile - level class label predictions 306 as compared to any one of the tile - level class label predictions 308A, 308B, 308C, and 308D.

[0055] In this way, the presently disclosed embodiments can provide predicted tile-level class labels (e.g., class labels or boundary geometries identifying specific regions of cancer cells, immune cell regions, one or more biomarkers corresponding to specific regions, or other clinically interesting feature regions) for low-level features within the H&E histopathology image 302 using only the corresponding extracted and downsampled pixel regions (e.g., one or more pixel tiles) and the sparse slide-level class labels pre-assigned to the H&E histopathology image 302. In fact, in the absence of the presently disclosed embodiments, such tile-level annotation tasks would otherwise require time-consuming, expensive, and potentially error-prone manual annotation. Additionally, by training a machine learning model (e.g., a machine learning model ensemble) to predict tile-level class labels at different magnifications, once trained, the machine learning model (e.g., a machine learning model ensemble) may be more suitable for predicting tile-level class labels for histopathology image features at different magnifications and / or resolutions (e.g., in a manner similar to how a histopathologist would analyze and classify histopathology image features).

[0056] Figure 4 FIG. 4 shows a flowchart of a method 400 according to presently disclosed embodiments for training one or more machine learning models to generate predictions of one or more tile-level class labels for a histopathology image based on slide-level class labels pre-assigned to the histopathology image. Method 400 may be carried out using one or more processing devices (e.g., one or more of the networks of the interactive computer system 100 as discussed above with respect to Figure 1 which may include hardware (e.g., a general-purpose processor, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a microcontroller, a field-programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a vision processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), a deep learning processor (DLP), a tensor processing unit (TPU), a neuromorphic processing unit (NPU), a wafer-scale engine (WSE), or any one of various hardware AI accelerators) that may be suitable for processing various omics data and making one or more decisions based thereon, software (e.g., instructions running / executing on one or more processing devices), firmware (e.g., microcode), or some combination thereof.

[0057] The method may include: at block 402, one or more processing devices access a histopathology image, where the histopathology image includes a slide-level class label. For example, in one embodiment, one or more processing devices may access histopathology image 202, which includes a slide-level class label (e.g., a high-level or sparse image label such as "carcinoma", or a hand-drawn boundary geometry covering different tissues, cells, or other features over a large area) pre-assigned to histopathology image 202 by, for example, a histopathologist or another expert human annotator. Method 400 may include: at block 404, one or more processing devices extract a plurality of pixel regions of the histopathology image at multiple magnifications based on the histopathology image. For example, in some embodiments, a plurality of pixel regions (e.g., one or more pixel tiles) of histopathology image 202 may be extracted and downsampled to multiple magnifications (e.g., low magnification "Nx", low magnification "Mx", low magnification "Yx" up to high magnification "Zx", where "N", "M", "Y", and "Z" each include an integer greater than 1).

[0058] Method 400 may include: at block 406, for each of the plurality of extracted pixel regions, one or more processing devices input the pixel region into a machine learning model that is trained to generate a prediction of a tile-level class label for the pixel region based on the pixel region and the slide-level class label, and output from the machine learning model a prediction of the class label for the pixel region. For example, in certain embodiments, as discussed above with respect to Figure 2 a plurality of pixel regions (e.g., one or more pixel tiles) may then be input into respective machine learning models 206A, 206B, 206C, and 206D that are trained to utilize the plurality of respective pixel regions (e.g., one or more pixel tiles) and the slide-level class label pre-assigned to histopathology image 202 to generate respective tile-level class label predictions 208A, 208B, 208C, and 208D for each of the plurality of pixel regions (e.g., one or more pixel tiles). Method 400 may include: at block 408, one or more processing devices generate a prediction of one or more tile-level class labels for the histopathology image based on the predictions of the class labels for each of the plurality of extracted pixel regions. For example, in certain embodiments, the tile-level class label predictions 208A, 208B, 208C, and 208D may be averaged to generate one or more tile-level class label predictions 214 for histopathology image 202.

[0059] In this manner, the presently disclosed embodiments can provide predicted tile-level class labels for low-level features within a histopathology image (e.g., identifying specific regions of cancer cells, immune cell regions, class labels or boundary geometries for one or more biomarkers or other clinically interesting feature regions corresponding to specific regions) using only the corresponding extracted and downsampled pixel regions (e.g., one or more pixel tiles) and the sparse slide-level class labels pre-assigned to the histopathology image. In fact, in the absence of the presently disclosed embodiments, such tile-level annotation tasks would otherwise require time-consuming, expensive, and potentially error-prone manual annotation. Additionally, by training a machine learning model (e.g., a machine learning model ensemble) to predict tile-level class labels at different magnifications, once trained, the machine learning model (e.g., a machine learning model ensemble) may be more suitable for predicting tile-level class labels for histopathology image features at different magnifications and / or resolutions (e.g., in a manner similar to how a histopathologist would analyze and classify histopathology image features).

[0060] Figure 5 FIG. 4 shows a flowchart of a method 500 according to presently disclosed embodiments for generating predictions of one or more tile-level class labels for a histopathology image using one or more machine learning models based on slide-level class labels pre-assigned to the histopathology image. Method 500 can be performed using one or more processing devices (e.g., one or more of the networks in network 100 of the interactive computer system discussed above with respect to Figure 1 which can include hardware (e.g., a general-purpose processor, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a microcontroller, a field-programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a vision processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), a deep learning processor (DLP), a tensor processing unit (TPU), a neuromorphic processing unit (NPU), a wafer-scale engine (WSE), or any one of various hardware AI accelerators) that can be suitable for processing various omics data and making one or more decisions based thereon, software (e.g., instructions running / executing on one or more processing devices), firmware (e.g., microcode), or some combination thereof.

[0061] The method may include: at block 502, one or more processing devices access a histopathology image. For example, in one embodiment, one or more processing devices may access an unannotated histopathology image. Method 400 may include: at block 404, one or more processing devices input the histopathology image into one or more machine learning models that are trained to generate predictions of one or more tile-level class labels for the histopathology image. For example, in one embodiment, one or more processing devices may input an unannotated histopathology image into a machine learning model ensemble (e.g., a CNN ensemble or other similar image classification neural network ensemble) that is trained to generate predictions of one or more tile-level class labels for the histopathology image (e.g., class labels or boundary geometries that identify specific regions of cancer cells, immune cell regions, one or more biomarkers corresponding to specific regions, or other regions of features of clinical interest).

[0062] Method 500 may include: at block 506, one or more processing devices output predictions of one or more tile-level class labels for the histopathology image via one or more machine learning models. For example, as previously described, one or more machine learning models (e.g., a CNN ensemble or other similar image classification neural network ensemble) may generate predictions of one or more tile-level class labels for the histopathology image (e.g., class labels or boundary geometries that identify specific regions of cancer cells, immune cell regions, one or more biomarkers corresponding to specific regions, or other regions of features of clinical interest).

[0063] Figure 6 Run instance 600 according to currently disclosed embodiments is shown, which utilizes one or more machine learning models to generate predictions of one or more tile-level class labels for a histopathology image based on a slide-level class label pre-assigned to the histopathology image. Specifically, in one embodiment, run instance 600 may be an illustrative embodiment of method 500 as discussed above with respect to Figure 5 As depicted, in certain embodiments, an unannotated histopathology image 602 may be accessed. In certain embodiments, the unannotated histopathology image 602 may then be input into one or more trained machine learning models 604 (e.g., a CNN ensemble or other similar image classification neural network ensemble) that are trained to generate predictions of one or more tile-level class labels for the unannotated histopathology image 602 according to currently disclosed embodiments.

[0064] For example, in one embodiment, an unannotated histopathology image 602 can be input into a trained machine learning model ensemble (e.g., a CNN ensemble or other similar image classification neural network ensemble) to generate and output one or more tile-level class label predictions for the unannotated histopathology image 602 (e.g., class labels or boundary geometries that identify specific regions of cancer cells, immune cell regions, one or more biomarkers corresponding to specific regions, or other clinically interesting feature regions). The histopathology image 606 shows the predicted class labels (e.g., pixel-level class labels that identify tumors) that can be output by one or more trained machine learning models 604.

[0065] Figure 7A and Figure 7B show one or more graphs or implementation examples of predicting tile-level class labels for histopathology images at different magnifications according to the presently disclosed embodiments. For example, in one embodiment, the annotated histopathology image 702 can include a 2x magnification, the annotated histopathology image 704 can include a 5x magnification, the annotated histopathology image 706 can include a 10x magnification, and the annotated histopathology image 708 can include a 20x magnification. As Figure 7A depicted, according to the presently disclosed embodiments, each of the annotated histopathology images 702, 704, 706, and 708 can include the predicted tile-level class labels (e.g., shown by the first boundary geometry 750 and the second boundary geometry 752). As Figure 7B depicted, the annotated histopathology image 710 shows the final prediction of one or more tile-level class labels (e.g., shown by the first boundary geometry 750 and the second boundary geometry 752), which is generated based on averaging the predicted tile-level class labels corresponding to the annotated histopathology images 702, 704, 706, and 708.

[0066] Figure 8Shows example machine learning model evaluation graph 800 (e.g., area under the curve (AUC) (802) and F1 score (804)) according to the presently disclosed embodiments. As shown, the machine learning model evaluation graph 800 (e.g., ROC / AUC 802 and 804) can include indications of the corresponding prediction accuracies of the predicted tile-level class labels corresponding to annotated histopathology images 702 (e.g., for 2x magnification, AUC value is approximately 0.90 and F1 score is approximately 0.99), 704 (e.g., for 5x magnification, AUC value is approximately 0.85 and F1 score is approximately 0.92), 706 (e.g., for 10x magnification, AUC value is approximately 0.82 and F1 score is approximately 0.90), and 708 (e.g., for 20x magnification, AUC value is approximately 0.78 and F1 score is approximately 0.87), e.g., as compared to the prediction accuracy of the final prediction of one or more tile-level class labels (e.g., for the averaged prediction, AUC value is approximately 0.99 and F1 score is approximately 1.00) generated based on averaging the predicted tile-level class labels corresponding to annotated histopathology images 702, 704, 706, and 708. Thus, according to the presently disclosed techniques, the prediction accuracy of the final prediction of one or more tile-level class labels can be significantly improved as compared to the prediction accuracy of any of the predicted tile-level class labels corresponding to annotated histopathology images 702, 704, 706, and 708.

[0067] Figure 9 Shows an example artificial intelligence (AI) architecture 902 (which can be included as described above with respect to Figure 1FIG. 900 of a portion of one or more networks in the network 100 of the interactive computer system under discussion, the exemplary artificial intelligence architecture can be used to train one or more machine learning models to generate predictions of one or more tile-level class labels for histopathology images based on slide-level class labels pre-assigned to the histopathology images. In some embodiments, the AI architecture 902 can be implemented using, for example, one or more processing devices, which can include hardware (e.g., general-purpose processor, graphics processing unit (GPU), application-specific integrated circuit (ASIC), system-on-chip (SoC), microcontroller, field-programmable gate array (FPGA), central processing unit (CPU), application processor (AP), vision processing unit (VPU), neural processing unit (NPU), neural decision processor (NDP), deep learning processor (DLP), tensor processing unit (TPU), neuromorphic processing unit (NPU), wafer-scale engine (WSE), or any one of various hardware artificial intelligence (AI) accelerators) that can be suitable for processing various omics data and making one or more decisions based thereon, software (e.g., instructions running / executing on one or more processing devices), firmware (e.g., microcode), or some combination thereof.

[0068] In some embodiments, as depicted by Figure 9 the AI architecture 902 can include a machine learning (ML) model 904, a natural language processing (NLP) model 906, an expert system 908, a computer-based vision model 910, a speech recognition model 912, a planning model 914, and a robotics model 916. In some embodiments, the ML model 904 can include any statistical-based algorithm that can be suitable for finding patterns in large amounts of data (e.g., "big data", such as genomics data, proteomics data, metabolomics data, metagenomics data, transcriptomics data, and / or other omics data). For example, in some embodiments, the ML model 904 can include a deep learning algorithm 918, a supervised learning algorithm 920, and an unsupervised learning algorithm 922.

[0069] In some embodiments, the deep learning algorithm 918 may include any artificial neural network (ANN) that can be used to learn deep representations and abstract concepts from large amounts of data. For example, the deep learning algorithm 918 may include ANNs such as perceptrons, multi-layer perceptrons (MLPs), autoencoders (AEs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTMs), gated recurrent units (GRUs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q-networks, neural autoregressive distribution estimators (NADEs), adversarial networks (ANs), attention models (AMs), spiking neural networks (SNNs), deep reinforcement learning, etc.

[0070] In some embodiments, the supervised learning algorithm 920 may include any algorithm that can be used to apply, for example, what has been learned in the past to new data using labeled examples to predict future events. For example, starting from the analysis of a known training data set, the supervised learning algorithm 920 may produce an inference function to make predictions about output values. The supervised learning algorithm 620 may also compare its output with the correct and expected output and discover errors in order to modify the supervised learning algorithm 920 accordingly. On the other hand, the unsupervised learning algorithm 922 may include, for example, any algorithm that can be applied when the data used to train the unsupervised learning algorithm 922 is neither classified nor labeled. For example, the unsupervised learning algorithm 922 may study and analyze how a system infers a function that describes hidden structures from unlabeled data.

[0071] In some embodiments, the NLP model 906 may include any algorithm or function that can be suitable for automatically manipulating natural language such as speech and / or text. For example, in some embodiments, the NLP model 906 may include a content extraction model 924, a classification model 926, a machine translation model 928, a question answering (QA) model 930, and a text generation model 932. In some embodiments, the content extraction model 924 may include means for extracting text or images from electronic documents (such as web pages, text editor documents, etc.) for use in, for example, other applications.

[0072] In some embodiments, classification model 926 can include any algorithm that can learn from the data inputs of a supervised learning model (e.g., logistic regression, naive Bayes, stochastic gradient descent (SGD), k-nearest neighbor, decision tree, random forest, support vector machine (SVM), etc.) and make new observations or classifications accordingly. Machine translation model 928 can include any algorithm or function that can be suitable for automatically converting source text in one language, for example, into text in another language. QA model 930 can include any algorithm or function that can be suitable for automatically answering questions posed by humans in, for example, natural language, such as the algorithms or functions performed by a voice-controlled personal assistant device. Text generation model 932 can include any algorithm or function that can be suitable for automatically generating natural language text.

[0073] In some embodiments, expert system 908 can include any algorithm or function that can be suitable for simulating the judgment and behavior of a human or organization with expertise and experience in a particular field (e.g., stock trading, medicine, sports statistics, etc.). Computer-based vision model 910 can include any algorithm or function that can be suitable for automatically extracting information from images (e.g., photo images, video images). For example, computer-based vision model 910 can include image recognition algorithm 934 and machine vision algorithm 936. Image recognition algorithm 934 can include any algorithm that can be suitable for automatically identifying and / or classifying objects, locations, people, etc. that may be included in, for example, one or more image frames or other display data. Machine vision algorithm 936 can include any algorithm that can be suitable for allowing a computer to "see" or, for example, acquire images using an image sensor camera with dedicated optical elements for processing, analyzing, and / or measuring various data characteristics for decision-making purposes.

[0074] In some embodiments, speech recognition model 912 can include any algorithm or function that can be suitable for recognizing spoken language and translating it into text, such as through automatic speech recognition (ASR), computer speech recognition, speech-to-text (STT) 938, or text-to-speech (TTS) 940, for example, in order to compute communication with one or more users via speech. In some embodiments, planning model 914 can include any algorithm or function that can be suitable for generating a sequence of actions, where each action can include its own set of preconditions to be satisfied before performing the action. Examples of AI planning can include classical planning, reduction to other problems, temporal planning, probabilistic planning, preference-based planning, conditional planning, etc. Finally, robot model 916 can include any algorithm, function, or system that can enable one or more devices to replicate human behavior through, for example, movement, posture, performing tasks, decision-making, emotions, etc.

[0075] As used herein, "or" is inclusive and not exclusive, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A or B" means "A, B, or both", unless expressly stated otherwise or the context indicates otherwise. Further, as used herein, "and" is both conjunctive and disjunctive, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "A and B, jointly or severally", unless expressly stated otherwise or the context indicates otherwise.

[0076] As used herein, "automatically" and derivatives thereof mean "without human intervention", unless expressly indicated otherwise or the context indicates otherwise.

[0077] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited thereto. The embodiments according to the present disclosure are particularly disclosed in the appended claims directed to methods, storage media, systems, and computer program products, wherein any feature (e.g., a method) recited in one claim category may be claimed for protection in another claim category (e.g., a system). The dependencies or references in the appended claims are chosen only for formal reasons. However, any subject matter resulting from intentionally referencing any of the foregoing claims (in particular multiple dependencies) may also be claimed, and thus any combination of the claims and their features may be disclosed and claimed, regardless of the dependencies chosen in the appended claims. The subject matter that may be claimed includes not only combinations of features as recited in the appended claims, but also any other combinations of features in the claims, wherein each feature recited in the claims may be combined with any other feature or combination of features in the claims. Further, any embodiments and features described or depicted herein may be claimed in separate claims and / or in any combination with any embodiments or features described or depicted herein or with any features of the appended claims.

[0078] Exemplary Embodiment

[0079] The embodiments disclosed herein may include:

[0080] 1. A method for training one or more machine learning models to generate predictions of one or more tile - level class labels for an image, the method comprising, by one or more computing devices: accessing a histopathology image, wherein the histopathology image includes a slide - level class label; extracting a plurality of pixel regions of the histopathology image at a plurality of magnifications; and for each of the plurality of extracted pixel regions: inputting the pixel regions into a machine learning model, the machine learning model being trained to generate a prediction of the class label of the pixel regions based on the pixel regions and the slide - level class label; and outputting, by the machine learning model, the prediction of the class label of the pixel regions; and generating a prediction of one or more tile - level class labels for the histopathology image based on the predictions of the class labels of each of the plurality of extracted pixel regions.

[0081] 2. The method according to embodiment 1, wherein the prediction of the one or more tile - level class labels includes the identification of one or more biomarkers associated with the tissue or cells included within the histopathology image.

[0082] 3. The method according to embodiment 1 or embodiment 2, wherein extracting the plurality of pixel regions of the histopathology image at the plurality of magnifications includes downsampling the plurality of pixel regions to the plurality of magnifications.

[0083] 4. The method according to any one of embodiments 1 to 3, further comprising: normalizing the predictions of the class labels of each of the plurality of extracted pixel regions before generating the prediction of the one or more tile - level class labels.

[0084] 5. The method according to embodiment 4, wherein normalizing the predictions of the class labels of each of the plurality of extracted pixel regions includes normalizing the predictions of the class labels of each of the plurality of extracted pixel regions to the scaling ratio of the pixel regions at the maximum magnification.

[0085] 6. The method according to any one of embodiments 1 to 5, wherein generating the prediction of the one or more tile - level class labels includes calculating the average of the predictions of the class labels of each of the plurality of extracted pixel regions.

[0086] 7. The method according to any one of embodiments 1 to 6 further includes: after training the one or more machine learning models to generate predictions of one or more patch-level class labels for the histopathology image: accessing a second histopathology image; inputting the second histopathology image into the trained one or more machine learning models to generate predictions of one or more patch-level class labels for the second histopathology image; and outputting, by the one or more machine learning models, the predictions of the one or more patch-level class labels for the second histopathology image.

[0087] 8. The method according to embodiment 7, wherein the prediction of the one or more patch-level class labels for the second histopathology image includes the identification of one or more biomarkers associated with the tissue or cells included within the second histopathology image.

[0088] 9. The method according to any one of embodiments 1 to 8, wherein the machine learning model includes an artificial neural network (ANN), a convolutional neural network (CNN), or a deep neural network (DNN).

[0089] 10. The method according to any one of embodiments 1 to 9, wherein the machine learning model includes one CNN in a convolutional neural network (CNN) ensemble.

[0090] 11. The method according to embodiment 10, wherein the convolutional neural network (CNN) ensemble is configured to be trained simultaneously.

[0091] 12. The method according to any one of embodiments 1 to 11, wherein training the one or more machine learning models to generate the predictions of the one or more patch-level class labels for the histopathology image includes training the one or more machine learning models according to a weakly supervised learning process.

[0092] 13. The method according to any one of embodiments 1 to 12, wherein each magnification of the plurality of magnifications is different from each other magnification of the plurality of magnifications.

[0093] 14. The method according to any one of embodiments 1 to 13, wherein the histopathology image includes at least one of a histological stain image, a fluorescence in situ hybridization (FISH) image, an immunofluorescence (IF) image, or a hematoxylin and eosin (H&E) image.

[0094] 15. The method according to any one of embodiments 1 to 14 further includes generating a report based on the predictions of the one or more patch-level class labels for the histopathology image.

[0095] 16. The method according to embodiment 15, further comprising causing a human-machine interface (HMI) associated with a pathologist or a clinician to display the report.

[0096] 17. A system comprising one or more computing devices for training one or more machine learning models to generate predictions of one or more tile-level class labels for an image, the one or more computing devices comprising: one or more non-transitory computer-readable storage media comprising instructions; and one or more processors coupled to the one or more storage media, the one or more processors configured to execute the instructions to: access a histopathology image, wherein the histopathology image comprises a slide-level class label; extract a plurality of pixel regions of the histopathology image at a plurality of magnifications; and for each of the plurality of extracted pixel regions: input the pixel regions into a machine learning model trained to generate a prediction of a class label for the pixel regions based on the pixel regions and the slide-level class label; and output, by the machine learning model, the prediction of the class label for the pixel regions; and generate a prediction of one or more tile-level class labels for the histopathology image based on the predictions of the class labels for each of the plurality of extracted pixel regions.

[0097] 18. The system according to embodiment 17, wherein the prediction of the one or more tile-level class labels comprises an identification of one or more biomarkers associated with tissue or cells included within the histopathology image.

[0098] 19. The system according to embodiment 17 or embodiment 18, wherein the instructions for extracting the plurality of pixel regions of the histopathology image at the plurality of magnifications further comprise instructions for downsampling the plurality of pixel regions to the plurality of magnifications.

[0099] 20. The system according to any one of embodiments 17 to 19, wherein the instructions further comprise instructions to normalize the predictions of the class labels for each of the plurality of extracted pixel regions before generating the prediction of the one or more tile-level class labels.

[0100] 21. The system according to embodiment 20, wherein the instructions for normalizing the predictions of the class labels for each of the plurality of extracted pixel regions further comprise instructions to normalize the predictions of the class labels for each of the plurality of extracted pixel regions to a scale of the pixel regions at a maximum magnification.

[0101] 22. The system according to any one of embodiments 17 to 21, wherein the instructions for generating the prediction of the one or more patch-level class labels further include instructions for calculating an average of the predictions of the class labels for each of the plurality of extracted pixel regions.

[0102] 23. The system according to any one of embodiments 17 to 22, wherein the instructions further include the following instructions: after training the one or more machine learning models to generate predictions of one or more patch-level class labels for a whole-slide histopathology image: accessing a second histopathology image; inputting the second histopathology image into the trained one or more machine learning models to generate predictions of one or more patch-level class labels for the second histopathology image; and outputting, by the one or more machine learning models, the predictions of the one or more patch-level class labels for the second histopathology image.

[0103] 24. The system according to embodiment 23, wherein the prediction of the one or more patch-level class labels for the second histopathology image includes an identification of one or more biomarkers associated with the tissue or cells included within the second histopathology image.

[0104] 25. The system according to any one of embodiments 17 to 24, wherein the machine learning model includes an artificial neural network (ANN), a convolutional neural network (CNN), or a deep neural network (DNN).

[0105] 26. The system according to any one of embodiments 17 to 25, wherein the machine learning model includes one CNN in a convolutional neural network (CNN) ensemble.

[0106] 27. The system according to embodiment 26, wherein the convolutional neural network (CNN) ensemble is configured to be trained simultaneously.

[0107] 28. The system according to any one of embodiments 17 to 27, wherein the instructions for training the one or more machine learning models to generate the prediction of the one or more patch-level class labels for the histopathology image further include instructions for training the one or more machine learning models according to a weak supervision learning process.

[0108] 29. The system according to any one of embodiments 17 to 28, wherein each magnification of the plurality of magnifications is different from each other magnification of the plurality of magnifications.

[0109] 30. The system according to any one of embodiments 17 to 29, wherein the histopathological image includes at least one of a histological staining image, a fluorescence in situ hybridization (FISH) image, an immunofluorescence (IF) image, or a hematoxylin and eosin (H&E) image.

[0110] 31. The system according to any one of embodiments 17 to 30, wherein the instructions further include instructions for generating a report based on the prediction of the one or more tile-level class labels of the histopathological image.

[0111] 32. The system according to embodiment 31, wherein the instructions further include instructions for causing a human-machine interface (HMI) associated with a pathologist or a clinician to display the report.

[0112] 33. A non-transitory computer-readable medium including instructions that, when executed by one or more processors of one or more computing devices, cause the one or more processors to: access a histopathological image, wherein the histopathological image includes a slide-level class label; extract a plurality of pixel regions of the histopathological image at a plurality of magnifications; and for each of the plurality of extracted pixel regions: input the pixel regions into a machine learning model that is trained to generate a prediction of a class label of the pixel regions based on the pixel regions and the slide-level class label; and output, by the machine learning model, the prediction of the class label of the pixel regions; and generate a prediction of one or more tile-level class labels of the histopathological image based on the predictions of the class labels of each of the plurality of extracted pixel regions.

[0113] 34. The non-transitory computer-readable medium according to embodiment 33, wherein the prediction of the one or more tile-level class labels includes an identification of one or more biomarkers associated with tissue or cells included within the histopathological image.

[0114] 35. The non-transitory computer-readable medium according to embodiment 33 or embodiment 34, wherein the instructions for extracting the plurality of pixel regions of the histopathological image at the plurality of magnifications further include instructions for downsampling the plurality of pixel regions to the plurality of magnifications.

[0115] 36. The non-transitory computer-readable medium according to any one of embodiments 33 to 35, wherein the instructions further include the following instructions: normalizing the predictions of the class labels of each of the plurality of extracted pixel regions before generating the prediction of the one or more tile-level class labels.

[0116] 37. The non-transitory computer-readable medium according to embodiment 36, wherein the instructions for normalizing the predictions of the class labels for each of the plurality of extracted pixel regions further include instructions for normalizing the predictions of the class labels for each of the plurality of extracted pixel regions to the scaling ratio of the pixel regions at the maximum magnification.

[0117] 38. The non-transitory computer-readable medium according to any one of embodiments 33 to 37, wherein the instructions for generating the prediction of the one or more tile-level class labels further include instructions for calculating an average value of the predictions of the class labels for each of the plurality of extracted pixel regions.

[0118] 39. The non-transitory computer-readable medium according to any one of embodiments 33 to 38, wherein the instructions further include the following instructions: after training the one or more machine learning models to generate predictions of one or more tile-level class labels for a whole-slide histopathology image: accessing a second histopathology image; inputting the second histopathology image into the trained one or more machine learning models to generate predictions of one or more tile-level class labels for the second histopathology image; and outputting, by the one or more machine learning models, the predictions of the one or more tile-level class labels for the second histopathology image.

[0119] 40. The non-transitory computer-readable medium according to embodiment 39, wherein the prediction of the one or more tile-level class labels for the second histopathology image includes an identification of one or more biomarkers associated with the tissue or cells included within the second histopathology image.

[0120] 41. The non-transitory computer-readable medium according to any one of embodiments 33 to 40, wherein the machine learning model includes an artificial neural network (ANN), a convolutional neural network (CNN), or a deep neural network (DNN).

[0121] 42. The non-transitory computer-readable medium according to any one of embodiments 33 to 41, wherein the machine learning model includes one CNN in a convolutional neural network (CNN) ensemble.

[0122] 43. The non-transitory computer-readable medium according to embodiment 42, wherein the convolutional neural network (CNN) ensemble is configured to be trained simultaneously.

[0123] 44. The non-transitory computer-readable medium according to any one of embodiments 33 to 43, wherein the instructions for training the one or more machine learning models to generate the prediction of one or more patch-level class labels for the histopathology image further include instructions for training the one or more machine learning models according to a weak supervision learning process.

[0124] 45. The non-transitory computer-readable medium according to any one of embodiments 33 to 44, wherein each magnification of the plurality of magnifications is different from each other magnification of the plurality of magnifications.

[0125] 46. The non-transitory computer-readable medium according to any one of embodiments 33 to 45, wherein the histopathology image includes at least one of a histological staining image, a fluorescence in situ hybridization (FISH) image, an immunofluorescence (IF) image, or a hematoxylin and eosin (H&E) image.

[0126] 47. The non-transitory computer-readable medium according to any one of embodiments 33 to 46, wherein the instructions further include instructions for generating a report based on the prediction of the one or more patch-level class labels for the histopathology image.

[0127] 48. The non-transitory computer-readable medium according to embodiment 47, wherein the instructions further include instructions for causing a human-machine interface (HMI) associated with a pathologist or a clinician to display the report.

[0128] The scope of the present disclosure covers all changes, substitutions, variations, alterations, and modifications to the exemplary embodiments described or illustrated herein that would be understood by a person of ordinary skill in the art. The scope of the present disclosure is not limited to the exemplary embodiments described or illustrated herein. Additionally, although the corresponding embodiments herein are described and illustrated as including specific components, elements, features, functions, operations, or steps, any one of these embodiments may include any combination or arrangement of any components, elements, features, functions, operations, or steps described or illustrated anywhere herein that would be understood by a person of ordinary skill in the art. Further, in the appended claims, a reference to a device or system or a component of a device or system being adapted to, arranged to, capable of, configured to, enabled to, operable to, or operating to perform a particular function covers the device, system, component, whether or not the particular function is activated, turned on, or unlocked, so long as the device, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operating. Additionally, although the present disclosure describes or illustrates certain advantages provided by certain embodiments, certain embodiments may not provide, partially provide, or provide all of these advantages.

Claims

1. A method for training one or more machine learning models to generate predictions of one or more tile-level class labels for an image, the method comprising, by one or more computing devices: Accessing a histopathology image, wherein the histopathology image includes a slide-level class label; Extracting, based on the histopathology image, a plurality of pixel regions of the histopathology image at a plurality of magnifications ； And For each of the plurality of extracted pixel regions: Inputting the pixel region into a machine learning model, the machine learning model being trained to generate a prediction of the class label of the pixel region based on the pixel region and the slide-level class label; And Outputting, by the machine learning model, the prediction of the class label of the pixel region; And Generating a prediction of one or more tile-level class labels for the histopathology image based on the prediction of the class label of each of the plurality of extracted pixel regions.

2. The method according to claim 1, wherein the prediction of the one or more tile-level class labels includes an identification of one or more biomarkers associated with tissue or cells included within the histopathology image.

3. The method according to claim 1 or claim 2, wherein extracting the plurality of pixel regions of the histopathology image at the plurality of magnifications includes downsampling the plurality of pixel regions to the plurality of magnifications.

4. The method according to any one of claims 1 to 3, further Comprising: Normalizing the prediction of the class label of each of the plurality of extracted pixel regions before generating the prediction of the one or more tile-level class labels.

5. The method according to claim 4, wherein normalizing the prediction of the class label of each of the plurality of extracted pixel regions includes normalizing the prediction of the class label of each of the plurality of extracted pixel regions to the scaling ratio of the pixel region at the maximum magnification.

6. The method according to any one of claims 1 to 5, wherein generating the prediction of the one or more tile-level class labels includes calculating an average of the predictions of the class label of each of the plurality of extracted pixel regions.

7. The method according to any one of claims 1 to 6, further Comprising: After training the one or more machine learning models to generate predictions of one or more tile-level class labels for the histopathology image: Accessing a second histopathology image; Inputting the second histopathology image into the trained one or more machine learning models to generate predictions of one or more tile-level class labels for the second histopathology image; And Outputting, by the one or more machine learning models, the predictions of the one or more tile-level class labels for the second histopathology image.

8. The method according to claim 7, wherein the prediction of the one or more tile-level class labels of the second histopathological image includes identifying one or more biomarkers associated with the tissue or cells included within the second histopathological image.

9. The method according to any one of claims 1 to 8, wherein the machine learning model includes an artificial neural network (ANN), a convolutional neural network (CNN), or a deep neural network (DNN).

10. The method according to any one of claims 1 to 9, wherein the machine learning model includes one CNN in a convolutional neural network (CNN) ensemble.

11. The method according to claim 10, wherein the convolutional neural network (CNN) ensemble is configured to be trained simultaneously.

12. The method according to any one of claims 1 to 11, wherein training the one or more machine learning models to generate the prediction of the one or more tile-level class labels of the histopathological image includes training the one or more machine learning models according to a weak supervision learning process.

13. The method according to any one of claims 1 to 12, wherein each magnification of the plurality of magnifications is different from each other magnification of the plurality of magnifications.

14. The method according to any one of claims 1 to 13, wherein the histopathological image includes at least one of a histological stained image, a fluorescence in situ hybridization (FISH) image, an immunofluorescence (IF) image, or a hematoxylin and eosin (H&E) image.

15. The method according to any one of claims 1 to 14, further comprising generating a report based on the prediction of the one or more tile-level class labels of the histopathological image.

16. The method according to claim 15, further comprising causing a human-machine interface (HMI) associated with a pathologist or a clinician to display the report.

17. A system, comprising one or more computing devices for training one or more machine learning models to generate predictions of one or more tile-level class labels of an image, the one or more computing devices comprising: one or more non-transitory computer-readable storage media, which include instructions; and one or more processors coupled to the one or more storage media, the one or more processors being configured to execute the instructions to: access a histopathological image, wherein the histopathological image includes a slide-level class label; extract a plurality of pixel regions of the histopathological image at a plurality of magnifications based on the histopathological image; and for each of the plurality of extracted pixel regions: input the pixel region into a machine learning model, the machine learning model being trained to generate a prediction of a class label of the pixel region based on the pixel region and the slide-level class label; and output, by the machine learning model, the prediction of the class label of the pixel region; and Based on the prediction of the class label for each of the plurality of extracted pixel regions, a prediction of one or more tile-level class labels for the histopathology image is generated.

18. The system according to claim 17, wherein the prediction of the one or more tile-level class labels includes the identification of one or more biomarkers associated with the tissue or cells included within the histopathology image.

19. The system according to claim 17 or claim 18, wherein the instructions for extracting the plurality of pixel regions of the histopathology image at the plurality of magnifications further include instructions for downsampling the plurality of pixel regions to the plurality of magnifications.

20. The system according to any one of claims 17 to 19, wherein the instructions further include instructions for: Normalizing the prediction of the class label for each of the plurality of extracted pixel regions prior to generating the prediction of the one or more tile-level class labels.

21. The system according to claim 20, wherein the instructions for normalizing the prediction of the class label for each of the plurality of extracted pixel regions further include instructions for normalizing the prediction of the class label for each of the plurality of extracted pixel regions to the scaling ratio of the pixel region at the maximum magnification.

22. The system according to any one of claims 17 to 21, wherein the instructions for generating the prediction of the one or more tile-level class labels further include instructions for calculating the average value of the prediction of the class label for each of the plurality of extracted pixel regions.

23. The system according to any one of claims 17 to 22, wherein the instructions further include instructions for: After training the one or more machine learning models to generate a prediction of one or more tile-level class labels for a whole-slide histopathology image: Accessing a second histopathology image; Inputting the second histopathology image into the trained one or more machine learning models to generate a prediction of one or more tile-level class labels for the second histopathology image; And Outputting, by the one or more machine learning models, the prediction of the one or more tile-level class labels for the second histopathology image.

24. The system according to claim 23, wherein the prediction of the one or more tile-level class labels for the second histopathology image includes the identification of one or more biomarkers associated with the tissue or cells included within the second histopathology image.

25. The system according to any one of claims 17 to 24, wherein the machine learning model includes an artificial neural network (ANN), a convolutional neural network (CNN), or a deep neural network (DNN).

26. The system according to any one of claims 17 to 25, wherein the machine learning model includes one CNN in a convolutional neural network (CNN) ensemble.

27. The system according to claim 26, wherein the convolutional neural network (CNN) ensemble is configured to be trained simultaneously.

28. The system according to any one of claims 17 to 27, wherein the instructions for training the one or more machine learning models to generate the prediction of the one or more patch-level class labels for the histopathological image further include instructions for training the one or more machine learning models according to a weakly supervised learning process.

29. The system according to any one of claims 17 to 28, wherein each magnification of the plurality of magnifications is different from each other magnification of the plurality of magnifications.

30. The system according to any one of claims 17 to 29, wherein the histopathological image includes at least one of a histological staining image, a fluorescence in situ hybridization (FISH) image, an immunofluorescence (IF) image, or a hematoxylin and eosin (H&E) image.

31. The system according to any one of claims 17 to 30, wherein the instructions further include instructions for generating a report based on the prediction of the one or more patch-level class labels for the histopathological image.

32. The system according to claim 31, wherein the instructions further include instructions for causing a human-machine interface (HMI) associated with a pathologist or a clinician to display the report.

33. A non-transitory computer-readable medium including instructions that, when executed by one or more processors of one or more computing devices, cause the one or more processors to: access a histopathological image, wherein the histopathological image includes a slide-level class label; extract a plurality of pixel regions of the histopathological image at a plurality of magnifications based on the histopathological image ； and for each of the plurality of extracted pixel regions: input the pixel region into a machine learning model that is trained to generate a prediction of a class label for the pixel region based on the pixel region and the slide-level class label; and output, by the machine learning model, the prediction of the class label for the pixel region; and generate a prediction of one or more patch-level class labels for the histopathological image based on the prediction of the class label for each of the plurality of extracted pixel regions.

34. The non-transitory computer-readable medium according to claim 33, wherein the prediction of the one or more patch-level class labels includes an identification of one or more biomarkers associated with tissue or cells included within the histopathological image.

35. The non-transitory computer-readable medium according to claim 33 or claim 34, wherein the instructions for extracting the plurality of pixel regions of the histopathological image at the plurality of magnifications further include instructions for downsampling the plurality of pixel regions to the plurality of magnifications.

36. The non-transitory computer-readable medium according to any one of claims 33 to 35, wherein the instructions further include instructions for performing the following: Before generating the prediction of the one or more tile-level class labels, normalizing the prediction of the class label for each of the plurality of extracted pixel regions.

37. The non-transitory computer-readable medium according to claim 36, wherein the instructions for normalizing the prediction of the class label for each of the plurality of extracted pixel regions further include instructions for normalizing the prediction of the class label for each of the plurality of extracted pixel regions to the scaling ratio of the pixel region at the maximum magnification.

38. The non-transitory computer-readable medium according to any one of claims 33 to 37, wherein the instructions for generating the prediction of the one or more tile-level class labels further include instructions for calculating the average value of the prediction of the class label for each of the plurality of extracted pixel regions.

39. The non-transitory computer-readable medium according to any one of claims 33 to 38, wherein the instructions further include instructions for performing the following: After training the one or more machine learning models to generate predictions of one or more tile-level class labels for a whole-slide histopathology image: Access a second histopathology image; Input the second histopathology image into the trained one or more machine learning models to generate predictions of one or more tile-level class labels for the second histopathology image; And Output, by the one or more machine learning models, the predictions of the one or more tile-level class labels for the second histopathology image.

40. The non-transitory computer-readable medium according to claim 39, wherein the prediction of the one or more tile-level class labels for the second histopathology image includes the identification of one or more biomarkers associated with the tissue or cells included within the second histopathology image.

41. The non-transitory computer-readable medium according to any one of claims 33 to 40, wherein the machine learning model includes an artificial neural network (ANN), a convolutional neural network (CNN), or a deep neural network (DNN).

42. The non-transitory computer-readable medium according to any one of claims 33 to 41, wherein the machine learning model includes one CNN in a convolutional neural network (CNN) ensemble.

43. The non-transitory computer-readable medium according to claim 42, wherein the convolutional neural network (CNN) ensemble is configured to be trained simultaneously.

44. The non-transitory computer-readable medium according to any one of claims 33 to 43, wherein the instructions for training the one or more machine learning models to generate the prediction of the one or more patch-level class labels for the histopathology image further include instructions for training the one or more machine learning models according to a weak supervision learning process.

45. The non-transitory computer-readable medium according to any one of claims 33 to 44, wherein each magnification of the plurality of magnifications is different from each other magnification of the plurality of magnifications.

46. The non-transitory computer-readable medium according to any one of claims 33 to 45, wherein the histopathology image includes at least one of a histological staining image, a fluorescence in situ hybridization (FISH) image, an immunofluorescence (IF) image, or a hematoxylin and eosin (H&E) image.

47. The non-transitory computer-readable medium according to any one of claims 33 to 46, wherein the instructions further include instructions for generating a report based on the prediction of the one or more patch-level class labels for the histopathology image.

48. The non-transitory computer-readable medium according to claim 47, wherein the instructions further include instructions for causing a human-machine interface (HMI) associated with a pathologist or a clinician to display the report.

Images