Histological staining patterns and artifact classification using few-shot learning

Abstract

This paper provides a method and system for classifying field-of-view (FOV) images of histological slides into various categories, including certain staining patterns, artifacts, and / or other target features. Few-shot learning (e.g., prototype networks) techniques are used to train a deep convolutional neural network with a small number of training samples for a few image categories to classify stained images belonging to a majority of image categories.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 069,421, filed on August 24, 2020, which is incorporated herein by reference in its entirety for all purposes. Background Technology

[0003] Histology is the microscopic study of animal and plant cells, tissues, or other biological samples through sectioning, staining, and examination under an electron or optical microscope. Histological studies are used in forensic investigations, autopsies, diagnosis, and teaching. Histology is also widely used in the medical field to facilitate diagnosis, prognosis prediction, and treatment selection.

[0004] Staining techniques are widely used in histology, as well as in histopathology, hematology, and cytopathology, to enhance the contrast of specific types of biological objects in biological samples at the microscopic level. For example, hematoxylin and eosin (H&E) staining is used as a diagnostic stain to assess tissue morphology, nuclear, and cytoplasmic characteristics in tissue sections. The staining process typically involves adding class-specific dyes (e.g., for DNA, proteins, lipids, carbohydrates, etc.) to sections of a sample to qualitatively or quantitatively determine the presence of specific compounds, structures, or features. For example, staining agents can help identify or highlight organelles within biological tissues (e.g., muscle fibers or connective tissue), cell populations (e.g., different blood cells), or individual cells. Biological staining can also be used to label cells in flow cytometry, and to label proteins or nucleic acids in gel electrophoresis.

[0005] In addition, immunohistochemical (IHC) staining is a staining technique used to identify proteins in cells within tissue sections. It is therefore widely used to study different types of cells, such as cancer cells and immune cells in biological tissues. Thus, IHC staining can be used to study the distribution and localization of differentially expressed biomarkers of immune cells (such as T cells or B cells) in cancerous tissues, for use in immune response studies. For example, tumors may contain infiltrations of immune cells, which may inhibit tumor development or promote tumor growth.

[0006] Histological staining involves a series of procedures to prepare a sample (e.g., tissue) for microscopic study by staining each of one or more sections of a sample with one or more histological staining agents. The histological staining process typically includes fixation, processing, embedding, sectioning, and staining. During each stage of the histological staining process, various types of artifacts or other artificial effects may be introduced into the sample. Artifacts and other artificial effects can affect the degree to which various biological objects can be detected and / or characterized (e.g., the number of identifiable objects, the size of one or more objects, and / or the shape of one or more objects). Therefore, artifacts can affect the extent to which pathological images can influence diagnosis, prognosis, and / or treatment selection. Summary of the Invention

[0007] Some embodiments of this disclosure relate particularly to biological sample preparation and analysis, such as digital pathology. More specifically, techniques are disclosed herein for classifying field-of-view (FOV) images of histological slides into various categories, including certain staining patterns, artifacts, and / or other target features. According to some embodiments, few-shot learning and transfer learning techniques are used to train a deep convolutional neural network (e.g., a prototype network) using a small number of training samples depicting a small number of feature categories (e.g., different types of artifacts). The trained deep convolutional neural network can then be used to classify stained images belonging to a large number of feature categories. The techniques disclosed herein outperform existing methods based solely on transfer learning techniques. The techniques disclosed herein can be used to classify cell types or other target biological objects (e.g., mitosis, debris, blood vessels, etc.) in FOV images. Various embodiments are described herein, including apparatuses, systems, modules, materials, methods, non-transitory computer-readable storage media (which store programs, code, or instructions executable by one or more processors), etc.

[0008] According to some embodiments, a computer-implemented method is provided for training a machine learning model for classifying stained images or other images. The computer-implemented method may include accessing a pre-trained machine learning model. The pre-trained machine learning model may include a set of parameter values ​​corresponding to a parameter set, wherein the set of parameter values ​​has been learned using a first training dataset. The first training dataset may include multiple images and corresponding classification sets, and each classification in the classification set may characterize a depiction in a corresponding image among the multiple images. In some cases, for at least 50% of each of the multiple images, the image depicts a macroscopic object.

[0009] The computer-implemented method may also include access to a digital pathology image set and a corresponding artifact classification set. Each digital pathology image in the digital pathology image set can depict a stained section of a sample and may also include artifacts. In some cases, the digital pathology images in the digital pathology image set depict tissue sections stained using IHC or H&E staining protocols. For each digital pathology image in the digital pathology image set, the digital pathology image can depict microscopic objects.

[0010] Each artifact classification in the artifact classification set indicates the type of artifact corresponding to the artifact, and the classification set of the first training dataset may differ from the artifact classification set. In some cases, the number of artifact classifications in the artifact classification set is less than the number of classifications in the first training dataset. In some cases, a specific artifact classification in the artifact classification set indicates that the corresponding digital pathology image does not contain artifacts.

[0011] The computer-implemented method may further include using a few-shot learning approach to further train a pre-trained machine learning model using a set of digital pathology images and a corresponding set of artifact classifications. Further training of the pre-trained machine learning model can generate a new set of parameter values ​​for the parameter set. In some cases, further training of the pre-trained machine learning model includes generating a similarity metric for each artifact classification in the artifact classification set using new digital pathology images and a representation of the artifact classification. The representation of the artifact classification may be generated based on a subset of the digital pathology image set associated with the artifact classification.

[0012] The computer-implemented method may also include receiving new digital pathology images and processing them using a further trained machine learning model to generate an output predicting that the new digital pathology images include specific types of artifacts. In some cases, the artifact classification set corresponding to the set of digital pathology images used to further train the pre-trained machine learning model may not include specific types of artifacts.

[0013] According to some embodiments, a computer-implemented method is provided for classifying a query image using a machine learning model and a support image set. This computer-implemented method may include obtaining a machine learning model, a support image set for a first stained image category within a first stained image category set, and an unclassified query image. The support image set for the first stained image category may include common types of stained artifacts. The machine learning model may include a VGG network, an Inception network, a Residual Neural Network (ResNet), a Dense Convolutional Network (DenseNet), or a DenseNet-121 network.

[0014] The computer-implemented method may also include generating a corresponding embedding for each supporting image in the supporting image set based on a machine learning model.

[0015] The computer-implemented method may further include computing a prototype of a first stained image category based on embeddings of a supporting image set. The prototype of the first stained image category may include embeddings representing the first stained image category. The computer-implemented method may also include generating embeddings of unclassified query images based on a machine learning model.

[0016] The computer-implemented method may also include determining a similarity measure between the embedding of the unclassified query image and the prototype of a first-colored image category. The similarity measure may include Manhattan distance, Euclidean distance, Chebyshev distance, Hamming distance, or cosine similarity. In some cases, the embedding and prototype are encoded in a multidimensional embedding space, and the distance between the embedding encoded in the multidimensional embedding space and the prototype is identified.

[0017] The computer-implemented method may further include classifying unclassified query images based on a similarity metric. Classifying unclassified query images may include determining that the similarity metric is greater than a predetermined threshold. In response to determining that the similarity metric is greater than the predetermined threshold, the unclassified query image may be classified as associated with an image category different from a first image category. Unclassified query images may also be classified based on a set of similarity metrics. A corresponding similarity metric may be determined between the embedding of the unclassified query image and the prototype of each remaining stained image category in the first stained image category set within the similarity metric set.

[0018] The computer-implemented method may further include tuning the parameters of the machine learning model. This tuning includes accessing training images belonging to a second set of stained image categories, selecting a subset of stained image categories from the second set of stained image categories, and adjusting the parameters based on the results of classifying the query image set within each stained image category in the subset of stained image categories. In some cases, the first stained image category is not included in the second set of stained image categories. The number of stained image categories in the subset of stained image categories may be less than the number of stained image categories in the first set of stained image categories.

[0019] To classify a query image set to tune the parameters of a machine learning model, a similarity metric can be calculated between the embeddings of the query image set and the prototypes of a subset of stained image categories. In some cases, the prototype of each stained image category in the subset of image categories is calculated based on embeddings generated from a set of supporting images selected from the stained image category. The prototype of the first stained image category can be the mean, median, or center of the embeddings of the supporting image set.

[0020] Adjusting the parameters of a machine learning model can include adjusting the parameters of the fully connected layers of a pre-trained deep neural network, which can be trained using images that do not include stained images.

[0021] Some embodiments of this disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein. Some embodiments of this disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein.

[0022] The terms and expressions used are descriptive rather than restrictive, and their use is not intended to exclude any equivalents of the features shown and described or portions thereof. However, it should be recognized that various modifications are possible within the scope of the claimed systems and methods. Therefore, it should be understood that although the claimed systems and methods of the invention have been specifically disclosed by way of example and optional features, those skilled in the art will recognize modifications and variations to the concepts disclosed herein, and consider such modifications and variations to be within the scope of the systems and methods defined by this application.

[0023] This invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to define the scope of the claimed subject matter. References to these illustrative examples are not intended to limit or restrict this disclosure, but rather to provide examples to aid in understanding it. Additional embodiments and examples are discussed in the detailed description, and further description is provided therein. The subject matter should be understood by referring to appropriate portions of the entire specification and any or all of the accompanying drawings.

[0024] The foregoing, along with other features and embodiments, will become clearer with reference to the following description and accompanying drawings. Attached Figure Description

[0025] The features, embodiments, and advantages of this disclosure will be better understood when the following detailed description is read with reference to the accompanying drawings.

[0026] Figure 1 shows an example of a histological staining process.

[0027] Figures 2A to 2F show examples of histopathological images.

[0028] Figure 3 shows an example of a convolutional neural network (CNN) used for classifying images or other objects.

[0029] Figure 4 illustrates an example of a machine learning model for classifying stained images according to certain embodiments.

[0030] Figure 5 includes a flowchart illustrating an example of a method for classifying a query image using a machine learning model and a supporting image set, according to certain embodiments.

[0031] Figure 6 includes a flowchart illustrating an example of a method for training a machine learning model for classifying stained images or other images according to certain embodiments.

[0032] Figure 7 shows an example of a t-sne visualization of the output of a machine learning model according to certain embodiments.

[0033] Figure 8 shows an example of a t-sne visualization of the output of a machine learning model according to certain embodiments.

[0034] Figure 9 includes a graph showing the accuracy of machine learning models trained using few-shot learning and different numbers of support images according to certain embodiments.

[0035] Figure 10 shows an example of a schematic diagram for classifying images of tissue sections depicting H&E staining, according to certain embodiments.

[0036] Figure 11 shows an example of a target region generated by processing an H&E staining image according to certain embodiments.

[0037] Figures 12A and 12B illustrate examples of confusion matrices according to certain embodiments, which respectively identify the performance results of trained machine learning models used to classify stained images.

[0038] Figure 13 shows an exemplary set of image results classified using a trained machine learning model according to some embodiments.

[0039] Figure 14 illustrates an example of a computer system used to implement some of the embodiments disclosed herein. Detailed Implementation

[0040] The techniques disclosed herein generally relate to biological sample preparation and analysis. More specifically, this document discloses techniques for classifying field-of-view (FOV) images of histological slides into various categories, including certain staining patterns, artifacts, and other target features. According to some embodiments, few-shot learning and transfer learning techniques are used to train a machine learning model (e.g., a prototype network) using a small number (e.g., less than 100, less than 50, less than 25, or less than 15, such as about 5 to 10) of training samples belonging to each of a small number of feature categories (e.g., specific proteins, different staining patterns, artifacts, and other target features). The trained machine learning model can then be used to classify stained images belonging to a large number of feature categories. Various embodiments of the invention are described herein, including apparatuses, systems, materials, methods, non-transitory computer-readable storage media (which store programs, code, or instructions executable by one or more processors), etc.

[0041] Histological staining is widely used in digital pathology workflows to highlight target features (e.g., tumor cell nuclei stained with H&E, target proteins stained with IHC) in tissue or cells on histological slides and to enhance the contrast of these features. For example, in samples containing tumors, staining can help to effectively locate pathological cells and their microenvironment, thereby facilitating more accurate medical diagnosis. In another instance, IHC staining, such as 3,3'-diaminobenzidine (DAB) staining, can be used to identify the presence or distribution of biomarkers within tissue sections depicted on a sample slide. The identified biomarkers can then be analyzed to facilitate, for example, the diagnosis of tumor tissue. In some cases, IHC is used to identify different specific proteins considered antigens (e.g., programmed death-ligand 1 (PD-L1)), primary antibodies (e.g., anti-PD-L1 antibodies), and secondary antibodies. However, stained sample sections on histological slides can have various types of problems that hinder the information that can be conveyed. For example, stained sections may be torn or folded, and / or the slide may include one or more watermarks. Additionally, a portion of the stained slide image may not be optimal (e.g., blurry). This type of problem is an instance of "artifacts," where a portion of the image depicts something that does not represent the sample. To accurately analyze slide images, it may be desirable to detect such artifacts or other human effects and, where possible, perform image processing to improve the accuracy with which the image conveys information about the subject for interpretation (e.g., improving pathological diagnosis, prognosis, and / or treatment options). For this purpose, digital pathology images are currently frequently evaluated by human pathologists to assess (e.g., image-, slide-, and / or overall) quality before analyzing images of stained sample slides (e.g., to detect and / or characterize specific biological objects or biomarkers in the stained sample slides). If the quality of the stained sample slides is poor, the corresponding digital pathology image may be discarded from digital pathology analyses performed on a given subject. However, assessing staining quality and detecting artifacts in stained images can be both subjective and time-consuming.

[0042] Machine learning techniques, such as convolutional neural network (CNN) models, have been successfully applied to image classification, object detection and recognition, and video action recognition. CNN models effectively achieve spatial locality and weight sharing by reusing local filters to extract features from local regions (also known as receptive fields). To obtain an accurate CNN model, a large number of samples are typically used to train the neural network. However, for digital pathology, universally labeled datasets may not be available, or building such datasets may be both expensive and time-consuming, given the need for expertise to manually analyze and annotate (e.g., classify and label) images.

[0043] According to some embodiments, few-shot learning techniques are used to train a machine learning model (e.g., a neural network) to classify FOV images of histological slides into various image categories, including certain staining patterns, artifacts, and other target features. The machine learning model may include a convolutional neural network (e.g., a deep neural network, such as a dense convolutional network) comprising one or more fully connected layers. For each of multiple (e.g., M) image categories corresponding to different feature (e.g., artifact) types, the few-shot learning techniques and a small number of training samples (e.g., stained images) can be used to train one or more fully connected layers of the machine learning model in multiple iterations (called stages).

[0044] In one instance, during each training iteration, K (K<=M) image categories can be selected from M image categories (e.g., randomly or pseudo-randomly). From the training images belonging to each of the K image categories, the training images can be separated (e.g., using random or pseudo-random techniques) into N “support” images and Q “query” images. The number of support images (N) can be equal to the number of samples in the few-shot learning technique being implemented, and can be (e.g.) less than 200, less than 100, less than 50, less than 25, less than 20, or less than 15 (e.g., approximately 5 to 10). For each of the K image categories, each of the N support images can be passed to a machine learning model that encodes the support image into a corresponding embedding (e.g., a multidimensional vector) in a multidimensional embedding space (also referred to herein as “multidimensional space”). For each image category, the average (or median) of the N embeddings of the N support images can be computed to determine the prototype representing that image category. Alternatively, the same machine learning model can encode each of the Q query images into a corresponding embedding. A similarity measure (e.g., distance or similarity) between the embedding of the query image and the prototype of each image category in multidimensional space can be determined. The query image can then be classified into the image category belonging to the embedding whose prototype is closest to the query image. A loss (e.g., classification error) can be calculated based on the classification result of the query image. The loss can then be backpropagated to the earlier layers of the machine learning model to adjust the parameters of the machine learning model, such as the parameters of fully connected layers (e.g., weights). The above training process (stages) can be performed iteratively using, for example, an Adam optimizer with a certain learning rate.

[0045] The techniques disclosed herein can be used to train machine learning models (e.g., deep convolutional neural networks, such as prototype neural networks) for image classification using a small number of training samples from a small number of known image categories. Therefore, the techniques disclosed herein may be applicable to situations where only a limited number of annotated samples are available.

[0046] Knowledge learned from a limited number of training samples in a few image categories can be transferred to classify query samples in other image categories, such as digital pathology images with artifacts that are not in the few image categories used to train the machine learning model during the training phase. For example, a trained machine learning model can be used to identify the presence of new categories (e.g., new biomarkers, new biological objects, other types of artifacts) in a query image by computing relationship scores (e.g., distance or similarity) between the query image and a small number of samples in each new image category, without retraining the machine learning model.

[0047] Furthermore, the techniques disclosed herein outperform existing methods based solely on transfer learning in terms of classification accuracy, and can therefore be used to more accurately identify stained images with artifacts or other defects, thereby improving the accuracy of sample analysis and diagnosis. The techniques disclosed herein can be further used to qualitatively or quantitatively determine the quality of stained images. The techniques disclosed herein can also be used to classify cell types (e.g., tumor cells, fetal cells in maternal plasma) or other target biomarkers (e.g., PD-L1, alpha-fetoprotein, cytokeratin) in FOV images.

[0048] As used herein, the term "neural network" refers to one or more computer-implemented networks that can be trained to achieve a goal. Unless otherwise stated, references to neural networks herein include a single neural network or multiple interconnected neural networks trained together. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), fully connected neural networks, encoder neural networks (e.g., "encoders"), decoder neural networks (e.g., "decoders"), densely connected neural networks, and other types of neural networks. In some embodiments, neural networks may be implemented using dedicated hardware (e.g., GPUs, Tensor Processing Units (TPUs), systolic arrays, single instruction multiple data (SIMD) processors, etc.), software code and general-purpose processors, or a combination of dedicated hardware and software code.

[0049] As used herein, the term "layer" or "network layer" refers to the analysis phase in a neural network. Layers perform different types of analysis depending on the type of neural network. For example, layers in an encoder can perform different types of analysis on the input image to encode it. In some cases, a particular layer provides features based on the specific analysis performed by that layer. In some cases, a particular layer downsamples the received image. Additional layers perform additional downsampling. In some cases, each round of downsampling reduces the visual quality of the output image but provides features based on the relevant analysis performed by that layer.

[0050] As used in this paper, the term "embedding" refers to a low-dimensional vector into which a high-dimensional vector can be encoded or transformed. In the context of neural networks, an embedding is a vector that maps a discrete (e.g., categorical) object or variable (e.g., an image or a word) to a continuous low-dimensional space to represent that discrete object. In some cases, an embedding might be a feature vector or feature map extracted from a large input feature map by a neural network or encoder. Embeddings can reduce the dimensionality of categorical variables and meaningfully represent categories in the transformed space. Embeddings can be used to find nearest neighbors in the embedding space. Embeddings can be used for cluster-based recommendations, can be used as input to machine learning models for supervised tasks, and can be used to visualize concepts and relationships between categories.

[0051] The following examples are provided to illustrate certain embodiments. In the following description, specific details are set forth for illustrative purposes to provide a thorough understanding of the examples of this disclosure. However, it will be apparent that various embodiments may be practiced without these specific details. For example, apparatus, systems, structures, components, methods, and other parts may be shown as parts in block diagram form to avoid obscuring the examples with unnecessary detail. In other instances, well-known apparatus, processes, systems, structures, and techniques may be shown without the need for necessary detail to avoid obscuring the examples. The drawings and descriptions are not intended to be limiting. The terms and expressions used in this disclosure are used as descriptive rather than limiting terms, and in using such terms and expressions, no equivalents of any features shown and described or portions thereof are intended to be excluded. The term “example” as used herein means “serving as an example, instance, or illustration.” Any embodiment or design described herein as an “example” is not necessarily to be construed as superior to or better than other embodiments or designs.

[0052] I. Tissue preparation and analysis

[0053] Histological staining is widely used to highlight target features and enhance the contrast of tissue or cells in biological sample sections. In samples containing tumors, histological staining can help effectively locate pathological cells and their microenvironment, thereby facilitating better medical diagnosis and treatment. For example, dyes can be applied to the posterior and anterior boundaries of the tissue sample to locate lesions or tumor cells or other pathological cells. In biological research, staining can be used to label cells and label nucleic acids, proteins, or for gel electrophoresis to aid in microscopic examination. The process of histological staining may include several stages, such as fixation, processing, embedding, sectioning, staining, and imaging.

[0054] In some embodiments, IHC staining of tissue sections is a histological staining used to identify the presence of proteins in a biological sample. For example, the expression level of a specific protein (e.g., an antigen) is determined by: (a) performing IHC analysis on a tumor sample with a specific antibody type; and (b) determining the presence and / or expression level of the protein in the sample. In some embodiments, the IHC staining intensity is determined relative to a reference determined from a reference sample (e.g., a control cell line stained sample, a tissue sample from a non-cancer patient, or a reference sample known to have a predetermined protein expression level).

[0055] Figure 1 illustrates an example of a histological staining procedure 100. Stage 110 of the histological staining procedure 100 may include tissue fixation, which can be used to preserve tissue and slow tissue degradation. In histology, fixation generally refers to an irreversible process using chemicals to retain chemical components, preserve natural tissue structure, and prevent cellular structure degradation. Fixation may also harden cells or tissue for sectioning. Fixatives can use cross-linked proteins to enhance tissue and cell preservation. Fixatives may bind to and cross-link some proteins and denature others through dehydration, which may harden the tissue and inactivate enzymes that might otherwise degrade it. Fixatives can also kill bacteria. Fixatives can be applied, for example, by perfusion and infiltration of the prepared tissue. Various fixatives can be used, including Bouin fixatives and formaldehyde fixatives such as neutral buffered formalin (NBF) and paraffin-formaldehyde (PFA). While fixation can be used to preserve the structure of tissues and cells for histological research purposes, tissue fixation may result in the concealment of tissue antigens, thus reducing antigen detection. Therefore, tissue fixation is generally considered a limiting factor for IHC because formalin can cross-link antigens and mask epitopes. In some cases, additional processes are performed to reverse the effects of cross-linking, including treating formalin-fixed tissue with citral anhydride (a reversible protein cross-linking agent) and heating.

[0056] Stage 120 of the histological staining process 100 may include tissue preparation and embedding. Tissue preparation may bring animal or human tissue to a state of complete infiltration with a suitable histological wax (such as paraffin) after fixation. Histological wax may be insoluble in water or alcohol, but soluble in paraffin solvents such as xylene. Therefore, water in the tissue may need to be replaced with xylene. For this purpose, the tissue may first be dehydrated by gradually replacing the water in the sample with alcohol, which can be achieved by passing the tissue through an increasing concentration of ethanol (e.g., from 0% to about 100%). After replacing water with alcohol, xylene, which is miscible with alcohol, can be used instead of alcohol. Tissue embedding may involve embedding the tissue in warm paraffin. Because paraffin is soluble in xylene, the molten wax may fill the space filled with xylene and previously filled with water. The wax-filled tissue may be cooled to form a hardened block, which can be clamped into a microtome for sectioning. In some cases, deviation from the above example procedure may cause paraffin to penetrate, thereby inhibiting the penetration of antibodies, chemicals, or other fixatives.

[0057] Stage 130 of the histological staining process 100 may include tissue sectioning. Tissue sectioning is the process of cutting thin slices of a sample from an embedded block for the purpose of fixing it onto a microscope slide for examination. A microtome can be used to perform tissue sectioning. In some cases, tissue may be rapidly frozen in dry ice or isopentane and then cut with a cold knife in a refrigerator (e.g., a cryostat). Other types of coolants may be used to freeze tissue, such as liquid nitrogen. Tissue sections used for optical microscopy are typically about 4 µm to 10 µm thick. In some cases, sections may be embedded in epoxy or acrylic resin, allowing for the cutting of thinner sections (e.g., < 2 µm). The tissue sections may be placed on a microscope slide.

[0058] Stage 140 of the histological staining process 100 may include staining of tissue sections. The purpose of staining is to identify different tissue components through color reactions. Most cells are colorless and transparent. Therefore, it may be necessary to stain tissue sections to make cells visible. The staining process typically involves adding a dye or staining agent to a sample of the section to identify or quantify the presence of a specific compound, structure, molecule, or feature (e.g., subcellular features). For example, staining may help identify or highlight specific biomarkers (e.g., PD-L1) from an image of a tissue section. In other instances, staining agents may be used to identify or highlight organelles within biological tissues (e.g., muscle fibers or connective tissue), cell populations (e.g., different blood cells), or individual cells. Many staining solutions are aqueous. Therefore, to stain sections, it may be necessary to dissolve the wax and replace it with water (rehydration) before applying the staining solution to the sections. For example, sections may be passed sequentially through xylene, reduced concentrations of ethanol (from about 100% to 0%), and water. Once stained, the sections may be dehydrated again and placed in xylene. The slides can then be mounted on a microscope slide containing a mounting medium dissolved in xylene. A coverslip can be placed on top to protect the sample slides. Evaporation of xylene around the edges of the coverslip may dry the mounting medium and firmly bond the coverslip to the slide.

[0059] Various types of staining protocols can be used to perform tissue staining. For example, IHC staining protocols include: using a hydrophobic barrier line around the tissue section to prevent reagent leakage from the slide during incubation; treating the tissue section with reagents to block endogenous sources of nonspecific staining (e.g., enzymes, free aldehydes, immunoglobulins, other irrelevant molecules that mimic specific staining); incubating the tissue section with permeation buffer to promote the penetration of antibodies and other staining reagents into the tissue; incubating the tissue section with primary antibody at a specific temperature (e.g., room temperature, 6°C to 8°C) for a period of time (e.g., 1 hour to 24 hours); rinsing the tissue section with washing buffer; incubating the tissue section with secondary antibody at another specific temperature (e.g., room temperature) for another period of time; rinsing the tissue section again with water buffer; incubating the rinsed tissue section with chromogen (e.g., DAB); and washing away the chromogen to stop the reaction. In some cases, counterstaining is subsequently used to identify the entire “landscape” of the tissue section and as a reference for the primary color used to detect tissue targets. Examples of counterstaining agents may include hematoxylin (staining from blue to purple), methylene blue (staining blue), toluidine blue (staining cell nuclei dark blue and polysaccharides from pink to red), nuclear solid red (also known as Kernechtrot dye, staining red), and methyl green (staining green); non-nuclear staining agents, such as eosin (staining pink), etc. Those skilled in the art will recognize that other IHC staining techniques can be performed to stain tissue sections.

[0060] In another example, H&E staining protocols can be used to stain tissue sections. H&E staining protocols involve applying a hematoxylin staining agent or mordant mixed with a metal salt to the tissue section. The tissue section can then be rinsed in a weakly acidic solution to remove excess staining (differentiation), and then turned blue in weakly alkaline water. After applying hematoxylin, the tissue can be counterstained with eosin. It should be understood that other H&E staining techniques can be performed.

[0061] In some embodiments, various types of staining agents can be used to perform tissue staining, depending on the target feature. For example, DAB can be used for various tissue sections stained with IHC, where DAB produces a brown color that depicts the target feature in the stained image. In another instance, alkaline phosphatase (AP) can be used for IHC-stained skin tissue sections because the DAB color may be masked by melanin. Regarding primary staining techniques, suitable staining agents may include, for example, basophilic and eosinophilic staining agents, heme and hematoxylin, silver nitrate, trichrome staining agents, etc. Acidic dyes react with cationic or basic components in tissues or cells, such as proteins and other components in the cytoplasm. Basic dyes react with anionic or acidic components in tissues or cells, such as nucleic acids. As mentioned above, one example of a staining system is H&E. Eosin may be a negatively charged pink acidic dye, and hematoxylin may be a purple or blue basic dye, which includes hematoxylin and aluminum ions. Other examples of dyes may include periodic acid-Schiff reaction (PAS) dyes, Masson's tricolor dyes, Alsin blue dyes, Vangelson dyes, reticular fiber dyes, etc. In some embodiments, different types of dyes may be used in combination.

[0062] Stage 150 of the histological staining process 100 may include medical imaging to examine the stained tissue in the slide using an electron or optical microscope. In some cases, the examination may include a gross examination prior to microscopic examination. A gross examination can help a pathologist identify cells or tissue with a mass (which may be cancerous), and microscopic examination can be used to confirm this. Microscopy magnifies images of tissue to reveal its details. For example, the resolution of an optical microscope may be less than 1 µm, such as about a few hundred nanometers. An electron microscope can be used to observe finer details in the nanometer or sub-nanometer range.

[0063] At stage 160, images of stained sections can be analyzed and results reported. This analysis can be qualitative or quantitative. In histopathology, stained images can be analyzed to identify disease signs, understand the biological mechanisms of disease processes, and more. For example, immunohistochemistry (IHC) histology can be used to detect the presence of specific protein markers that can help in accurate tumor classification and diagnosis. In cytology, stained images can be used to study cell structure, function, and chemistry; the presence of isolated cells and cell clusters; and the absence of more complex structures such as glands.

[0064] Due to the presence of various artifacts, the aforementioned histological staining processes may introduce various alterations to normal morphological and cytological features. As used herein, the term "artifact" can refer to artificial structural or tissue alterations in a prepared microscope slide due to external factors. For example, artifacts may occur during surgical resection, fixation, tissue processing, embedding, sectioning, staining, and mounting. Artifacts in histopathology may include, for example, prefixation artifacts, fixation artifacts, bone-related artifacts, tissue processing artifacts, sectioning artifacts, flotation and mounting artifacts, staining artifacts, mounting artifacts, biological artifacts, imaging artifacts, etc.

[0065] For example, prefixation artifacts may include injection artifacts, compression artifacts (e.g., tissue tearing or wrinkling), electrocautery artifacts, starch artifacts, autolysis artifacts, etc. Fixation artifacts may include, for example, formalin pigments, mercury pigments, ice crystal artifacts, freezing artifacts, flow artifacts, etc. Artifacts related to floating and mounting may include, for example, folds and wrinkles in the cross-section, contaminants, air bubbles, etc. Staining artifacts may include, for example, residual wax, artifacts related to the addition of acetic acid to eosin, artifacts caused by the corrosion of hematoxylin, artifacts caused by the fluorescent luster of hematoxylin, etc. Mounting artifacts may include, for example, residual water and air bubbles.

[0066] Figures 2A through 2F show examples of histopathological staining images. Figure 2A shows an example of a tissue image stained with hematoxylin. Figure 2B shows an example of an image of tissue with a watermark. Figure 2C shows an example of an image of tissue with membrane / cytoplasmic staining. Figure 2D shows an example of a staining image with nonspecific staining of connective fibers or matrix components. Figure 2E shows an example of an image with nuclear staining. Figure 2F shows an example of an image with tissue folds.

[0067] Artifacts can result in low image quality of stained sample sections, potentially leading to misdiagnosis or delayed diagnosis. For example, multiple artifacts present in images of stained samples (e.g., defocusing, watermarks, and tissue folds) can potentially obscure diagnostic features. These artifacts can even render the tissue completely useless. Accurate analysis of slide images may require detecting such artifacts or other human influences in the slide images and, if possible, processing the slide images in a way that prevents artifacts and other human influences from interfering with pathological diagnosis. However, assessing the quality of stained images and detecting artifacts in stained images can be both subjective and time-consuming.

[0068] II. Machine Learning Models

[0069] Machine learning techniques, such as CNN models, have been successfully applied to image classification, object detection and recognition, and video action recognition. CNN models effectively achieve spatial locality and weight sharing by reusing local filters to extract features from local regions. Convolutional neural networks can perform operations including, for example, convolution, nonlinear (or activation) functions (e.g., ReLU), pooling or subsampling, and classification. Different CNNs may have different combinations of these main operations, as well as other additional operations. For example, a residual neural network (ResNet) may include network layers that primarily contain convolutional layers and a small number of pooling layers, and may also perform residual summation operations for residual learning.

[0070] Figure 3 illustrates an example of a CNN 300 used for image or other object classification. As described above, the CNN 300 performs operations including convolution, non-linear (or activation) functions, pooling or subsampling, and classification. The object 310 to be classified, such as one or more input images or other input datasets (called input feature maps), can be represented by a matrix of pixel values. For example, object 310 can include multiple channels (e.g., multiple input feature maps), each channel representing a component of object 310. For example, a color image from a digital camera or microscope can have red, green, and blue channels, where each channel can be represented by a pixel matrix with pixel values ​​ranging from, for example, 0 to 255 (i.e., 8 bits). A grayscale image can have only one channel. In the following description, the processing of a single image channel using the CNN 300 is described. Other channels can be processed similarly.

[0071] As shown in Figure 3, object 310 (e.g., input image) may first be processed by a first convolutional layer 315 using a first set of filters, wherein the first convolutional layer 315 may perform convolution between a matrix representing the input image and a matrix representing each filter in the first set of filters. Convolution may include multiple matrix multiplications. The first convolutional layer 315 may also perform a non-linear activation function (e.g., ReLU). The output matrix 320 from the first convolutional layer 315 may have a smaller dimension than the input image. The first convolutional layer 315 may perform convolution on the input image using the first set of filters to generate multiple output matrices 320, which may be referred to as output feature maps of the first convolutional layer 315. The number of filters used may be referred to as the depth of the convolutional layer. In the example shown in Figure 3, the depth of the first convolutional layer 315 may be three. Each output matrix 320 (e.g., output feature map) may be passed to a pooling layer 325, wherein each output matrix 320 may be quadratized or downsampled to generate matrix 330.

[0072] Each matrix 330 can be processed by a second convolutional layer 335 using a second set of filters. As described above, a non-linear activation function (e.g., ReLU) can also be performed by the second convolutional layer 335. The output matrix 340 (e.g., output feature map) from the second convolutional layer 335 can have a smaller dimension than matrix 330. The second convolutional layer 335 can perform convolution on matrix 330 using the second set of filters to generate multiple output matrices 340. In the example shown in Figure 3, the depth of the second convolutional layer 335 can be six. Each output matrix 340 can be passed to a pooling layer 345, where each output matrix 340 can be quadratized or downsampled to generate an output matrix 350.

[0073] The output matrix 350 from pooling layer 345 can be flattened into a vector by flattening layer 355. The flattened vector can be passed through fully connected layer 360 (e.g., a multilayer perceptron (MLP)). Fully connected layer 360 may include input layer 370, which obtains the output vector from flattening layer 355. Fully connected layer 360 may also include hidden layer 380 and output layer 390. Fully connected layer 360 can use feature maps or output matrix 350 and, for example, a softmax function to classify objects in the input image into one of several categories. The operations of fully connected layers can be represented by matrix multiplication. For example, if there are M nodes on input layer 370 and N nodes on hidden layer 380, the weights of the connections between the M nodes on input layer 370 and the N nodes on hidden layer 380 can be represented by matrix W, which includes M × N elements, and the output Y of hidden layer 380 can be determined by Y = X × W.

[0074] Convolution operations in CNNs can be used to extract features from input images. Convolution preserves the spatial relationships between pixels by extracting image features using small regions of the input image. In convolution, matrices (called filters, kernels, or feature detectors) slide across the input image (or feature map) with a stride (called stride). For each location (or step), element-wise multiplications between the filter matrix and the overlapping matrix in the input image are computed and summed to generate a final value representing a single element in the output matrix (e.g., the feature map). Filters can be used to detect certain features from the original input image.

[0075] Convolving an input pixel array with a single filter (or a filter bank) produces one feature map, while convolving the same input pixel array with a different filter (or another filter bank) generates different feature maps. CNNs can learn the filter weights themselves during training based on user-specified parameters (called hyperparameters), such as the number of filters, filter size, and network architecture. Specific numbers and / or filter sizes can be determined for generating feature maps to improve image feature extraction and pattern recognition in new images.

[0076] The size of the output feature map can be determined based on parameters such as depth, stride, and zero-padding. As mentioned above, the depth of a convolutional layer can correspond to the number of filters (or filter banks) used for the convolution operation. For example, in the CNN300 shown in Figure 3, three different filters are used in the first convolutional layer 315 to perform a convolution operation on the input image, resulting in three different output matrices 320 (or feature maps). The stride is the number of pixels that the filter matrix slides across as it moves across the input pixel array. For example, with a stride of one, the filter matrix moves one pixel at a time. With a stride of two, the filter matrix moves two pixels at a time. A larger stride may result in a smaller feature map. In some implementations, the input matrix can be padded with zeros around its boundaries so that the filter matrix can be applied to the boundary elements of the input pixel array. Zero-padding controls the size of the feature map.

[0077] As shown in Figure 3, an additional nonlinear operation can be applied after each convolution operation, using an activation function (e.g., ReLU). ReLU is an element-wise operation that replaces all negative pixel values ​​in the feature map with zeros. The purpose of the ReLU operation is to introduce nonlinearity into the CNN. Other nonlinear functions, such as tanh or sigmoid, can also be used, but ReLU has been found to perform better in many cases.

[0078] Spatial pooling (also known as quadratic sampling or downsampling) reduces the dimensionality of each feature map while preserving some important information. Specifically, pooling makes feature dimensions smaller and easier to manage, and reduces the number of parameters and computations in the network. Spatial pooling can be performed in different ways, such as max pooling, average pooling, sum pooling, etc. In max pooling, the spatial neighborhood is represented by the maximum element in each spatial neighborhood (e.g., a 2×2 window). Instead of taking the maximum element, the spatial neighborhood can be represented by the average of all elements in each window (for average pooling) or the sum (for sum pooling).

[0079] In the example shown in Figure 3, two sets of convolutional and pooling layers are used. It should be noted that these operations can be repeated any number of times within a single CNN. Furthermore, pooling layers are not necessarily used after each convolutional layer. For example, in some implementations, the CNN may perform multiple convolutional and ReLU operations before performing pooling operations.

[0080] The training process of a convolutional neural network (such as CNN 300) is similar to that of any feedforward neural network. First, all parameters and weights (including weights in filters and fully connected layers) are initialized with random values ​​or the parameters of the known neural network. Second, the convolutional neural network takes training samples (e.g., training images) as input, performs a forward propagation step (including convolution, non-linear activation, and pooling operations, along with forward propagation operations in fully connected layers), and determines the output probability for each possible class. The training samples can be classified using the Softmax function based on the output probabilities. The classification result of the convolutional neural network on the training samples can be compared with the expected result, and the error between the expected result and the classification output can be propagated back to the previous layer. The weights can be adjusted accordingly based on the error. For example, techniques such as backpropagation can be used to calculate the error gradient relative to the parameters to be trained in the network, and gradient descent can be used to update the parameters to be trained in the network to minimize the output error. This process can be repeated a certain number of times, for a certain number of training samples, or until the output error is below a predetermined threshold.

[0081] At the end of the training process, all parameters of the CNN are likely optimized to correctly classify training samples from the training dataset. When unseen samples (e.g., test samples or new samples) are fed into the CNN, the CNN can go through a forward propagation step and use the trained parameters to output the probability of each class; this can be called the inference (or prediction) process compared to the training process. If the training dataset is large enough, the trained network can classify unseen samples into the correct category.

[0082] One example of a CNN is the Dense Convolutional Network (DenseNet). In DenseNet, each layer can be connected to all preceding layers in a feedforward manner, where the output feature maps of all preceding layers and the original input feature maps can be concatenated to form the input feature map of the current layer. DenseNet can use fewer parameters than an equivalent traditional CNN because it does not need to learn redundant feature maps. DenseNet can achieve high performance with less memory and computation.

[0083] III. Image Classification Using Few-Shot Learning and Transfer Learning

[0084] Typically, a large number of samples may be needed to train neural networks or other machine learning models. For digital pathology, there may not be widely labeled datasets available, or it may be both expensive and time-consuming for medical experts to manually analyze and annotate stained images to train machine learning models.

[0085] According to some embodiments, few-shot learning techniques are used to train a machine learning model (e.g., a neural network) to classify field-of-view images of histological slides into various categories, including certain staining patterns, artifacts, and other target features. The machine learning model may include a pre-trained convolutional neural network comprising one or more fully connected layers. The pre-trained convolutional neural network may include a deep neural network, such as VGG, Inception V3, ResNet (e.g., ResNet-50), DenseNet, etc., trained and published (available for download) using large benchmark datasets (e.g., CIFAR-10, CIFAR-100, SVHN, or ImageNet datasets). Layers other than the fully connected layers may not be retrained. For each of multiple (e.g., M) image categories, one or more fully connected layers of the machine learning model may be retrained in multiple iterations (called stages) using few-shot learning techniques and a small number of training samples (e.g., stained images). Knowledge learned from a finite number of training samples in M ​​image categories can be transferred to classify query samples in other categories (such as stained images with artifacts that are not in the M image categories used to train the machine learning model). For example, a trained machine learning model can be used to classify query images in new image categories by computing a relational score (e.g., embedding distance) between the query image and a small number of instances in each new image category, without retraining the machine learning model.

[0086] Figure 4 illustrates an example of a machine learning model 400 for classifying stained images of biological samples according to certain embodiments. In the illustrated example, the machine learning model 400 may include a neural network 430 (e.g., DenseNet or another deep neural network) that may include multiple convolutional layers 432 and one or more fully connected layers 434 at the ends of the neural network 430. A small number (e.g., less than about 20, such as about 5 to 15) of support images 410 (e.g., stained images) belonging to each of multiple image categories (e.g., stained images without artifacts and stained images with different types of artifacts, such as protein markers) may be encoded by the neural network 430 into corresponding embeddings (e.g., multidimensional feature vectors). Embeddings of support images belonging to the same image category may form clusters in a multidimensional space 440. The mean, median, or center of the embeddings of support images belonging to the same image category may be computed to determine the prototype representing the image category in the multidimensional space 440.

[0087] Each query image 420 to be classified (e.g., a stained image) can also be encoded into a corresponding embedding by a neural network 430. The distance or similarity metric (e.g., Manhattan, Euclidean, Chebyshev, or Hamming distance, or cosine similarity) between the embedding of the query image 420 and the prototype of each image category can be determined by a similarity metric engine 450. The query image 420 can then be classified by a prediction engine 460 based on the similarity metric between the embedding of the query image 420 and the prototype of each category. For example, the query image 420 can be classified by the prediction engine 460 into the category whose prototype is closest to the embedding of the query image 420.

[0088] The neural network 430, the similarity measurement engine 450, and the prediction engine 460 can be implemented using, for example, one or more general-purpose CPUs, GPUs, TPUs or systolic arrays, SIMD processors, etc., and instructions generated by a compiler based on a description of the machine learning model 400.

[0089] Machine learning model 400 (more specifically, neural network 430) can be trained using images from a small number of known image categories. The trained machine learning model 400 (or neural network 430) can also be used to classify query images for new image categories (e.g., those with new types of artifacts). For example, to determine whether a query image belongs to a new image category, a small number (e.g., less than 20, such as 1, 5, 10, 15, etc.) of support images for the new image category can be provided and used as input. The support images can be encoded by neural network 430 or otherwise processed to generate embeddings of the support images. A prototype representing the new image category can be determined, for example, by calculating the mean, median, or center of the embeddings of the support images. The embedding of the query image can also be extracted by neural network 430. A similarity measure of the query image relative to existing image categories and prototypes of the new image category can be determined by similarity metric engine 450. Prediction engine 460 can then use the similarity measure to determine which category the query image belongs to.

[0090] Figure 5 includes a flowchart 500 illustrating an example of a method for classifying a query image using a machine learning model and a supporting image set according to certain embodiments. The operations described in flowchart 500 can be performed by a computer system, for example, implementing a machine learning model (such as machine learning model 400). Although flowchart 500 may describe the operations as a sequential process, in various embodiments, many operations may also be performed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. Operations may have additional steps not shown in the figure. Moreover, embodiments of the method can be implemented via hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments performing the associated tasks may be stored in a computer-readable medium, such as a storage medium.

[0091] In operation 510, the computer system may receive a machine learning model, a set of support images from a first image category, and a query image. The support images and the query image may be, for example, stained images with or without artifacts. For example, support images in the first image category may include common types of artifacts described above, such as watermarks, tissue wrinkles, blurring, etc. The query image may or may not belong to the first image category. An instance of the machine learning model may be machine learning model 400. The machine learning model may be trained using training images from multiple image categories, which may or may not include the first image category. The machine learning model may be trained to generate embeddings of input images, such as stained images. The machine learning model may include a deep neural network (e.g., neural network 430, such as DenseNet) capable of extracting feature vectors or generating embeddings of input images. The machine learning model may also include a similarity metric engine (e.g., similarity metric engine 450) capable of determining a similarity metric (e.g., squared Euclidean distance) between embeddings, and a prediction engine (e.g., prediction engine 460) capable of classifying the query image based on the similarity metric associated with the query image.

[0092] In Operation 520, the computer system can generate a corresponding embedding for each supporting image in the supporting image set based on a machine learning model. For example, the supporting images can be fed into a deep convolutional neural network (such as ResNet or DenseNet), which applies various filters to the supporting images through convolution, activation, and pooling operations to extract low-dimensional feature vectors as embeddings representing the supporting images in multidimensional space.

[0093] In operation 530, the computer system may compute a prototype of a first image category. The embeddings supporting the image set may form clusters in a multidimensional space. The prototype may be, for example, the mean, median, or center of the embeddings supporting the image set, such as the center of a cluster of embeddings supporting the image set in a multidimensional space. The prototypes of the image categories used during the training of the machine learning model may be known, or may be determined using supporting images from those image categories.

[0094] In operation 540, the computer system can generate the embedding of the query image based on the same machine learning model used to generate the embedding of the supporting image. Therefore, the query image can be mapped to the same multidimensional space as the supporting image.

[0095] In operation 550, the computer system may determine a similarity measure between the embedding of the query image and a prototype of a first image category. As described above, the similarity measure may include, for example, the squared Euclidean distance or cosine similarity between two vectors in a multidimensional space. In some embodiments, a similarity measure may also be determined between the embedding of the query image and a known prototype of each other image category.

[0096] In operation 560, the computer system may classify a query image based on a similarity measure between the embedding of the query image and a prototype of a first image category and / or a similarity measure between the embedding of the query image and a known prototype of each other image category. For example, the computer system may determine that the query image is not in the first image category based on the determination that the squared Euclidean distance between the embedding of the query image and a prototype of the first image category is greater than a threshold. In another instance, the computer system may determine that the query image can be in the second image category based on the determination that the squared Euclidean distance between the embedding of the query image and a prototype of a second image category is the lowest among the prototypes of the second image category.

[0097] The machine learning models described above can be trained using few-shot learning and transfer learning techniques. The machine learning models can be based on pre-trained convolutional neural networks (CNNs). Pre-trained CNNs can include deep neural networks trained on large benchmark datasets (e.g., CIFAR-10, CIFAR-100, SVHN, or ImageNet datasets) and released (available for download), such as VGG, Inception, ResNet, DenseNet, etc. A small number of layers (such as one or more fully connected layers) of the pre-trained CNN can be retrained to classify stained images. One or more fully connected layers can be retrained using few-shot learning techniques with a small number of training images.

[0098] Figure 6 includes a flowchart 600 illustrating an example of a method for training a machine learning model for classifying stained images or other images according to certain embodiments. The operations described in flowchart 600 can be performed by a computer system, for example, including one or more CPUs, GPUs, TPUs, systolic arrays, or SIMD processors. Although flowchart 600 may describe the operations as a sequential process, in various embodiments, many operations may also be performed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. Operations may have additional steps not shown in the figure. Moreover, embodiments of the method can be implemented via hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments performing the associated tasks may be stored in a computer-readable medium, such as a storage medium.

[0099] At operation 610, the computer system may receive a pre-trained deep neural network model and training images belonging to M categories. The pre-trained convolutional deep neural network model may include deep neural networks trained on and released (available for download) using large benchmark datasets (e.g., CIFAR-10, CIFAR-100, SVHN, or ImageNet datasets), such as VGG, Inception, ResNet, DenseNet, etc. In one instance, the pre-trained deep neural network model may include a DenseNet-121 containing multiple convolutional layers, where the input to each convolutional layer may include the original input feature map and the output feature maps of all preceding layers. Because each convolutional layer has access to feature maps at different levels of preceding convolutional layers, DenseNet-121 may have better performance than conventional convolutional neural networks. Like many other neural networks, DenseNet-121 may also include, for example, one or more fully connected layers at its ends. DenseNet-121 may also include a Softmax layer. Training images may include, for example, stained images with different types of patterns and artifacts as described above. Training images can be labeled with corresponding image categories (e.g., types of patterns or artifacts). Training images can include a set of training images for each image category.

[0100] At operation 620, the computer system can randomly, pseudo-randomly, or otherwise select K categories from M categories, where K ≤ M. From the corresponding training image set of each of the selected K categories, N training images can be randomly, pseudo-randomly, or otherwise selected as support images, and Q training images can be randomly, pseudo-randomly, or otherwise selected as query images. N can be a small number, such as less than 20, for example, about 5 to 10. Q is also a small number, such as about 5 to 10. In one instance, the training image set for each category may include 20 training images, K can be 5, N can be 10, and Q can be 5. Therefore, 50 support images and 25 query images can be selected from 100 training images belonging to 5 categories.

[0101] At operation 630, for each of the K categories, the computer system can use a pre-trained deep neural network model to generate embeddings for N support images and Q query images. Each support image or query image can be fed into the pre-trained deep neural network model to generate a corresponding feature vector (i.e., embedding). In one instance, each training image may include 256 × 256 pixels. Each training image may be represented by a 2D matrix (e.g., 256 × 256) or a 3D matrix (e.g., 256 × 256 × 3 color channels). The feature vector (or embedding) of each training image may include a 1024-element vector. The embeddings of the support images can form clusters in a multidimensional space.

[0102] At operation 640, for each of the K categories, the computer system can compute a prototype representing the category in multidimensional space. For example, the prototype of a category could be the mean or median of the embeddings of the N supporting images for that category, or the center of a cluster of the embeddings of the N supporting images for that category.

[0103] At operation 650, a similarity measure between the embedding of each of the Q query images and the prototypes of the K categories can be determined. In one instance, each similarity measure among the query image similarity measures can be the squared Euclidean distance between the query image embedding and the prototype of the corresponding category. Each of the Q query images can then be classified based on the similarity measure between the query image embedding and the prototypes of the K categories. For example, the query image can be classified into the category whose prototype is closest to the embedding of the query language (e.g., with the shortest Euclidean distance).

[0104] At operation 660, based on the classification results of the Q query images, the parameters of the pre-trained deep neural network model, such as the parameters of the fully connected layers (e.g., weights), can be adjusted. As described above, backpropagation can be used to adjust the parameters of the pre-trained deep neural network model, where the gradient of the classification error relative to the parameters can be computed and used to update the parameters. In some embodiments, the classical stochastic gradient descent process can be used to update the parameters of the pre-trained deep neural network model. In some embodiments, the adaptive moment (Adam) estimation technique can be used to update the parameters of the pre-trained deep neural network model. Adam estimation is an adaptive learning rate optimization technique for training deep neural networks. Adam estimation utilizes an adaptive learning rate method to find an individual learning rate for each parameter. Adam estimation adjusts the parameter learning rate based on the average first moment (mean) and the average second moment (uncentered variance) of the gradient.

[0105] Operations 620 to 660 can be performed iteratively to optimize the parameters of the deep neural network model. In some embodiments, operations 620 to 660 may be performed a predetermined number of times. In some embodiments, operations 620 to 660 may be performed until each of the received training images in the M categories has been selected. In some embodiments, operations 620 to 660 may be performed until the classification error is below a certain threshold.

[0106] According to some embodiments, based on classification results, staining images classified into different artifact categories can be excluded, marked, or re-enacted to improve the quality of the staining images for subsequent analysis. For example, staining images with certain artifacts can be excluded in subsequent analysis. In some staining images with certain artifacts, the portions containing artifacts can be identified or marked so that these portions can be excluded in subsequent analysis. For some staining images with certain artifacts, the staining images of the stained sections may have been captured from different fields of view.

[0107] IV. Exemplary artifact classification of IHC-stained tissues

[0108] In one example, images of IHC-stained tissue with different biomarkers (such as BCL2, HER2, CD10, Ki67, etc.) are scanned at 40× magnification. The scanned images may include various staining features and artifacts. A training dataset can be organized by extracting 256 × 256 pixel blocks from the tissue images on slides and manually annotating these blocks by a pathologist. The training dataset for tissue may include stained images belonging to six image categories. The six image categories may include two staining categories (nuclear staining and membrane cytoplasmic staining), three artifact categories (tissue folds, watermarks on tissue, and nonspecific matrix staining), and one non-staining category. The training dataset may include 20 samples (images or image blocks) for each of the six categories, for a total of 120 samples. The training dataset may be divided into a support set and a query set. For example, to perform few-shot learning, 10 samples from each category may be randomly, pseudo-randomly, or otherwise selected as the support set, while the remaining 10 samples in each category may be used as the query set. In addition, 20 samples of each of the three new artifact categories (surgical anatomical ink, incision (i.e., torn tissue) and blurred areas) were used to evaluate the accuracy of the trained machine learning model.

[0109] A prototype network with a DenseNet-121 network can be used as a machine learning model. The DenseNet-121 network can be a pre-trained and / or published model (e.g., trained using CIFAR-10, CIFAR-100, SVHN, or ImageNet datasets), and only the fully connected layers of the DenseNet-121 network can be retrained using the training dataset. The final fully connected layers can include 1024 nodes and can be used to generate low-dimensional representations of the input images. This helps to overcome the overfitting problem caused by using a small number of samples. Batch normalization is used to normalize the network. A single network is used to generate embeddings for both the support images and the query images. As described above, representations of the six image categories (e.g., the prototype) are determined based on the support images, and the embeddings of the query images are evaluated using a similarity metric (e.g., Euclidean distance) to classify the query images.

[0110] During the training phase, the machine learning model can be trained in multiple iterations based on few-shot learning. Each training iteration can be called a phase. In each phase, the selected support images and query images are fed into the network, the network loss (e.g., classification error) is determined, and the error gradient is backpropagated to the fully connected layers. In each phase, K image categories can be randomly, pseudo-randomly, or otherwise selected from M image categories, where K ≤ M. For each of the K image categories, N support images of that image category can be randomly, pseudo-randomly, or otherwise selected, where N is a hyperparameter (number of samples) and can be selected based on the number of available samples. For example, N can be approximately 5 to 10. For each of the K image categories, Q query images can also be randomly, pseudo-randomly, or otherwise selected. Embeddings of the support images (e.g., d-dimensional feature vectors, where in one instance, d = 1024) can be generated by passing the support images through the machine learning model. Therefore, N embeddings can be generated for each image category. The mean, median, or center of the N embeddings for the N supporting images in each category can be calculated to determine the prototype representing that image category. To classify Q query images, each query image in the Q query images can be encoded as a query embedding. A squared Euclidean distance metric can be calculated to determine the distance from each prototype to each query embedding. Each query image can be classified into the category whose prototype is closest to the query embedding. The classification error can be backpropagated using a negative log-likelihood loss and an Adam optimizer with a fixed learning rate of 0.001 to optimize the parameters of the machine learning model. The machine learning model is trained for 10 iterations.

[0111] Figure 7 includes a graph 700 showing an example of a t-SNE visualization of the output of a machine learning model after 10 iterations of training, according to certain embodiments. Graph 700 shows the category representations (e.g., prototypes) of the six image categories mentioned above, including two staining categories (nuclear staining and membrane cytoplasmic staining), three artifact categories (tissue folds, watermarks on tissue, and nonspecific matrix staining), and one unstained category. Graph 700 also shows the embeddings of the query images. The prototype for each image category is created using 10 support images. Five query images from each image category are used for visualization. Figure 7 illustrates how the trained machine learning model can effectively cluster the embeddings of query images around the corresponding prototypes.

[0112] During the inference phase, the trained machine learning model can classify stained images into image categories used in the training phase and new image categories not used in the training phase. The new image category may correspond to, for example, a new type of artifact. The machine learning model does not need to be retrained using images from the new image category. To predict whether a query image belongs to the new image category, the trained machine learning model can generate clusters of embeddings of supporting images using a small number of supporting images of the new image category, determine a prototype representing the new image category based on the embeddings of the supporting images, encode the query image into a query embedding, and classify the query image based on a similarity measure between the query embedding and the prototype of the new image category as described above.

[0113] Figure 8 includes a graph 800 illustrating an example of a t-SNE visualization of the output of a trained machine learning model for input images of different categories, according to certain embodiments. In the example shown in Figure 8, the machine learning model is the same as that described with respect to Figure 7. The machine learning model is trained using images from the six image categories mentioned above, including two staining categories (nuclear staining and membrane cytoplasmic staining), three artifact categories (tissue folds, watermarks on tissue, and non-specific matrix staining), and one unstained category. The prototypes of these six image categories and the embeddings of the query images in these six image categories are shown in Figure 8.

[0114] Figure 8 also shows the output of the machine learning model for stained images from three new image categories that were not used during the training phase of the machine learning model. The three new image categories include surgical anatomical ink, incisions (i.e., torn tissue), and artifacts in blurred areas. Figure 8 shows the prototypes of the three new image categories determined using support images associated with them. Figure 8 also shows the embeddings of the query images for the three new image categories. Figure 8 demonstrates that, even for the three new image categories not used to train the machine learning model, the machine learning model can effectively cluster the embeddings of the query images around the corresponding prototypes.

[0115] For comparative purposes, the machine learning model based on the pre-trained DenseNet-121 model was retrained using transfer learning instead of few-shot learning. The machine learning model was retrained for 10 iterations using an iterative training process with a learning rate of 0.001. The classification accuracy of the machine learning models trained using transfer learning and different numbers of training samples, as well as the classification accuracy of the machine learning models trained using both few-shot learning and transfer learning, are shown in Table 1. The accuracy of the trained machine learning models was evaluated using a separate set of 60 test samples (10 images per class). As shown in Table 1, the machine learning model trained using transfer learning on 60 support images achieved approximately 80% accuracy. The machine learning model trained using transfer learning on 120 training samples, including both support and query images, achieved approximately 86% accuracy. The machine learning model trained using both transfer learning and few-shot learning on 120 training samples achieved approximately 96% accuracy (an improvement of approximately 10% compared to the machine learning model trained using only transfer learning).

[0116] Table 1. Comparison of Training Techniques

[0117]

[0118] Figure 9 includes a graph 900 illustrating the accuracy of machine learning models trained using few-shot learning and varying numbers of support images (sample counts) according to certain embodiments. Figure 9 shows that increasing the number of support instances (sample counts) leads to better classification accuracy. However, the accuracy curve begins to flatten when the sample count is greater than approximately 5. When the sample count is greater than approximately 15, the improvement gradually diminishes, and the accuracy may saturate at an upper limit.

[0119] Exemplary artifact classification of VH&E stained tissues

[0120] (A) Process Overview

[0121] Figure 10 illustrates an example of a schematic diagram 1000 for classifying images of tissue sections depicting H&E staining according to certain embodiments. At block 1005, the computer system receives a scanned image of H&E-stained tissue. In some embodiments, the scanned image includes one or more staining patterns, artifacts, and / or other target features. At block 1010, the computer system identifies target regions in the scanned image. In some cases, a mask is generated to isolate the target regions in the scanned image. The mask may include pixel information representing the boundaries of the target regions, such that the mask can be used to remove information surrounding the target regions in the scanned image.

[0122] Various techniques can be used to identify target regions. For example, the pixels of a scanned image, represented in an initial color space (e.g., RGB color space), can be processed to identify the pixel intensity information of the scanned image as a Hue, Saturation, and Value (HSV) color space. A series of image processing operations (e.g., masking, filtering, smoothing) can then be applied to the pixel intensity information to extract the target region. In another example, the Okapi-Chamfer matching algorithm can be applied to a scanned image to identify target regions. In this example, the target region can be identified based on distance information calculated between the feature vector representing the scanned image and the feature vectors of a training image clustered based on their feature similarity.

[0123] At box 1015, the computer system divides the target region into multiple image blocks. Each of these image blocks can then be processed to predict whether it contains artifacts. In some embodiments, the computer system applies a trained machine learning model to the multiple image blocks. The machine learning model may have been trained iteratively using few-shot learning. Regarding the exemplary classification of H&E-stained tissue, two training iterations using few-shot learning are performed to identify five image categories, including four artifact categories and one tissue category. In the first training iteration, the machine learning model is trained using training images corresponding to the tissue fold category and the handwriting category. These two image categories are more common and more distinctive than the other image categories. In the second training iteration, the machine learning model is trained using training images corresponding to the hemorrhage category and the slicer fremitus category.

[0124] After the training phase is complete, the computer system can apply the trained machine learning model to each of the multiple image blocks for classification. For example, a first set of image blocks 1020 is classified as containing at least one artifact (e.g., bleed area, handwriting), and a second set of image blocks 1025 is classified as having no artifacts. Additionally or alternatively, the computer system applies the trained machine learning model to predict a specific type of artifact or tissue depicted in each image block. For example, regarding the classification of H&E-stained tissue, representations (e.g., prototypes) of five image categories are determined based on supporting images, where each image category indicates the presence of a specific type of artifact. The five image categories include bleed category, slicer fremitus category, handwriting category, tissue fold category, and tissue category. The representations of the five image categories are evaluated using a similarity metric (e.g., Euclidean distance) for the embedding of each image block to classify the image block as having a specific type of artifact.

[0125] In some embodiments, a prototype network with a DenseNet-121 network is used as a machine learning model. The DenseNet-121 network can be a pre-trained and / or published model (e.g., trained using CIFAR-10, CIFAR-100, SVHN, or ImageNet datasets), and one or more layers (e.g., fully connected layers of the DenseNet-121 network) can be retrained using the training dataset. A single network can be used to generate embeddings for both the supporting image and the query image (e.g., multiple image patches).

[0126] At box 1030, the computer system identifies the location of one or more artifacts in an image block for each image block predicted to contain one or more artifacts. In some cases, the location of one or more artifacts includes the x and y coordinate values ​​of each pixel in the image block corresponding to the one or more artifacts.

[0127] (B) Exemplary target area

[0128] Figure 11 illustrates examples of target regions 1110, 1115, and 1120 generated by processing an H&E stained image 1105 according to certain embodiments. In some embodiments, each of the target regions 1110, 1115, and 1120 is generated by performing various image processing techniques, such as HSV conversion and the Okapi-Chamfer matching algorithm. For example, target region 1110 may correspond to an output image generated based on processing a scanned image via HSV conversion. In another instance, target region 1110 may correspond to an output image generated based on processing a scanned image via the Okapi-Chamfer matching algorithm. In yet another instance, target region 1120 may correspond to an output image generated based on processing a scanned image using a Pyris image processing system. As shown in Figure 11, the shape, size, and pixel intensity of the target regions may vary depending on the type of image processing technique used to process the scanned image.

[0129] (C) Performance evaluation of the trained machine learning model

[0130] The accuracy of the trained machine learning model can be evaluated using a test sample set. In an exemplary test scenario, the machine learning model was trained with 100 training samples for each of the five image categories mentioned above (i.e., hemorrhage, slicer fremitus, handwriting, tissue folds, and tissue). For each image category, the trained machine learning model was evaluated using a test sample of approximately 30 slide images. As shown in Table 2, the trained machine learning model achieved an average accuracy of 96.67% in predicting specific types of artifacts. Furthermore, the trained machine learning model achieved even higher accuracy (98%) in predicting whether an image (e.g., an image patch in an H&E stained image) corresponds to tissue or an artifact.

[0131] Table 2. Performance results based on 30 test samples

[0132]

[0133] Figure 12A illustrates an example of a confusion matrix 1200A according to certain embodiments, which identifies the performance results of a trained machine learning model used to classify stained images. Confusion matrix 1200A is generated based on the training and test sample sets identified in Table 2, where numbers 1 through 5 correspond to the bleeding category, slicer tremor category, handwriting category, tissue fold category, and tissue category, respectively. Each column of confusion matrix 1200A represents an instance in the predicted image category, while each row represents an instance in the actual image category. As shown in Figure 12A, the trained machine learning model achieves very high accuracy in determining whether a given image corresponds to a bleeding area, slicer tremor area, or tissue fold area. The accuracy of the trained machine learning model decreases slightly when classifying images with handwriting. However, the error rate remains relatively low (e.g., 2 out of 29 test samples corresponding to handwriting were misclassified).

[0134] In another exemplary test scenario, the machine learning model was trained with 100 training samples and 20 validation samples for each of the five image categories. For each image category, the trained machine learning model was evaluated using 15 support samples and 70 test samples. As shown in Table 3, the trained machine learning model achieved an average accuracy of 85.14% in predicting specific types of artifacts. Furthermore, the trained machine learning model achieved a relatively high accuracy (91.71%) in predicting whether an image (e.g., an image patch in an H&E stained image) corresponds to tissue or an artifact.

[0135] Table 3. Performance results based on 70 test samples

[0136]

[0137] Figure 12B illustrates an example of a confusion matrix 1200B according to certain embodiments, which identifies the performance results of a trained machine learning model used to classify stained images. Confusion matrix 1200B is generated based on the training and test sample sets identified in Table 3, where numbers 1 through 5 correspond to the bleeding category, slicer tremor category, handwriting category, tissue fold category, and tissue category, respectively. Each column of confusion matrix 1200B represents an instance in the predicted image category, while each row represents an instance in the actual image category. As shown in Figure 12B, the trained machine learning model achieves relatively high accuracy on the five image categories. However, the accuracy for the image category corresponding to the tissue fold region drops to approximately 74.3%.

[0138] (D) Exemplary artifact classification output

[0139] Figure 13 illustrates an exemplary set of image results 1300 classified using a trained machine learning model according to some embodiments. The trained machine learning model correctly classified 35 out of 39 tissue images, achieving an accuracy of 89.74%. Of the four misclassifications, image 1305 depicting a handwriting region was misclassified as a bleeding region. Two images 1310 depicting tissue sections were misclassified as handwriting regions. Image 1315 depicting a handwriting region was misclassified as a tissue fold region. As mentioned above, the trained machine learning model misclassified images with handwriting regions more frequently compared to images with other artifact types.

[0140] VI. Computing Environment

[0141] Figure 14 illustrates an example of a computer system 1400 for implementing some of the embodiments disclosed herein. The computer system 1400 may have a distributed architecture, where some components (e.g., memory and processor) are part of an end-user device, while other similar components (e.g., memory and processor) are part of a computer server. The computer system 1400 includes at least a processor 1402, memory 1404, storage device 1406, input / output (I / O) peripherals 1408, communication peripherals 1410, and an interface bus 1412. The interface bus 1412 is configured to communicate, send, and transmit data, control, and commands between the various components of the computer system 1400. The processor 1402 may include one or more processing units, such as a CPU, GPU, TPU, systolic array, or SIMD processor. Memory 1404 and storage device 1406 include computer-readable storage media, such as RAM, ROM, electrically erasable programmable read-only memory (EEPROM), hard disk drive, CD-ROM, optical storage device, magnetic storage device, electronic non-volatile computer storage device, such as Flash® memory, and other tangible storage media. Any of such computer-readable storage media may be configured to store instruction or program code embodying aspects of this disclosure. Memory 1404 and storage device 1406 also include computer-readable signal media. Computer-readable signal media include propagated data signals embodying computer-readable program code therein. Such propagated signals take any of a variety of forms, including but not limited to electromagnetic, optical, or any combination thereof. Computer-readable signal media include any computer-readable medium that is not a computer-readable storage medium but can convey, propagate, or transmit a program used in conjunction with computer system 1400.

[0142] In addition, memory 1404 includes an operating system, programs, and application programs. Processor 1402 is used to execute stored instructions and includes, for example, a logic processing unit, a microprocessor, a digital signal processor, and other processors. Memory 1404 and / or processor 1402 may be virtualized and may be hosted within another computing system, such as a cloud network or data center. I / O peripherals 1408 include user interfaces such as a keyboard, a screen (e.g., a touchscreen), a microphone, a speaker, other input / output devices, and computing components such as a graphics processing unit, serial ports, parallel ports, a universal serial bus, and other input / output peripherals. I / O peripherals 1408 are connected to processor 1402 via any port coupled to interface bus 1412. Communication peripherals 1410 are configured to facilitate communication between computer system 1400 and other computing devices via a communication network and include, for example, a network interface controller, a modem, wireless and wired interface cards, an antenna, and other communication peripherals.

[0143] While this subject matter has been described in detail with respect to specific embodiments thereof, it should be understood that modifications, variations, and equivalents of such embodiments will be readily apparent to those skilled in the art upon understanding the foregoing. Therefore, it should be understood that this disclosure is presented for illustrative purposes rather than limiting, and does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter, which will be apparent to those of ordinary skill. In fact, the methods and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions, and changes may be made to the form of the methods and systems described herein without departing from the spirit of this disclosure. The appended claims and their equivalents are intended to cover such forms or modifications that fall within the scope and spirit of this disclosure.

[0144] Unless otherwise expressly stated, it should be understood that throughout this specification, the use of terms such as “processing,” “computing,” “calculating,” “determining,” and “identifying” refers to the actions or processes by which a computing device (such as one or more computers or one or more similar electronic computing devices) manipulates or transforms data represented as physical electronic or magnetic quantities in memory, registers, or other information storage, transmission, or display devices of a computing platform.

[0145] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide a result conditioned on one or more inputs. Suitable computing devices include multipurpose microprocessor-based computing systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more embodiments of this subject matter. The teachings contained herein can be implemented in software using any suitable programming, scripting, or other type of language or combination of languages ​​for programming or configuring the computing device.

[0146] Embodiments of the methods disclosed herein can be executed in the operation of such computing devices. The order of the blocks presented in the above examples can be varied—for example, blocks can be reordered, combined, and / or broken down into sub-blocks. Some blocks or processes can be executed in parallel.

[0147] The conditional language used in this document, such as, in particular, “may,” “can,” “possibly,” “able to,” “for example,” etc., unless otherwise expressly stated or otherwise understood in the context, is generally intended to convey that certain instances include certain features, elements, and / or steps while other instances do not. Therefore, such conditional language is not generally intended to imply that one or more instances require features, elements, and / or steps in any way, or that one or more instances necessarily include logic (with or without author input or prompting) for determining whether such features, elements, and / or steps are included in or will be performed in any particular instance.

[0148] The terms “including,” “comprising,” “having,” etc., are synonyms and are used in an open-ended manner, not excluding additional elements, features, actions, operations, etc. Furthermore, the term “or” is used in its inclusive (not exclusive) sense, such that, for example, when used to connect lists of elements, the term “or” indicates one, some, or all of the elements in the list. The terms “applies to” or “configured as” used herein are intended as an open and inclusive language that does not exclude the device from being applicable to or configured to perform additional tasks or steps. Additionally, the use of “based on” is intended to be open and inclusive, because a process, step, calculation, or other action “based on” one or more enumerated conditions or values ​​may actually be based on additional conditions or values ​​beyond the enumerated items. Similarly, the use of “at least partially based on” is intended to be open and inclusive, because a process, step, calculation, or other action “at least partially based on” one or more enumerated conditions or values ​​may actually be based on additional conditions or values ​​beyond the enumerated items. The headings, lists, and numbering included herein are for ease of interpretation only and are not intended to be limiting.

[0149] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some specific implementations. The methods and processes described herein are not limited to any particular order, and the associated blocks or states may be executed in other suitable orders. For example, the blocks or states may be executed in an order different from that of a specific disclosure, or multiple blocks or states may be combined in a single block or state. Exemplary blocks or states may be executed serially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed instance. Similarly, the exemplary systems and components described herein may be configured differently from those described. For example, elements may be added to, removed from, or rearranged compared to the disclosed instance.

Claims

1. A computer-implemented method, comprising: Access a pre-trained machine learning model, the pre-trained machine learning model including a set of parameter values ​​corresponding to a parameter set, wherein the set of parameter values ​​is learned using a first training dataset, the first training dataset including multiple images and corresponding classification sets, each classification in the classification sets representing a depiction in a corresponding image among the multiple images; Access a digital pathology image set and a corresponding artifact classification set, wherein each digital pathology image in the digital pathology image set depicts a stained section of a sample and further includes artifacts, and wherein each artifact classification in the artifact classification set indicates an artifact type corresponding to the artifact, wherein the classification set of the first training dataset is different from the artifact classification set. Using the digital pathology image set and the corresponding artifact classification set, the pre-trained machine learning model is further trained using few-shot learning, wherein the further training generates a new set of parameter values ​​for the parameter set; Receive new digital pathology images; The new digital pathology images are processed using a further trained machine learning model to generate an output that predicts the new digital pathology images to include specific types of artifacts; Identify the portions of the new digital pathology image that depict the specific type of artifact; as well as The portion of the new digital pathology image is excluded from subsequent digital pathology analysis.

2. The method of claim 1, wherein the artifact classification set does not include the specific type of artifact.

3. The method of claim 1 or 2, wherein using few-shot learning to further train the pre-trained machine learning model comprises generating a similarity metric for the new digital pathology image and a representation of the artifact classification for each artifact classification in the artifact classification set, the representation being generated based on a subset of the digital pathology image set associated with the artifact classification.

4. The method according to claim 1 or 2, wherein the digital pathology image corresponding to the specific artifact classification indicator in the artifact classification set does not contain artifacts.

5. The method according to claim 1 or 2, wherein the number of artifact classifications in the artifact classification set is less than the number of classifications in the first training dataset.

6. The method of claim 1 or 2, wherein the digital pathology images in the digital pathology image set depict tissue sections stained using an immunohistochemical (IHC) staining protocol, and wherein the tissue sections corresponding to the digital pathology images include one or more biomarkers.

7. The method according to claim 1 or 2, wherein the digital pathology images in the digital pathology image set depict tissue sections stained using a hematoxylin and eosin (H&E) staining scheme, and wherein the tissue sections corresponding to the digital pathology images comprise one or more biological objects.

8. The method of claim 1 or 2, wherein for each of at least 50% of the plurality of images, the images depict macroscopic objects, and wherein for each digital pathology image in the digital pathology image set, the digital pathology image depicts microscopic objects.

9. A system for classifying field images of histological slides, comprising: One or more data processors; as well as A non-transitory computer-readable storage medium comprising instructions that, when executed on the one or more data processors, cause the one or more data processors to perform the method according to any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium storing computer-executable instructions, wherein the instructions, when executed by one or more processing means, cause the one or more processing means to perform the method according to any one of claims 1 to 8.

Images