Adaptive learning framework for digital pathology

Abstract

The present disclosure relates to techniques for efficient development of an initial model and efficient model update and / or adaptation to different image domains using an adaptive learning framework. In order to realize efficient development of an initial model, the following two steps of development strategies can be carried out: stage 1, model pre-processing, in which an artificial intelligence system utilizes existing annotated data sets, and learning skills are improved by training the data sets; and stage 2: target model training in which the artificial intelligence system extends itself to different image domains (target domains) using the learning skills learned from stage 1, where a smaller number of annotations are required in the target domains as compared to conventional learning methods. In order to efficiently perform model update and adaptation to a new data set after initial model development, a digital pathology scene is identified, an adaptive learning method is selected based on the scene, and the model is updated and adapted to the new data set using the adaptive learning method.

Description

Technical Field

[0001] The present disclosure relates to digital pathology, and in particular to techniques for efficient development of initial models and efficient model updating and / or adaptation to different image domains using an adaptive learning framework. Background Art

[0002] Digital pathology involves scanning a slide (e.g., a histopathology or cytopathology slide) into a digital image that can be interpreted on a computer screen. For various reasons including the diagnosis of the disease, the evaluation of the response to treatment, and the development of pharmaceutical preparations to combat the disease, the tissue and / or cells in the digital image can then be examined by digital pathology image analysis and / or interpreted by a pathologist. In order to examine the tissue and / or cells in the digital image (which are almost transparent), various staining assays (e.g., immunohistochemistry) that selectively bind to tissue and / or cell components can be used to prepare pathology slides. Immunofluorescence (IF) is a technique for analyzing the determination of binding fluorescent dyes to antigens. Multiple assays in response to different wavelengths can be utilized on the same slide. These multiple IF slides enable the complexity and heterogeneity of the immune background of the tumor microenvironment to be understood as well as the potential impact on the response of the tumor to immunotherapy. In some assays, the target antigen of the stain in the tissue can be referred to as a biomarker. Thereafter, digital pathology image analysis may be performed on the digital images of the stained tissue and / or cells to identify and quantify staining for antigens (eg, biomarkers indicative of various cells such as tumor cells) in the biological tissue.

[0003] Methods and / or techniques based on artificial intelligence and machine learning have shown great promise in digital pathology image analysis (such as in cell detection, counting, localization, classification, and patient prognosis). Many computing systems equipped with machine learning techniques (including convolutional neural networks (CNNs)) have been proposed for image classification and digital pathology image analysis, such as cell detection and classification. For example, CNN can have a series of convolutional layers as hidden layers, and this network structure is able to extract representative features for object / image classification and digital pathology image analysis. In addition to object / image classification, machine learning techniques for image segmentation have also been implemented. Image segmentation is the process of segmenting a digital image into multiple fragments (pixel sets, also referred to as image objects). The goal of segmentation is to simplify and / or change the representation of an image to something more meaningful and easier to analyze. For example, image segmentation is often used to locate objects in an image, such as cells and boundaries (lines, curves, etc.). In order to perform image segmentation on large data (e.g., entire slide pathology images), the image is first divided into many small pieces. A computing system equipped with machine learning techniques is trained to classify each pixel in these blocks, all pixels in the same category are combined into one segmented region in each block, and then all segmented blocks are combined into one segmented image (e.g., a segmented whole slide pathology image). Thereafter, based on representative features associated with the segmented regions, machine learning techniques can be further implemented to predict or further classify the segmented regions (e.g., positive cells for a given biomarker, negative cells for a given biomarker, or cells with no staining expression). Summary of the invention

[0004] Methods based on artificial intelligence and machine learning have achieved excellent performance in digital pathology. However, developing such models is extremely time-consuming and resource-intensive. Not only do hundreds of thousands of annotations need to be made in the initial development phase to build a reliable model from scratch, but once developed, the model is limited in its generalization to unseen data, which leads to inevitable continuous investment in developing new models, even for related tasks. This paper discloses a framework for reducing resource requirements in the entire development process of AI-based digital pathology algorithms, including initial model development and subsequent model updates, improvements, and adaptations to different data sets. Specifically, this paper discloses a model pre-treatment phase that utilizes existing annotated data sets (related but not necessarily similar to the target data set on which the model is built), so that in the initial model development, only a small amount of annotations are required to generate a model with reasonable accuracy. For subsequent model updates and adaptation periods, an adaptive learning workflow is used for multiple digital pathology scenarios and strategies to select the best learning method for efficient model updates without having to train all data from scratch.

[0005] In various embodiments, a computer-implemented method is provided, comprising: obtaining, at a data processing system, a first annotated training set of images for training a machine learning algorithm to detect, characterize, classify, or a combination thereof, some or all regions or objects within the images, wherein the first annotated training set of images is in a first image domain; splitting, by the data processing system, the first annotated training set of images into image mini-sets, each mini-set representing a different modeling subtask and including a finite number of examples; using the image mini-sets by the data processing system in a first stage to train a machine learning algorithm to generate a pre-conditioned machine learning model, wherein the pre-conditioned machine learning model is configured to detect, characterize, classify, or a combination thereof, some or all regions or objects within a new image. or a combination thereof; labeling, by a data processing system, a limited number of images from a target data set to generate a second annotated training set of images for training a machine learning algorithm to detect, characterize, classify, or a combination thereof, some or all regions or objects within the images, wherein the second annotated training set of images is in a second image domain; and using, by the data processing system in a second stage, the second annotated training set of images to train a pre-conditioned machine learning model to generate a target machine learning model, the target machine learning model being configured to detect, characterize, classify, or a combination thereof, some or all regions or objects within a new image, wherein the number of categories trained in the first stage is a fraction of or matches the number of categories trained in the second stage.

[0006] In some embodiments, the first annotated training set of images are digital pathology images comprising one or more types of cells.

[0007] In some embodiments, the splitting includes: when only one image mini-set is available, for each different modeling subtask, selecting a subset of categories to be part of or match the number of categories trained in the second stage, and selecting a limited number of examples based on the selected subset of categories; and when multiple image mini-sets are available, for each different modeling subtask, performing either of the following: (i) mixing examples from multiple image mini-sets, selecting a subset of categories to be part of or match the number of categories trained in the second stage, and selecting a limited number of examples from the mixed examples based on the selected subset of categories, or (ii) selecting an image mini-set from multiple image mini-sets, selecting a subset of categories to be part of or match the number of categories trained in the second stage, and selecting a limited number of examples from the selected image mini-set based on the selected subset of categories.

[0008] In some embodiments, the second stage further includes: applying the pre-trained machine learning model to generate a feature vector representation for each example in a second annotated training set of images; combining the feature vector representations of examples from the same category to generate one representation for each target category, each target category using one representation as a prototype for the target category; generating feature vector representations for images or image regions of the remainder of the unlabeled images from the target dataset; and comparing each feature vector representation from the unlabeled image with the prototype for the target category based on the distance between the feature vector representation from the unlabeled image and the prototype for the target category.

[0009] In some embodiments, the first stage further includes: an inner learning loop, in which the machine learning algorithm updates the model weights or parameters on one subtask using a predefined or flexible number of rounds to initialize the treated machine learning model to adapt to the target dataset, which generates a loss for the image validation set after the model is updated, denoted as L-subtask-i for the i-th subtask; and an outer learning loop, in which the purpose is to search for a set of model initializations that, when used to update all subtasks, generates a pre-treated machine learning model, each of which has only this finite number of examples, by finding the model initialization that minimizes the sum of all losses denoted as the sum of L-subtask-i, where i ranges from 1 to the number of subtasks, which is calculated based on the subtasks relative to the image validation set for the model initialization.

[0010] In some embodiments, the first phase of training includes performing an iterative operation to learn a set of parameters to detect, characterize, classify, or a combination thereof, for some or all regions or objects within a mini-set of images, the set of parameters maximizing or minimizing a cost function, wherein each iteration involves finding a set of parameters for the machine learning algorithm such that a value of the cost function using the set of parameters is greater or less than a value of the cost function using another set of parameters in a previous iteration, and wherein the cost function is constructed to measure the difference between predictions made for some or all regions or objects using the machine learning algorithm and true value labels provided for the mini-set of images.

[0011] In some embodiments, the second stage of training includes performing an iterative operation to learn a set of parameters to detect, characterize, classify, or a combination thereof, for some or all regions or objects within a second annotated training set of images, the set of parameters maximizing or minimizing a cost function, wherein each iteration involves finding a set of parameters for a pre-treated machine learning model such that a value of the cost function using the set of parameters is greater or less than a value of the cost function using another set of parameters in a previous iteration, and wherein the cost function is constructed to measure the difference between predictions made for some or all regions or objects using the pre-treated machine learning model and true value labels provided for the second annotated training set of images.

[0012] In some embodiments, the computer-implemented method further includes: identifying a digital pathology scenario; selecting an adaptive continuous learning method for updating a target machine learning model given the digital pathology scenario; and updating the target machine learning model based on the adaptive continuous learning method to generate an updated machine learning model.

[0013] In some embodiments, the digital pathology scenario is a data increment scenario, a domain increment scenario, a category increment scenario, or a task increment scenario.

[0014] In some embodiments, the adaptive continuous learning method is selected from the group consisting of elastic weight consolidation (EWC), forgetfulness-free learning (LWF), incremental learner and representation learning (iCaRL), continuous prototype evaluation (CoPE), A-GEM, and parameter isolation methods.

[0015] In some embodiments, the computer-implemented method further includes providing a target machine learning model and / or an updated machine learning model.

[0016] In some embodiments, the providing includes deploying the target machine learning model and / or the updated machine learning model in the digital pathology system.

[0017] In some embodiments, the computer-implemented method further includes: receiving a new image by a data processing system; inputting the new image into a target machine learning model or an updated machine learning model; detecting, characterizing, classifying, or a combination thereof, some or all regions or objects within the new image by the target machine learning model or the updated machine learning model; and outputting an inference based on the detection, characterization, classification, or a combination thereof, by the target machine learning model or the updated machine learning model.

[0018] In some embodiments, the computer-implemented method further includes determining, by a user, a diagnosis of the subject associated with the new image, wherein the diagnosis is determined based on an inference output by the target machine learning model or an updated machine learning model.

[0019] In some embodiments, the computer-implemented method further includes administering, by a user, treatment to the subject based on (i) inferences output by the target machine learning model or an updated machine learning model and / or (ii) a diagnosis of the subject.

[0020] In some embodiments, training the machine learning algorithm includes implementing meta-learning principles to enable a first stage to generate a pre-conditioned machine learning model using a limited number of examples.

[0021] In some embodiments, training the pre-conditioned machine learning model includes implementing meta-learning principles to enable the second stage to generate a target machine learning model using a limited number of images.

[0022] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0023] In some embodiments, a computer program product is provided, which is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein.

[0024] The terms and expressions that have been adopted are used as terms of description and not of limitation, and when such terms and expressions are used, there is no intention to exclude any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the claimed invention. Therefore, it should be understood that although the claimed invention has been specifically disclosed through embodiments and optional features, modifications and variations of the concepts disclosed herein may be adopted by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Aspects and features of various embodiments will become more apparent by describing examples with reference to the accompanying drawings, in which:

[0026] Figure 1 We demonstrate how AI-based digital pathology algorithms can efficiently adapt to the challenges of increasing data complexity.

[0027] Figure 2 Efficient model development using an adaptive learning framework according to various embodiments is demonstrated.

[0028] Figure 3 A comparison of the resource requirements of training from scratch, transfer learning, and adaptive learning is shown.

[0029] Figure 4 An exemplary network for generating digital pathology images according to various embodiments is shown.

[0030] Figure 5 An overview of an adaptive learning framework according to various embodiments is shown.

[0031] Figure 6 Example IHC images and ground truth values ​​for cell detection according to various embodiments are shown.

[0032] Figure 7 An example of adaptive learning on data with an increasing number of similarities for hematoxylin and eosin (H&E) tissue type classification according to various embodiments (data increment) is shown.

[0033] Figure 8 An example of adaptive learning of an increasing number of data sources for H&E tissue type classification (domain increment) is shown in accordance with various embodiments.

[0034] Fig. 9 An example of adaptive learning for an increasing number of tissue types for H&E tissue type classification (class increments) is shown in accordance with various embodiments.

[0035] Fig.10 An exemplary computing environment for processing digital pathology images using machine learning / deep learning models in accordance with various embodiments is shown.

[0036] Fig.11 A flowchart is shown that demonstrates a process for training and using a machine learning model according to various embodiments of the present disclosure.

[0037] Fig.12 Example images from a CRC training dataset are shown in accordance with various embodiments.

[0038] Fig.13 Example images from the PatchCam dataset are shown in accordance with various embodiments.

[0039] Fig.14 The accuracy of the test streams according to various embodiments at the end of each main experience in the class increment setting with the enhanced CRC dataset using COPE with two baselines is shown.

[0040] Fig.15 Example images from an augmented training dataset are shown in accordance with various embodiments.

[0041] Fig.16The accuracy (left) and forgetting (right) of the test streams at the end of each primary experience in the domain increment setting with an enhanced CRC dataset using LwF according to various embodiments are shown.

[0042] Fig.17 The accuracy of the test flows according to various embodiments at the end of each master experience under the data increment setting with the enhanced CRC data set using iCaRL is shown.

[0043] Fig.18 The overall accuracy at the end of training for data, domain, and category increment scenarios from different continual learning methods according to various embodiments is shown along with two baselines.

[0044] Fig.19 The performance of the continuous learning method in a data increment scenario according to various embodiments is shown.

[0045] Fig. 20 The performance of the continual learning method in a domain increment scenario according to various embodiments is shown.

[0046] Fig.21 The performance of the continual learning method in a class increment scenario according to various embodiments is shown.

[0047] Fig. 22 Results from A-GEM utilizing enhanced CRC data, set for a mission increment scenario, are shown in accordance with various embodiments.

[0048] Fig.23 The accuracy of the test flows according to various embodiments at the end of each master experience in the domain increment setting using CoPE with an enhanced CRC data set is shown.

[0049] Fig.24 The impact of category grouping and empirical settings on category increment performance using iCaRL according to various embodiments is shown.

[0050] Fig.25 Continuous learning from different tumor types in a domain-increment setting with LwF according to various embodiments is shown. DETAILED DESCRIPTION

[0051] Although certain embodiments are described, these embodiments are presented only by way of example and are not intended to limit the scope of protection. The devices, methods, and systems described herein may be embodied in a variety of other forms. In addition, various omissions, substitutions, and changes may be made to the forms of the example methods and systems described herein without departing from the scope of protection.

[0052] I. Overview

[0053] Artificial intelligence and machine learning-based methods have achieved unprecedented performance in solving complex problems in digital pathology-based analysis, reducing the workload of pathologists by automatically suggesting potential lesions and thereby increasing the pathologist's confidence in the diagnosis and reducing subjectivity. A general strategy to ensure reliable and trustworthy digital pathology algorithm performance is to collect a large amount of annotations for each target image domain in digital pathology from scratch during both (a) the initial model training phase and (b) the model update phase and use this large amount of annotations to train the model to achieve performance improvement or adaptation for different image domains. However, for basic tasks of histopathology image analysis, such as image classification, semantic segmentation, and object detection, annotations need to be manually created and curated by pathologists. This time-consuming and resource-intensive model development poses two major challenges in digital pathology.

[0054] First, there is the challenge of efficiently and effectively building AI models for target digital pathology datasets. Due to their nature of large data requirements and restricted generalization to unseen data distributions, most deep learning models take months to develop, where large annotation sets specific to the target dataset need to be generated and validated by qualified experts. However, with such a time-consuming and resource-intensive development process, it is challenging to meet the growing demand for such models. Consider the emerging multiplexing technologies in the market: In the upcoming years, a diverse set of new assays will be developed in conjunction with deep learning-based digital pathology analysis. Furthermore, evolving patient populations and unpredictable disease outbreaks require efficient model development strategies to cope with the ever-changing demands in diagnostics.

[0055] Second, there is the challenge of efficiently adapting existing model systems to related but disparate datasets. The conditions under which images are digitized are constantly evolving, especially as technology advances. The use of different stain chromogens, changes in staining, digital scanners, and vendor platforms will result in a shift in the appearance of digitized images. The proliferation and availability of digital pathology also generates a near-continuous stream of data that is biologically distinct due to the heterogeneous disease samples being analyzed. The increasing volume and complexity of data requires the development of AI algorithms that can adapt to a wide variety of conditions and maintain performance across that variety. Traditional approaches of retraining on newer batches of data using manual, task-specific annotations can quickly increase the need for data storage, training time, and computing power, thereby increasing costs, delaying product releases, and ultimately reaching a point where it becomes cost-prohibitive for organizations, such as Figure 1 shown.

[0056] Since the pre-deep learning era, methods based on active learning have been designed to reduce the number of annotations required for model development. These methods use heuristic scoring strategies to query a small number of unlabeled examples that are the most informative in the data set. Active learning aims to gradually improve model performance by only iteratively adding a smaller set of such selected examples for model retraining, thereby avoiding annotating a large number of examples that may be less informative. However, models developed using active learning are only applicable to test images with the same or identical distribution as training images. For example, a model trained using active learning on an immunohistochemistry (IHC) assay (domain 1) cannot be generalized to another domain, such as another IHC assay (domain 2). As another example, a model trained with one tissue type is not easily transferable to other tissue types. Therefore, such methods are different from the adaptive learning framework disclosed herein, and the challenges of model inflexibility and lack of model adaptability cannot be fully addressed. In addition, active learning requires iterative model retraining using all existing and newly annotated examples at each development iteration, and therefore cannot help reduce the demand for computing power and storage space.

[0057] Alternatively, transfer learning and domain adaptation have improved model generalization to some extent. With transfer learning, all or part of the model weights are trained on a large annotated dataset applied to a new image domain; while the goal of domain adaptation is to use only a small amount of annotations from the target domain or no annotations for model development. However, both frameworks can suffer from catastrophic forgetting: when a model trained on one image domain (the source domain) is used to train on another image domain (the target domain), the model's performance on the source domain is greatly reduced, and thus "forgets" the knowledge learned from the previous training process on the source domain data. Although the goal of domain adaptation algorithms is to generate models that have good performance on both the source and target domains, even without annotations from the target domain (unsupervised domain adaptation), current practice still relies on a validation set from the target domain for model selection, which inevitably tends to overfit to the validation set.

[0058] To address these and other challenges, various embodiments disclosed herein relate to methods, systems, and computer-readable storage media to: (1) reduce the resource requirements needed to develop an artificial intelligence model for an unseen distribution of digital pathology data during an initial model development phase; and (2) reduce the resource requirements for subsequent iterations of model development, which is intended to improve or adapt the initial artificial intelligence model built after (1) to a related but not identical dataset.

[0059] In order to reduce the resource requirements in the initial model development phase, a technique for pre-processing the artificial intelligence system to learn useful features from existing digital pathology data is implemented. This design utilizes existing annotated digital pathology datasets that are related but not necessarily similar to the target data, which enables the artificial intelligence system to use these related datasets to refine its learning skills through a pre-training phase. In this article, "learning skills" include one or more of the following: the best model initialization set, the best model weight set that can be generalized to unseen data, the best model architecture set, etc. Using these learning skills, the pre-processed model can achieve reasonable performance with only a small annotation set. For example, 75% accuracy is achieved by using <50 annotated images to classify tissue types from tumor types that are not used to train the model, while >2000 images are used to train a model specific to this tumor type using a traditional artificial intelligence model.

[0060] In order to reduce the resource requirements for subsequent model development or model adaptation for different datasets, continuous learning techniques and algorithms are implemented to utilize sequentially obtained data to achieve model updates without having to train the model from scratch using all existing and new datasets. Although continuous learning algorithms provide a solution for learning from sequential data streams, ensuring that old knowledge is not forgotten (catastrophic forgetting) remains a challenge. In order to select the most effective continuous learning algorithm for diverse digital pathology data, a strategy set is designed with the various model update requirements commonly encountered in digital pathology applications as the purpose, and the corresponding algorithms are implemented for continuous learning without the need to train from scratch and without model performance degradation for previously encountered data (e.g., catastrophic forgetting of machine learning models).

[0061] These various techniques and algorithms are implemented in an adaptive learning framework that includes the following features and advantages.

[0062] (1) Effectively reduce the resource requirements of the entire development process of AI-based digital pathology algorithms. The adaptive learning framework addresses the challenges of other resource-demanding processes both in initial model development and subsequent model updates / adaptations, such as Figure 2 As shown. Compared with conventional artificial intelligence strategies (training points from scratch), the adaptive learning framework achieves efficient initial model development with a small number of annotations (e.g., less than 50) and faster model improvement to converge to good model performance (adaptive learning points). The arrows show the model update process, where various methods can be applied, including self-supervised learning, continuous adaptive learning, pre-conditioned continuous adaptive learning (using a small annotation set), and conventional training (using sufficient annotations).

[0063] (2) The adaptive learning framework can be applied to model development targeting images from different image domains in digital pathology. In this article, an image domain refers to a set of images with a specific sample distribution. Examples of different image domains include different image modalities (IHC images vs. H&E images), IHC assays (IHC targeting Ki67 vs. CK7 vs. PDL1-CK7), tumor types and subtypes (breast cancer vs. lung cancer), scanners from different vendors, etc.

[0064] (3) The adaptive learning framework utilizes existing digital pathology datasets (from one or more image domains) during the pre-treatment phase of initial model development, and the corresponding pre-treated models can be applied to image domains that are the same as or related but different from the image domain in the pre-treatment phase.

[0065] (4) The selection algorithm for the design of model updating strategy enables the effective extension of the existing model to adapt to different image domains after the initial model development.

[0066] (5) Users such as developers can flexibly apply only the initial model development strategy, only the model update strategy, or both, depending on the availability of the initial model and the need to update the model to a new image dataset or new domain. Alternatively, self-supervised pre-training can be combined with pre-training and model update strategies at one or all multiple model development iterations.

[0067] (6) The adaptive learning framework is domain-independent and can be applied to other imaging modalities as well as other computational studies, such as multimodal analysis, gene sequencing signal analysis, survival modeling, etc.

[0068] Figure 3 A comparison of the resource requirements of training from scratch, transfer learning, and the adaptive learning framework described in this paper is shown. Resources (y-axis) refer to the total number of images (x-axis) that need to be computed for model training for various subsequent model versions: if an image is processed N times, it counts as N images. Training from scratch consumes most of the resources, and the requirements grow quickly. Transfer learning only requires resources for computing new batches of data. Adaptive learning requires similar computational resources, with a negligible increase compared to transfer learning.

[0069] In one illustrative embodiment, a computer-implemented process is provided, comprising: obtaining, at a data processing system, a first annotated training set of images for training a machine learning algorithm to detect, characterize, classify, or a combination thereof, some or all regions or objects within the images, wherein the first annotated training set of images is in a first image domain; splitting, by the data processing system, the first annotated training set of images into image mini-sets, each mini-set representing a different modeling subtask and including a finite number of examples; training, by the data processing system, the machine learning algorithm using the image mini-sets in a first stage to generate a preconditioned machine learning model, the preconditioned machine learning model being configured to detect, characterize, classify, or a combination thereof, some or all regions or objects within a new image. classes or a combination thereof; labeling, by a data processing system, a limited number of images from a target data set to generate a second annotated training set of images for training a machine learning algorithm to detect, characterize, classify, or a combination thereof, some or all regions or objects within the images, wherein the second annotated training set of images is in a second image domain; and training, by the data processing system in a second stage, the pre-conditioned machine learning model using the second annotated training set of images to generate a target machine learning model, the target machine learning model being configured to detect, characterize, classify, or a combination thereof, some or all regions or objects within a new image, wherein the number of classes trained in the first stage is a fraction of or matches the number of classes trained in the second stage.

[0070] In some embodiments, the computer-implemented process further includes: identifying a digital pathology scenario; selecting an adaptive continuous learning method for updating a target machine learning model given the digital pathology scenario; and updating the target machine learning model based on the adaptive continuous learning method to generate an updated machine learning model.

[0071] Advantageously, the various techniques described herein can improve the robustness of machine learning models (e.g., improve the accuracy of cell classification).

[0072] II. Definitions

[0073] As used herein, when an action is "based on" something, it means that the action is at least partially based on at least a portion of the something.

[0074] As used herein, the terms "substantially," "approximately," and "about" are defined as being largely, but not necessarily completely, specified as understood by one of ordinary skill in the art (and including completely specified). In any disclosed embodiment, the terms "substantially," "approximately," or "about" may be replaced with "within [a certain percentage]" as specified, where percentages include 0.1%, 1%, 5%, and 10%.

[0075] As used herein, the terms "sample," "biological sample," "tissue," or "tissue sample" refer to any sample including biomolecules (such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (such as humans; therianthropes, such as cats, dogs, horses, cows, and pigs; and laboratory animals, such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections and puncture biopsies of tissue), cell samples (such as cytological smears, such as cervical smears or blood smears, or cell samples obtained by microdissection), or cell fractions, fragments, or organelles (such as obtained by lysing cells and separating their components by centrifugation or other means). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical biopsy or needle biopsy), nipple aspirate, cerumen, breast milk, vaginal secretions, saliva, swabs (e.g., oral swabs), or any material containing biomolecules and derived from a first biological sample. In certain embodiments, the term "biological sample" as used herein refers to a sample (e.g., a homogenized or liquefied sample) prepared from a tumor or a portion thereof obtained from a subject.

[0076] As used herein, the terms "biological material," "biological structure," or "cellular structure" refer to natural materials or structures, including whole or partial living structures (e.g., cell nucleus, cell membrane, cytoplasm, chromosomes, DNA, cells, cell clusters, etc.).

[0077] As used herein, "digital pathology image" refers to a digital image of a stained sample.

[0078] As used herein, the term "cell detection" refers to detecting pixel locations and properties of cells or cell structures (eg, cell nuclei, cell membranes, cytoplasm, chromosomes, DNA, cells, cell clusters, etc.).

[0079] As used herein, the term "target region" refers to a region of an image that includes image data that is intended to be evaluated in an image analysis process. The target region includes any region of a tissue region of an image that is intended to be analyzed in an image analysis process (e.g., tumor cells or staining expression).

[0080] As used herein, the term "tile" or "tile image" refers to a single image corresponding to a portion of a full image or full slide. In some embodiments, a "tile" or "tile image" refers to an area or region of interest scanned from a full slide, the area or region of interest having (x, y) pixel dimensions (e.g., 1000 pixels x 1000 pixels). For example, consider splitting the entire image into M columns of tiles and N rows of tiles, where each tile within the M x N mosaic includes a portion of the entire image, i.e., at position M 1 ,N 1 The tile at position M includes the first part of the image, while the tile at position M 1 ,N 2 The tile at comprises a second portion of the image, the first portion and the second portion being different. In some embodiments, the tiles can each have the same dimensions (pixel size x pixel size). In some cases, the tiles can partially overlap, representing overlapping areas of a full slide scan or a region of interest.

[0081] As used herein, the term "block", "image block" or "mask module" refers to a pixel container corresponding to a portion of a full image, a full slide or a full mask. In some embodiments, a "block", "image block" or "mask module" refers to an area or target area of ​​an image or mask, which has (x, y) pixel dimensions (e.g., 256 pixels x 256 pixels). For example, an image of 1000 pixels x 1000 pixels divided into 100 pixel x 100 pixel blocks will include 10 blocks (each block contains 1000 pixels). In other embodiments, a block overlaps with each "block", "image block" or "mask module" that has (x, y) pixel dimensions and shares one or more pixels with another "block", "image block" or "mask module".

[0082] III. Generating Digital Pathology Images

[0083] Digital pathology involves the interpretation of digitized images in order to correctly diagnose a subject and guide treatment decisions. In a digital pathology solution, an image analysis workflow can be established to automatically detect or classify biological objects of interest, such as positive, negative tumor cells, etc. An exemplary digital pathology solution workflow includes obtaining a tissue slide, scanning a preselected area or the entire slide using a digital image scanner (e.g., a whole slide image (WSI) scanner) to obtain a digital image, performing image analysis on the digital image using one or more image analysis algorithms, and possibly detecting, quantifying each object of interest (e.g., counting or identifying object-specific or cumulative areas of each object of interest) based on the image analysis (e.g., quantitative or semi-quantitative scores, such as positive, negative, moderate, weak, etc.).

[0084] Figure 4An exemplary network 400 for generating digital pathology images is shown. A fixation / embedding system 405 uses a fixative (e.g., a liquid fixative, such as a formaldehyde solution) and / or an embedding substance (e.g., a histological wax such as paraffin and / or one or more resins such as styrene or polyethylene) to fix and / or embed tissue samples (e.g., a sample comprising at least a portion of at least one tumor). Each sample can be fixed by exposing the sample to a fixative for a predetermined period of time (e.g., at least 3 hours) and then dehydrating the sample (e.g., via exposure to an ethanol solution and / or a clarifying intermediate). The embedding substance can infiltrate the sample when the sample is in a liquid state (e.g., when heated).

[0085] Specimens fixation and / or embedding are used to preserve specimens and slow degradation. In histology, fixation generally refers to the irreversible process of using chemicals to retain chemical composition, preserve native specimen architecture, and keep cellular structures from degradation. Fixation may also stiffen cells or tissues for sectioning. Fixatives may enhance specimen and cell preservation by cross-linking proteins. Fixatives may bind to and cross-link some proteins and denature others by dehydration, which may stiffen tissues and inactivate enzymes that might otherwise degrade the specimen. Fixatives may also kill bacteria.

[0086] Fixative can be applied, for example, by perfusion and infiltration of the prepared sample. Various fixatives can be used, including methanol, Bouin fixative and / or formaldehyde fixative, such as neutral buffered formalin (NBF) or paraffin-formalin (paraformaldehyde-PFA). In the case where the sample is a liquid sample (e.g., a blood sample), the sample can be smeared on a slide and dried before fixation. Although for the purpose of histological research, the fixation process can be used to preserve the structure of the sample and the cell, fixation may lead to hiding tissue antigens, thereby reducing antigen detection. Therefore, fixation is generally considered to be a limiting factor in immunohistochemistry, because formalin can cross-link antigens and mask epitopes. In some cases, an additional process is performed to reverse the influence of cross-linking, including processing fixed samples with citraconic anhydride (a reversible protein cross-linking agent) and heating.

[0087] Embedding can include infiltrating a sample (e.g., a fixed tissue sample) with a suitable histological wax, such as paraffin. Histological wax may be insoluble in water or alcohol, but soluble in paraffin solvents, such as xylene. Therefore, the water in the tissue may need to be replaced with xylene. For this reason, the tissue can first be dehydrated by gradually replacing the water in the sample with alcohol, which can be achieved by passing the tissue through an increased concentration of ethanol (e.g., from 0% to about 100%). After water is replaced with alcohol, the alcohol can be replaced with xylene that is miscible with alcohol. Because histological wax is soluble in xylene, the molten wax may fill the space filled with xylene and previously filled with water. The sample filled with wax can be cooled to form a hardened block, which can be clamped into a microtome, a vibrating microtome, or a compression vibrating microtome for slicing. In some cases, deviation from the above-mentioned example procedure may result in paraffin infiltration, thereby inhibiting the penetration of antibodies, chemicals, or other fixatives.

[0088] Then, tissue slicer 410 can be used to slice fixed and / or embedded tissue samples (e.g., samples of tumors). Slicing is the process of cutting thin slices (e.g., 4 to 5 μm thick) of samples from tissue blocks so that they can be placed on microscope slides for inspection. Slicing can be performed using a slicer, a vibrating slicer, or a compression vibrating slicer. In some cases, tissue can be quickly frozen in dry ice or isopentane and then cut with a cold knife in a refrigerator (e.g., a cryostat). Other types of coolants can be used for freezing tissue, such as liquid nitrogen. Slices for use with bright field and fluorescence microscopes are generally about 4 to 10 μm thick. In some cases, slices can be embedded in epoxy or acrylic resins so that thinner slices (e.g., <2 μm) can be cut. Slices can then be mounted on one or more slides. A cover glass can be placed on top to protect the sample slices.

[0089] Because the tissue section and the cells therein are actually transparent, the preparation of the slide usually further includes staining the tissue section (e.g., automatic staining) to make the relevant structure more visible. In some cases, staining is performed manually. In some cases, staining is performed semi-automatically or automatically using staining system 415. The staining process includes exposing the slice of the tissue sample or the fixed liquid sample to one or more different staining agents (e.g., continuously or simultaneously) to express the different characteristics of the tissue.

[0090] For example, staining can be used to mark specific types of cells and / or label specific types of nucleic acids and / or proteins to assist in microscopic examination. The staining process typically involves adding a dye or stain to a sample to identify or quantify the presence of a specific compound, structure, molecule, or feature (e.g., subcellular feature). For example, staining can help identify or highlight specific biomarkers in a tissue section. In other examples, stains can be used to identify or highlight biological tissues (e.g., muscle fibers or connective tissue), cell populations (e.g., different blood cells), or organelles within individual cells.

[0091] An exemplary type of tissue staining is a histochemical staining, which uses one or more chemical dyes (e.g., acid dyes, basic dyes, chromogens) to stain tissue structures. Histochemical staining can be used to indicate general aspects of tissue morphology and / or cell histology (e.g., to distinguish between nuclei and cytoplasm, indicate lipid droplets, etc.). An example of a histochemical stain is H&E. Other examples of histochemical stains include trichrome stains (e.g., Masson's trichrome stain), periodic acid-Schiff (PAS), silver stains, and iron stains. The molecular weight of a histochemical staining reagent (e.g., dye) is typically about 500 kilodaltons (kD) or less, although some histochemical staining reagents (e.g., Alcian blue, phosphomolybdic acid (PMA)) may have a molecular weight of up to two or three thousand kD. An example of a high molecular weight histochemical staining reagent is alpha-amylase (about 55kD), which can be used to indicate glycogen.

[0092] Another type of tissue staining is IHC, also known as "immunostaining," which uses a primary antibody that specifically binds to a target antigen of interest (also known as a biomarker). IHC can be direct or indirect. In direct IHC, a primary antibody is directly conjugated to a marker (e.g., a chromophore or fluorophore). In indirect IHC, a primary antibody first binds to the target antigen, and then a secondary antibody conjugated to a marker (e.g., a chromophore or fluorophore) binds to the primary antibody. The molecular weight of IHC reagents is much higher than that of histochemical staining reagents, as the molecular weight of antibodies is approximately 150 kD or higher.

[0093] Various types of staining protocols can be used for staining. For example, an exemplary IHC staining protocol includes: using a hydrophobic barrier line around the sample (e.g., tissue section) to prevent leakage of reagents from the slide during incubation; treating the tissue section with a reagent to block endogenous sources of nonspecific staining (e.g., enzymes, free aldehyde groups, immunoglobulins, other irrelevant molecules that can mimic specific staining); incubating the sample with a permeabilization buffer to promote penetration of antibodies and other staining reagents into the tissue; incubating the tissue section with a primary antibody for a period of time (e.g., 1 to 24 hours) at a specific temperature (e.g., room temperature, 6°C to 8°C); using a wash buffer to rinse the sample; incubating the sample (tissue section) with a secondary antibody for another period of time at another specific temperature (e.g., room temperature); rinsing the sample again with an aqueous buffer; incubating the rinsed sample with a chromogen (e.g., DAB: 3,3'-diaminobenzidine); and washing away the chromogen to stop the reaction. In some cases, counterstaining is then used to identify the entire "landscape" of the sample and as a reference for the primary color used to detect tissue targets. Examples of counterstains may include hematoxylin (stains from blue to purple), methylene blue (stains blue), toluidine blue (stains nuclei dark blue and polysaccharides from pink to red), nuclear fast red (also known as Kernechtrot dye, stains red), and methyl green (stains green); non-nuclear chromogenic stains such as eosin (stains pink), etc. One of ordinary skill in the art will recognize that other immunohistochemical staining techniques may be implemented to perform staining.

[0094] In another example, an H&E staining protocol may be performed on a tissue section. The H&E staining protocol includes applying a hematoxylin stain or mordant mixed with a metal salt to the sample. The sample may then be rinsed in a weak acid solution to remove excess stain (differentiation) and then blued in weak alkaline water. After applying the hematoxylin, the sample may be counterstained with eosin. It should be understood that other H&E staining techniques may be implemented.

[0095] In some embodiments, various types of stains can be used for staining, depending on the target features targeted. For example, DAB can be used for various tissue sections stained by IHC, wherein DAB produces a brown color that depicts the target features in the stained image. In another example, alkaline phosphatase (AP) can be used for skin tissue sections stained by IHC because the DAB color may be masked by melanin. Regarding primary staining techniques, applicable stains may include, for example, basophilic and eosinophilic stains, heme and hematoxylin, silver nitrate, trichrome stains, etc. Acidic dyes may react with cationic or alkaline components in tissues or cells, such as proteins and other components in the cytoplasm. Basic dyes may react with anionic or acidic components in tissues or cells, such as nucleic acids. As described above, an example of a staining system is H&E. Eosin may be a negatively charged pink acidic dye, and hematoxylin may be a purple or blue basic dye, which includes hematoxylin and aluminum ions. Other examples of stains may include periodic acid-Schiff (PAS) stains, Masson's trichrome stains, Alcian blue stains, van Gieson stains, reticulum stains, etc. In some embodiments, different types of stains may be used in combination.

[0096] The slices can then be placed on the corresponding slides, and then the imaging system 420 can scan or image the corresponding slides to generate original digital pathology images 425a to 425n. A microscope (e.g., an electron or optical microscope) can be used to magnify the stained sample. For example, the resolution of an optical microscope may be less than 1 μm, such as about several hundred nanometers. In order to observe finer details in the nanometer or sub-nanometer range, an electron microscope can be used. An imaging device (combined with a microscope or separated from a microscope) images the magnified biological sample to obtain image data, such as a multi-channel image (e.g., multi-channel fluorescence) with multiple (such as, for example, ten to sixteen) channels. The imaging device can include, but is not limited to, a camera (e.g., an analog camera, a digital camera, etc.), an optical device (e.g., one or more lenses, a sensor focusing lens group, a microscope objective, etc.), an imaging sensor (e.g., a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) image sensor, etc.), a photosensitive film, etc. In a digital embodiment, the imaging device can include multiple lenses, which can cooperate to prove that they have an instant focus function. An image sensor (e.g., a CCD sensor) can capture a digital image of a biological sample. In some embodiments, the imaging device is a bright field imaging system, a multi-spectral imaging (MSI) system, or a fluorescence microscope system. The imaging device can capture images using invisible electromagnetic radiation (e.g., UV light) or other imaging techniques. For example, the imaging device can include a microscope and a camera arranged to capture images magnified by the microscope. The image data received by the analysis system can be identical to the raw image data captured by the imaging device and / or can be derived from the raw image data.

[0097] The images of the stained sections can then be stored in a storage device 425 such as a server. The images can be stored locally, remotely and / or in a cloud server. Each image can be stored in association with an identifier and a date (e.g., the date the sample was collected and / or the date the image was captured) of the subject. The images can further be transmitted to another system (e.g., a system associated with a pathologist, an automatic or semi-automatic image analysis system, or a machine learning training and deployment system, as described in further detail herein).

[0098] It should be understood that modifications are contemplated to the process described with respect to network 400. For example, if the sample is a liquid sample, embedding and / or sectioning may be omitted from the process.

[0099] IV. Adaptive Learning Framework

[0100] Figure 5 An adaptive learning framework is shown that includes two components: (505) initial model development, and (510) model update and / or adaptation to different image domains. The two components 505, 510 can be applied separately or all together: initial model development and subsequent model update and / or adaptation to different image domains.

[0101] Initial model development

[0102] In order to achieve efficient development of the initial model, a two-step development strategy ( Figure 5 ) as follows: Phase 1: Model pre-processing, in which the artificial intelligence system (e.g., Fig.10 The AI ​​system described in detail (hereinafter referred to as the AI ​​system) utilizes existing annotated datasets and improves its learning skills by training on these datasets. Phase 2: Target model training, in which the AI ​​system uses the learning skills learned from Phase 1 to extend itself to a different image domain (target domain), where a smaller number of annotations are required compared to conventional learning methods.

[0103] The aforementioned "learning skills" include one or more of the following: the best model initialization set, the best model weight set that can generalize to unseen data, the best model architecture set, etc. The best is determined using one or more metrics for measuring model performance, such as accuracy or area under the curve (AUC). The learning skills are then applied to the target image domain, so that the target domain requires a smaller number of annotations than conventional machine learning.

[0104] To achieve model preconditioning, a meta-learning strategy can be adopted in which an existing dataset (e.g., a training dataset) is split into mini-sets that each represent a different modeling subtask and contain only a small number of examples, thereby forming a large number of subtasks. The splitting can be performed at training time, and a different data split can be performed at each model training iteration. Using these subtasks to train an artificial intelligence system enables the artificial intelligence system to search for excellent solutions in the model optimization industry that can generalize to any related small subtask without overfitting to a specific subtask, thereby preconditioning the artificial intelligence system to stage 2 for unseen target domains. In general, the number of categories in stage 1 training is configured to be a portion of or match the number of categories in stage 2 training. For category increment scenarios in stage 2, the number of categories in stage 1 can be increased; while for domain and data increment scenarios in stage 2, the number of categories from stage 1 should be matched. In some cases, the number of categories in stage 2 may be greater than that in stage 1 (regarding Fig. 9 Detailed description). In classification tasks (e.g., binary classification and multi-class classification), the class labeling is at the image level, e.g., the class (or classes) is set for the entire image. In prediction tasks (e.g., dense prediction) such as image segmentation and object detection, the class labeling is at a more fine-grained level: for example, for object detection, each distinct object in the image has a label for its class and location. The training datasets used for model preconditioning can be related to each other, but have varying degrees of similarity to the training dataset of the target domain.

[0105] The following criteria can be implemented to sample existing annotated datasets. (1) If only one annotated dataset is available, for each subtask, a subset of categories can be randomly selected to be part of or match the number of categories in stage 2, and some examples can be randomly selected from the selected categories. In this scenario, there are a large number of subtasks with different or partially different categories. (2) If multiple annotated datasets are available, for each subtask, examples can be mixed from multiple datasets, and then the mixed dataset can be implemented in the same way as in (1); it is also possible to initially randomly select a dataset, and then a subset of categories in the dataset can be randomly selected to be part of or match the number of categories in stage 2. In both cases, (a) sampling strategies other than completely random can also be adopted, for example, some categories or some datasets can be sampled more frequently than other categories or datasets; and / or (b) the examples in each subtask can be split into a training subset and a validation subset.

[0106] More specifically, for image-level prediction, e.g., image classification tasks, an "example" refers to an image in the dataset with its class label, i.e., each subtask consists of a set of images, all of which belong to a selected class. Within each subtask, the image class labels are redefined for the training process in stage 1. For example, 15 images can be selected, 5 images of each class, for example, a total of 3 classes; regardless of the original class label of each image, for stage 1 training, a random class ordering is generated, where any of the 3 classes can be set as class 0, another class can be set as class 1, and the last remaining class can be set as class 2. In the next subtask, another 15 images from a different class set can be selected, and then the class labels of these images are reset to 0, 1, and 2 again in a random order. In this way, the model becomes class-independent because the model does not focus on learning information from each specific class, but learns how to improve the learning skills for all possible subtasks encountered by the model.

[0107] For prediction tasks (e.g., dense prediction), examples include all annotated entities (regions or objects) of the same category in the image and their labels; when sampling some examples from a selected category subset, some images containing at least one labeled entity from the selected category are initially selected, and then the labeled entities not from the selected category are set to the background category. For example, in an image segmentation task, the category labels may include foreground categories and background categories (i.e., regions that are not intended for modeling); if the selected categories are tumor nests and blood vessels, all regions in the image that do not belong to these two categories are relabeled as background categories, and all regions that belong to tumor nests are randomly relabeled as category 0 or 1, while the blood vessel regions are relabeled as any category index remaining after relabeling the tumor regions.

[0108] Methods for learning subtasks can be thought of as follows. Metric-based methods learn representations of existing datasets and extract from these datasets the skill of comparing similarities between examples of any class from an unseen target domain. With these methods, the representations learned in stage 1 can be applied to stage 2 in the following way: (a) select some images in the target domain, (b) annotate these images and apply a pre-trained model to generate a feature vector representation for each example (e.g., the output of the last layer in a convolutional network before the classification layer), (c) combine representations of examples from the same class to generate one representation per target class, (d) use these processed representations as prototypes for the target class, (e) generate feature vector representations for the remaining unlabeled images or image regions (or other entities in the case of dense prediction tasks) in the target domain, (f) compare each feature vector representation from the unlabeled target domain to the prototype by computing the distance between these vectors (e.g., cosine distance), and then assign a class label for the unlabeled image or image region to the prototype class with the smallest distance (e.g., most similar). Other techniques for learning subtasks can be used in conjunction with metric-based methods. For example, one can use adversarial generative models that synthesize images based on a distribution of existing images to increase the number of examples of each class in the target image domain using the generated images.

[0109] Alternatively, optimization-based methods can learn the best model weights for initializing the model so that the model can efficiently adapt to an unseen target dataset using only a small number of examples. Using these methods, stage 1 training involves two model optimization loops: (a) an inner learning loop in which the AI ​​model updates its model weights on a subtask using a predefined or flexible number of rounds, which generates a loss on the validation set for that subtask after the model update, denoted as L-subtask-i for the i-th subtask; and (b) an outer learning loop in which the goal is to search for a set of model initializations that, when used to update all subtasks, generates the best model, each with only a few annotated examples; this is achieved by finding the model initialization that minimizes the sum of all losses (summed over L-subtask-i, where i ranges from 1 to the number of subtasks), calculated based on the subtask's validation set relative to its model initialization's image validation set. Other techniques for learning subtasks can be used in conjunction with optimization-based methods. For example, searching not only for the best model initialization but also for the best model architecture (e.g., performing neural architecture search).

[0110] The designed workflow described herein for initial model development was applied as follows to cell detection in brightfield IHC assays and tissue type classification in H&E assays. However, it should be understood that similar workflows can also be applied to other staining methods, such as brightfield assays (e.g., tricolor Masson assays, which simultaneously stain muscle, collagen fibers, red blood cells, and nuclei) and special stains in fluorescent IHC assays.

[0111] (i) Model pretreatment for cell detection in IHC assay:

[0112] The goal is to identify staining phenotypes, cell types, and cell locations in each image. For example, for a DAB-Ki67 IHC assay, a cell detection model can be designed to identify tumor cells that stain positively for Ki67 (Ki67+ tumors), tumor cells that stain negatively for Ki67 (Ki67- tumors), all other cell types / staining types along with the location of each cell nucleus center at a single pixel or along with a bounding box for each cell nucleus (e.g., pixel locations of a rectangle circumscribing each cell nucleus) or pixels of each cell nucleus (e.g., a cell nucleus segmentation mask).

[0113] In general, images from brightfield IHC assays are related and have a certain degree of similarity in their appearance: hematoxylin stain is used in most of these assays and stains the cell nuclei as a guide to where the cells are located in the whole slide image, while the IHC staining protocol targets one or more biomarkers, and correspondingly, cells expressing these biomarkers are shown in color by the application of a chromogen. "Staining pattern" refers to the appearance of image areas that are positive for the target biomarker in terms of: (a) subcellular and / or cellular localization of the biomarker, (b) stained cell type, (c) staining intensity, (d) frequency of occurrence, and (e) spatial distribution of positive areas. For example, Ki67 has a nuclear staining pattern, for example, positive staining signals in IHC images from a Ki67 assay are observed to be located in the cell nuclei, primarily in scattered tumor cells or clustered tumor nests, with the positive staining signals ranging from low staining intensity to very high intensity.

[0114] A workflow for pre-processing can be designed using images with various chromogens and biomarker staining patterns from various IHC assays (see, Table 1 for example assays). An exemplary workflow may include the following: (1) If the chromogen used in the existing annotated dataset is the same as the chromogen used in the target IHC assay: (1.1) If annotations from only one assay are available, then at each model training iteration, split this dataset into subtasks and sample a subtask for training that has some images with annotations of cells from one or more classes and ensure that all classes of cells selected for this development stage are present in at least one of the images. For example, the DAB-Ki67 assay is pre-processed and applied to the DAB-PDL1 assay. Figure 6 An example DAB-Ki67 IHC image (A) is shown, where the brown signal is the image area where the chromogen DAB generates color, indicating these cells where Ki67 expressed in the cell nucleus is detected by this IHC assay; and the gray-blue signal is the cell nucleus stained with hematoxylin. Also shown are example true values ​​(B) for cell detection for the image in (A), where the dots of different colors overlaid on the center of the cell nucleus show the class label for each cell. (1.2) If annotations from multiple assays are available, in order to train the pre-conditioned artificial intelligence system to be as assay-independent as possible, at each model training iteration, an assay can be initially selected from all assays, and then some examples can be sampled from the assay. For example, the artificial intelligence system can be pre-conditioned for the DAB-Ki67 and DAB-CK7 assays and applied to the DAB-PDL1 assay.

[0115] (2) If the chromogen used in the existing annotated dataset is different from the chromogen used in the target IHC assay: a workflow similar to (1.1) or (1.2) can be performed. In addition, in order to match the chromogen in both stage 1 and stage 2, stain unmixing can be performed to decompose the IHC image into each stain component and remix these components with the color vectors extracted from the target IHC assay. For example, the artificial intelligence system can be pre-processed for the Tamra-PDL1 / Dabsyl-CK7 dual assay and applied to the DAB-Ki67 assay, where the training system or user can choose to decompose the Tamra-PDL1 / Dabsyl-CK7 dual image into PDL1, CK7 and hematoxylin grayscale intensity images and remix with the color vector of DAB to obtain two image sets for the pre-processing stage: a synthesized DAB-PDL1 single image and a synthesized DAB-CK7 single image. Figure 6An example Tamra-PDL1-Dabsyl-CK7 IHC image is shown (C). Also shown are example ground truth values ​​for a DAB-PR IHC image (D), where the differently colored dots overlaid on the center of the cell nucleus show the class label for each cell.

[0116] The measurements used in the pre-treatment phase do not necessarily need to have a high similarity to the target domain measurements, but if there is a certain degree of similarity, this may be beneficial due to the accumulation of learning skills and acquired knowledge in Phase 1.

[0117] Table 1. Example assay list

[0118] Assay No. Biomarkers Main subcellular localization Mainly expressing cell types Example Tumor Types 1 Ki67 Nucleus Tumor Breast 2 ER Nucleus Tumor Breast 3 PR Nucleus Tumor Breast 4 CD3 / CD8 membrane Lymphocytes colon 5 PDL1 Cytoplasm and membrane Tumor, macrophage Breast 6 PDL1 Cytoplasm and membrane Tumor, macrophage lung 7 CK7 Cytoplasm, membrane Tumor, normal epithelial lung 8 ER-PR Nucleus Tumor Breast 9 PDL1-CK7 Cytoplasm and membrane Tumor, macrophages, normal epithelial lung

[0119] (ii) Model pre-treatment for tissue type classification in H&E assay:

[0120] In this example, the goal is to generate a model to identify tissue types for each image tile from a whole slide image. For example, a model is constructed that can classify each image tile into tumor, stroma, normal tissue, and other types. Model pre-treatment can be performed using existing datasets from other disease types (e.g., different tumor types), from different disease stages, etc., other than the target domain, using a workflow similar to that described herein. Each subtask can be sampled from the same dataset or from a mixture of different datasets (if available).

[0121] Model updating and adaptation

[0122] In order to efficiently perform model updating and adaptation to new data sets after initial model development, scenarios commonly encountered in digital pathology settings are identified, and techniques are developed to select the most appropriate adaptive learning method for the scenario. These techniques are illustrated using tissue type classification for H&E images as an example. However, it should be understood that the techniques described for model updating and adaptation can be applied to various other scenarios commonly encountered in digital pathology.

[0123] When models are trained sequentially, the data flow may differ in different aspects. Common settings encountered in digital pathology can be categorized into the following scenarios:

[0124] 1. Data Increment Scenario: Creating annotated datasets curated by pathologists is a time-consuming process, and it is preferred to train the model in batches as and when the data arrives. The incoming data (which is considered a new stream on which the model is trained) has minimal differences from the previous data, usually comes from the same underlying distribution, and has all the categories as previously seen.

[0125] 2. Domain Increment: When the successive data streams come from different domains or distributions, the scenario is considered a domain increment scenario. This scenario is similar to the data increment scenario in that all data streams have all the categories represented in them.

[0126] 3. Category Increment: The category increment scenario is when the model is expanded in terms of the number of categories. Clinical needs for this scenario arise when the batch of annotated images contains a different subset of categories or the model definition changes to classify the data into more tissue subtypes. It should be understood that each data stream can include images from new and unseen categories as well as images of seen categories introduced in the previous data stream.

[0127] 4. Task Increment: Any of the above scenarios can be considered as task increment if each data stream is defined as a new task. This scenario uses a different architectural design - the network has shared layers between tasks as well as task-specific layers.

[0128] Each incremental data stream is called an experience. Each experience is then split into a training stream, a validation stream, and a test stream. The model is trained on the training stream, validated on the validation stream at the end of each epoch, and evaluated on the test stream at the end of each experience. At the end of training each experience to study forward and backward transfer, model performance is evaluated on the test stream from all experiences.

[0129] (i) Data incremental digital pathology model update:

[0130] (1.1) Digital Pathology Scenario: After the initial development of an AI-based digital pathology algorithm, it may be necessary to incorporate more annotated data from the same domain (e.g., with small differences and mostly similar; from the same underlying data distribution) into the model. In the digital pathology setting, expert annotations and / or expert-verified annotations are time-consuming and may not be available at the time of initial model development, and therefore annotated data is typically expected to arrive in batches. In addition, due to the large number of parameters in AI models, especially deep neural networks, additional data is often learned to prevent model overfitting. For example, an initial model may be trained to annotate images from colon cancer images (e.g., Figure 7 As shown - the lower left figure shows the class labels and number of examples under each model version) to classify tumors and normal tissues, and then in subsequent model development / updates, similar examples of the same category can be incorporated into the model, and such examples will arrive in batches over time.

[0131] (1.2) Workflow for selecting the best adaptive learning method: (1.2a) In order to efficiently and effectively update the model without training from scratch with existing and newly arrived data, a benchmark dataset is established from a given digital pathology domain, and the data assets are randomly split into batches, simulating the case of sequentially arriving data. (1.2b) In order to evaluate the effectiveness of candidate learning methods in the presence of small random changes in individual examples within each batch, an augmented dataset can be created to simulate such small changes, and the image order can be randomly shuffled so that each data batch contains images augmented in different ways. The different types and degrees of changes to the original data can be carefully balanced so that the distribution of the entire dataset is not skewed. Augmentation of the data can be performed dynamically during training or before each training iteration. Example augmentation techniques include: (i) changing the staining space, where a stain decomposition (e.g., unmixing) is initially performed, and each stain is then remixed with a predefined color vector to change the hue, saturation, and intensity of each stain, (ii) increasing image resolution by resizing the image and then cropping it back to the original size, (iii) similar techniques, or (iv) any combination thereof.

[0132] (1.3) Artificial Intelligence-Based Methods for Adaptive Learning: The generated benchmark data batches from (1.2b) can be run against candidate learning methods such as the following:

[0133] (1.3a) Regularization-based methods: These methods mainly focus on prioritizing privacy and reducing memory. Privacy is maintained by avoiding storing the original input. Among the different regularization-based methods, two common methods are the elastic weight consolidation (EWC) method and the learning without forgetting (LWF) method. EWC is a prior-focused method, where the model parameters are used as priors when learning from new data, and the method estimates the distribution over the model parameters. On the other hand, LWF is a data-focused method. The main design of data-focused methods is based on the knowledge distillation from the previous model to the current model trained on new data. This concept has also been introduced in LWF, where forgetting is dealt with using a knowledge transfer mechanism. These methods prevent the model from forgetting the learned knowledge by imposing constraints on the current model weights so that when the current model is updated, the model weights do not deviate too far from the current version of the model.

[0134] (1.3b) Replay-based methods: These methods retain the most informative examples or their feature representations (e.g., feature vectors extracted from the hidden layers of a neural network) from existing data batches and revisit them during training on newly arrived data. During learning a new task, previous task samples are replayed to overcome forgetting. Different methods that fall into this category are: incremental learners and representation learning (iCaRL), continuous prototype evaluation (CoPE), and A-GEM. The iCaRL method stores a subset of the best samples per class, which are selected based on the approximate class mean in the learned feature space. The upper bound of this method is determined by joint training on previous and current tasks. CoPE is an online data incremental learner where the prototype always represents the most salient features of the class population. Rapidly evolving prototypes enable learning and prediction at any point in time. Using replay and a balanced memory filling scheme, CoPE is robust to class imbalance. GEM is designed based on task increment occasions. This method focuses only on the new task by constraining the update of the new task and thus does not interfere with the previous tasks. This is done by projecting the calculated gradient direction to the feasible region delineated by the previous task gradient according to the first-order Taylor series approximation. The A-GEM method is an improved version of the GEM method. A-GEM helps to relax the problem of projecting in a direction estimated by randomly selected samples from the previous task data buffer. A-GEM produces performance accuracy similar to GEM, where the computational and memory efficiency is similar to the regularization method.

[0135] (1.3c) Combining Regularization and Replay

[0136] (1.3d) Methods that exploit meta-learning principles to implement model updates where there are a few examples per batch (see the “CoPE” method implemented in the Experiments section).

[0137] (1.3e) Parameter isolation methods: These methods assign different model parameters to each task to cope with forgetting. There is no fixed architecture for these methods because there is no constraint on the size of the architecture. Therefore, new branches can grow for new tasks. This can be achieved by freezing the parameters of the previous task or dedicating a copy of the model to each task. These types of architectures are called dynamic architectures.

[0138] (1.3f) Other techniques applied with the above methods. For example, self-supervised learning is performed on new unlabeled data before applying the above methods. As another example, adversarial generative models are performed to generate examples similar to those from previous data batches.

[0139] (ii) Domain-incremental digital pathology model update:

[0140] (2.1) Digital Pathology Scenario: After the initial development of AI-based digital pathology algorithms, it may be necessary to incorporate annotated data from different domains (e.g., similar but from different underlying data distributions) into the model. In the digital pathology setting, there is a practical need to be able to generate models that are robust to: (a) changes in experimental settings, such as changes or differences in (i) staining reagents, (ii) staining protocols or instruments, (iii) scanner vendors, (iv) sample sources (e.g., different clinical sites or tissue banks), etc.; (b) Other aspects such as changes in tissue type, organ type, patient population, disease stage, disease subtype (e.g., tumor type and subtype), etc.

[0141] (2.2) Workflow for selecting the best adaptive learning method: (2.2a) In order to update the model efficiently and effectively without training from scratch using both existing data and newly arrived data, a benchmark dataset is set from the DP domain and synthetic images that simulate realistic changes to data from different sources / domains are generated. (2.2b) In order to evaluate which candidate adaptive learning method is most effective for this scenario, an enhanced dataset can be created to simulate multiple types of domain transitions, and the candidate methods can be evaluated for how they perform under such changes. The data can be enhanced dynamically during training or before each training iteration. Example enhancement techniques include: (i) changing the color space, where a color decomposition (e.g., unmixing) is initially performed, and each color is then remixed with a predefined color vector to change the hue, saturation, and intensity of each color, (ii) increasing the image resolution by resizing the image and then cropping it back to the original size, (iii) similar techniques, or (iv) any combination thereof. One or more types of changes can be applied to each subset, simulating a series of sequential domain transitions. For example, for H&E images of the same tissue type used for a classification task, the dataset can be randomly split and the following enhanced subsets can be generated: (i) one of the stains becomes more intense due to a change in the protocol; (ii) both stains fade (i.e., reduced staining intensity) due to aging of the prepared slides; (iii) one of the stains becomes more saturated than the other in the HSV color space due to a change in the staining machine; (iv) both stains change chromaticity in the HSV color space due to a change in the scanner. Figure 8 Figure 2.1 shows the changes described in (i) and (ii) above (the lower left plot shows the class labels and number of examples for each model version).

[0142] (2.3) Artificial intelligence-based methods for adaptive learning: The generated benchmark data batches from (2.2b) can be run against candidate learning methods, such as the following: (2.3a) Regularization-based methods: These methods prevent the model from forgetting learned knowledge by imposing constraints on the current model weights so that when the current model is updated, the model weights do not deviate too far from the current version of the model. (2.3b) Replay-based methods: These methods retain the most informative examples or their feature representations (e.g., feature vectors extracted from the hidden layers of a neural network) from existing data batches and revisit the most informative examples or their feature representations during training on newly arrived data. (2.3c) Methods that combine regularization and replay. (2.3d) Methods that use meta-learning principles to implement model updates, where there are a few examples per batch (see the "CoPE" method implemented in the Experiments section). (2.3e) Parameter isolation methods. (2.3f) Other techniques applied in conjunction with the above methods. For example, self-supervised learning is performed on new, unlabeled data before applying the above methods. As another example, an adversarial generative model is trained to generate examples that are similar to examples from a previous batch of data.

[0143] (iii) Category Incremental Digital Pathology Model Update:

[0144] (3.1) Digital Pathology Scenario: After the initial development of an AI-based digital pathology algorithm, there may be a need to expand the model in terms of the number of categories. In digital pathology scenarios, this scenario is often encountered due to reasons such as: (i) the changing needs of the end user, (ii) the annotated data comes in batches, each batch has a partially or completely different set of categories from the existing data, and / or (iii) model design choices, for example, the model is initially designed to classify tumors vs. normals, and during development, it is discovered that necrosis and lymphocyte clusters need to be identified to ensure good model performance.

[0145] (3.2) Workflow for selecting the best adaptive learning method: (3.2a) In order to update the model efficiently and effectively without training from scratch using both existing and newly arrived data, a benchmark dataset can be set from the digital pathology domain, and the categories can be split into multiple different subsets, and thus the corresponding examples are split according to their category labels. For example, Fig. 9 This class split is shown for each model version (the lower left plot shows the class labels and number of examples for each model version). (3.2b) Candidate adaptive learning methods are then evaluated to determine which ones are most effective for learning the classes incrementally, and the performance of learning a different set of class rankings for each data batch can be compared.

[0146] (3.3) Artificial intelligence-based methods for adaptive learning: The generated benchmark data batches from (3.2b) can be run against candidate learning methods, such as the following: (3.3a) Regularization-based methods: These methods prevent the model from forgetting learned knowledge by imposing constraints on the current model weights so that when the current model is updated, the model weights do not deviate too far from the current version of the model. (3.3b) Replay-based methods: These methods retain the most informative examples or their feature representations (e.g., feature vectors extracted from the hidden layers of a neural network) from existing data batches and revisit the most informative examples or their feature representations during training on newly arrived data. (3.3c) Methods that combine regularization and replay. (3.3d) Methods that use meta-learning principles to implement model updates, where there are a few examples per batch (see the "CoPE" method implemented in the Experiments section). (3.3e) Parameter isolation methods. (3.3e) Other techniques applied in conjunction with the above methods. For example, self-supervised learning is performed on new, unlabeled data before applying the above methods. As another example, an adversarial generative model is trained to generate examples that are similar to examples from a previous batch of data.

[0147] (iv) Task-based incremental digital pathology model update:

[0148] (4.1) In each of the previously described scenarios, a decision can be made as to whether to formulate each different batch of data as a new task, thereby determining whether to apply a task-incremental approach. Such an approach can specify a separate model part (e.g., a separate set of neurons or a separate layer in a neural network model) for each task, and at each model iteration, only train the model-specific part while the rest of the previously trained parts remain unchanged. Thus, the model can utilize the separate parts to adapt to the changing data and avoid forgetting knowledge learned from previous tasks.

[0149] Application of the adaptive learning framework to other modalities

[0150] The adaptive learning framework can be applied to other imaging modalities and other research areas. The adaptive learning framework is domain-agnostic in the following ways. (1) The preconditioning strategy for initial model training can be applied to other types of data where extensive annotations are required to generate the initial model. (2) The three adaptive learning scenarios for model updating / adaptation have commonalities with scenarios encountered in other computational biomedical research, and thus the adaptive learning method selection strategy can be leveraged by these studies.

[0151] The adaptive learning framework can be applied to federated learning. Federated learning aims to update a global model without sharing data from various data sources and without explicitly sharing local models. The adaptive learning framework can be utilized by federated learning in the following ways: (1) pre-processing the local model and / or the global model to achieve more effective and efficient model updates by utilizing a smaller number of annotations in the target image domain; and (2) updating the local model and / or the global model via one or more of the adaptive learning methods to continuously update the model without retraining on previous data, and selecting the best learning method by applying the model selection workflow described in this article.

[0152] The adaptive learning framework can be applied to multi-model learning. Multi-modal learning aims to integrate knowledge learned from data of different modalities. The adaptive learning framework can be utilized by multi-model learning in the following ways: (1) pre-conditioning models from one or more data modalities to achieve more effective and efficient model updates with a smaller number of annotations in the target image domain; (2) updating from one or more data modalities via one or more of the adaptive learning methods to continuously update the model without retraining on previous data, and selecting the best learning method by applying the model selection workflow described herein; and (3) generating a model to adaptively integrate representations of data from multiple modalities by applying the adaptive learning framework for initial learning iterations and / or continuous updating / adaptation.

[0153] V. Exemplary Systems for Adaptive Learning

[0154] Fig.10 A block diagram illustrating a computing environment 1000 for processing digital pathology images using an artificial intelligence system (e.g., one or more machine learning models) is shown. As further described herein, processing the digital pathology images can include using the digital pathology images to train a machine learning algorithm and / or using a trained (or partially trained) version of the machine learning algorithm (i.e., machine learning model) to transform some or all of the digital pathology images into one or more results.

[0155] like Fig.10 As shown, the computing environment 1000 includes several phases: an image storage phase 1005 , a preprocessing phase 1010 , a labeling phase 1015 , a data enhancement phase 1017 , a training phase 1020 , and a result generation phase 1025 .

[0156] The image storage period 1005 includes one or more image data storage areas 1030 (e.g., Figure 4The one or more image data storage areas are accessed (e.g., by a pre-processing period 1010) to provide a set of digital images 1035 of a pre-selected area from a biological sample slide or an entire biological sample slide (e.g., a tissue slide). Each digital image 1035 stored in each image data storage area 1030 and accessed during the image storage period 1010 may include information about the image data storage area 1030. Figure 4 The depicted network 400 and some or all of the processes described herein may be used to generate digital pathology images. In some embodiments, each digital image 1035 includes image data from one or more scanned slides. Each of the digital images 1035 may correspond to image data from a single sample and / or image data on the day that the underlying image data corresponding to the image was collected.

[0157] Image data may include images, and any information associated with color channels or color wavelength channels, and details about the imaging platform on which the image is generated. For example, tissue sections may need to be stained by applying a staining assay, which contains one or more different biomarkers associated with a chromogenic dye or fluorophore for bright field imaging or fluorescence imaging. The staining assay can use a chromogenic dye for bright field imaging, use an organic fluorophore, quantum dots or an organic fluorophore with quantum dots for fluorescence imaging, or use a stain, a biomarker and any other combination of an observation or imaging device. Exemplary biomarkers include biomarkers of estrogen receptor (ER), human epidermal growth factor receptor 2 (HER2), human Ki-67 protein, progesterone receptor (PR), programmed cell death protein 1 (PD1), etc., wherein tissue sections are detectably labeled with a binding agent (e.g., antibody) for each of ER, HER2, Ki-67, PR, PD1, etc. In some embodiments, digital image and data analysis operations (such as classification, scoring, cox modeling and risk stratification) depend on the type of biomarker used and field of view (FOV) selection and annotation. In addition, typical tissue sections are processed in an automated staining / assay platform that applies a staining assay to the tissue section to produce a stained sample. There are a variety of commercial products on the market that are suitable for use as staining / assay platforms, one example being the product of assignee Ventana Medical Systems, Inc. The stained tissue sections can be provided to an imaging system, such as a microscope or a whole slide scanner having a microscope and / or imaging components, an example of which is the product of assignee Ventana Medical Systems, Inc. iScan DP200. Multiplex tissue slides can be scanned on an equivalent multiplex slide scanner system. Additional information provided by the imaging system can include any information related to the staining platform, including the concentration of the chemicals used for staining, the reaction time of the chemicals applied to the tissue during staining, and / or the pre-analysis conditions of the tissue, such as tissue age, fixation method, duration, how the sections were embedded, cut, etc.

[0158] In the preprocessing phase 1010, each of one, more or all of the digital images in the digital image 1035 set is preprocessed using one or more techniques to generate a corresponding preprocessed image 1040. Preprocessing may include cropping the image. In some cases, preprocessing may further include standardization or resizing (e.g., normalization) to place all features on the same scale (e.g., the same size scale or the same color scale or color saturation scale). In some cases, the image is resized using a minimum size (width or height) of predetermined pixels (e.g., 2500 pixels) or a maximum size (width or height) of predetermined pixels (e.g., 3000 pixels), and optionally maintaining the original aspect ratio. Preprocessing may further include removing noise. For example, the image may be smoothed such as by applying a Gaussian function or Gaussian blur to remove unwanted noise.

[0159] The pre-processed images 1040 may include one or more training images, validation images, test images, and unlabeled images. It should be understood that the pre-processed images 1040 corresponding to the training group, validation group, and unlabeled group need not be accessed simultaneously. For example, an initial group 1040 of training and validation pre-processed images may be accessed first and used to train the machine learning algorithm 1055, and unlabeled input images may subsequently be accessed or received (e.g., at a single or multiple subsequent times) and used by the trained machine learning model 1060 to provide a desired output (e.g., cell classification).

[0160] In some cases, the machine learning algorithm 1055 is trained using supervised training, and some or all of the pre-processed images 1040 are partially or completely manually, semi-automatically, or automatically labeled during the labeling phase 1015 with labels 1045 that identify the “correct” interpretation (i.e., “ground truth”) of various biological materials and structures within the pre-processed images 1040. For example, the labels 1045 may identify a feature of interest (e.g.), a classification of a cell, a binary indication as to whether a given cell is a particular type of cell, a binary indication as to whether a pre-processed image 1040 (or a particular region of a pre-processed image 1040) includes a particular type of depiction (e.g., necrosis or artifact), a slide-level or region-specific representation of the type of depiction (e.g., which identifies a particular type of cell), a count (e.g., which identifies the number of cells of a particular type within a region, the number of artifacts depicted, or the number of necrotic regions), the presence or absence of one or more biomarkers, etc. In some cases, the labels 545 include a location. For example, the marker 1045 can identify the point location of the nucleus of a particular type of cell or the point location of a particular type of cell (e.g., a raw point marker). As another example, the marker 1045 can include a boundary or demarcation, such as the boundary of a depicted tumor, blood vessel, necrotic area, etc. As another example, the marker 1045 can include one or more biomarkers based on biomarker pattern recognition observed using one or more stains. For example, a tissue slide stained for a biomarker such as programmed cell death protein 1 ("PD1") can be observed and / or processed to mark cells as positive cells or negative cells based on the expression level and pattern of PD1 in the tissue. Depending on the purpose feature, a given labeled pre-processed image 1040 can be associated with a single marker 1045 or multiple markers 1045. In the latter case, each marker 1045 can be associated with, for example, an indication of which position or portion within the pre-processed image 1045 the marker corresponds to.

[0161] The labels 1045 assigned during the labeling phase 1015 can be identified based on input from a human user (e.g., a pathologist or image scientist) and / or an algorithm (e.g., an annotation tool) configured to define the labels 1045. In some cases, the labeling phase 1015 can include transmitting and / or presenting part or all of one or more pre-processed images 1040 to a computing device operated by a user. In some cases, the labeling phase 1015 includes presenting at a computing device operated by a user by a labeling controller 1050 using an interface (e.g., using an API), wherein the interface includes an input component to accept input identifying a label 1045 for a feature of interest. For example, a user interface can be provided by the labeling controller 1050, which enables selection of an image or image region (e.g., FOV) for labeling. A user operating a terminal can select an image or FOV using the user interface. Several image or FOV selection mechanisms can be provided, such as specifying a known or irregular shape, or defining an anatomical region of interest (e.g., a tumor region). In one example, the image or FOV is a whole tumor region selected on an IHC slide stained with a H&E stain combination. Image or FOV selection can be performed by a user or by an automatic image analysis algorithm, such as tumor region segmentation on an H&E tissue slide, etc. For example, a user can select an image or FOV as a whole slide or whole tumor, or a whole slide or whole tumor region can be automatically designated as an image or FOV using a segmentation algorithm. Thereafter, a user operating the terminal can select one or more markers 1045 to apply to the selected image or FOV, such as a point location on a cell, a positive marker for a biomarker expressed by the cell, a negative biomarker for a biomarker not expressed by the cell, a boundary around a cell, etc.

[0162] In some cases, the interface can identify which specific markers 1045 are being requested and / or the extent to which specific markers are being requested, which can be communicated to the user via (for example) text instructions and / or visualization. For example, a particular color, size, and / or symbol can indicate that a marker 1045 is being requested for a particular depiction (e.g., a particular cell or region or staining pattern) relative to other depictions within an image. If markers 1045 corresponding to multiple depictions are to be requested, the interface can identify each of the depictions simultaneously or can identify each depiction in sequence (such that providing a marker for one identified depiction triggers identification of the next depiction for labeling). In some cases, each image is presented until a user has identified a particular number of markers 1045 (e.g., markers of a particular type). For example, a given whole slide image or a given patch of a whole slide image can be presented until the user has identified the presence or absence of three different biomarkers, at which point the interface can present different whole slide images or images of different patches (e.g., until a threshold number of images or patches are labeled). Thus, in some cases, the interface is configured to request and / or accept markers 1045 for an incomplete subset of features of interest, and the user can determine which of the possible many depictions will be marked.

[0163] In some cases, the marking stage 1015 includes a marking controller 1050 that implements an annotation algorithm to semi-automatically or automatically mark various features of an image or a region of interest within an image. The marking controller 1050 annotates the image or FOV on the first slide based on input from the user or the annotation algorithm and maps the annotation across the rest of the slide. Depending on the defined FOV, a variety of methods for annotation and registration are possible. For example, the annotation can be automatically or by the user in a process such as VIRTUOSO / VERSO TMOr the like interface selects the annotated whole tumor region on the H&E slide in a plurality of consecutive slides. Since the other tissue slides correspond to consecutive sections from the same tissue block, the marking controller 1050 performs an inter-marker registration operation to map and transfer the whole tumor annotation from the H&E slide to each of the remaining IHC slides in the series. An exemplary method for inter-marker registration is further described in detail in the commonly assigned international application WO2014140070A2 "Whole Slide Image Registration and Cross-Image Annotation Apparatus, System and Method" filed on March 12, 2014, which is hereby incorporated by reference in its entirety for all purposes. In some embodiments, any other method for image registration and generation of whole tumor annotations may be used. For example, a qualified reader such as a pathologist may annotate the whole tumor region on any other IHC slide and execute the marking controller 1050 to map the whole tumor annotation to other digitized slides. For example, a pathologist (or an automated detection algorithm) can annotate a whole tumor region on an H&E slide, thereby triggering analysis of all adjacent serially sectioned IHC slides to determine a whole-slide tumor score for the annotated regions on all slides.

[0164] In the enhancement phase 1017, a set of training images (original images) labeled or unlabeled from preprocessed images 1040 is enhanced using a synthetic image 1052 generated using an enhancement control 1054 that executes one or more enhancement algorithms. Enhancement techniques are used to artificially increase the amount and / or type of training data by adding slightly modified synthetic copies of existing training data or synthetic data newly created from existing training data. As described herein, differences between scanners and laboratories may cause intensity and color variations within digital images. In addition, poor scanning may cause gradient changes and blurring effects, assay staining may produce staining artifacts such as background washing, and cell sizes may vary from tissue to tissue / patient samples. These changes and perturbations may have a negative impact on the quality and reliability of deep learning and artificial intelligence systems. The enhancement techniques implemented in the enhancement phase 1017 can act as regularizers for these changes and perturbations and help reduce overfitting when training machine learning models. Example enhancement techniques include: (i) changing the color space, where a color decomposition is initially performed (e.g., unmixing) and each color is then remixed with a predefined color vector to change the hue, saturation, and intensity of each color, and (ii) increasing image resolution by resizing the image and then cropping it back to the original size, (iii) similar techniques, or (iv) any combination thereof.

[0165] During the training period 1020, the labels 1045 and the corresponding pre-processed images 1040 may be used by the training controller 1065 to train the machine learning algorithm 1055 according to the various workflows described herein. For example, to train the algorithm 1055, the pre-processed images 1040 may be divided into a subset of images 1040a for training (e.g., 90%) and a subset of images 1040b for validation (e.g., 10%). The segmentation may be performed randomly (e.g., 90% / 10% or 70% / 30%), or the segmentation may be performed according to more complex validation techniques (such as K-fold cross validation, leave-one-out cross validation, leave-one-out cross validation, nested cross validation, etc.) to minimize sampling bias and overfitting. The segmentation may also be performed based on the inclusion of enhanced or synthesized images 1052 within the pre-processed images 1040. For example, it may be beneficial to limit the number or ratio of synthesized images 1052 included in the image subset 1040a for training. In some cases, the ratio of original image 1035 to composite image 1052 is maintained at 1:1, 1:2, 2:1, 1:3, 3:1, 1:4, or 4:1.

[0166] In some cases, the machine learning algorithm 1055 includes a CNN, a modified CNN with an encoding layer replaced by a residual neural network ("Resnet"), or a modified CNN with encoding and decoding layers replaced by Resnet. In other cases, the machine learning algorithm 1055 can be any suitable machine learning algorithm configured to locate, classify and / or analyze the pre-processed image 1040, such as a two-dimensional CNN ("2DCNN"), Mask R-CNN, U-Net, Feature Pyramid Network (FPN), Dynamic Time Warping ("DTW") technology, Hidden Markov Model ("HMM"), a pure attention-based model, etc., or a combination of one or more of such technologies - for example, a visual transformer, a CNN-HMM, or an MCNN (multi-scale convolutional neural network). The computing environment 1000 can employ the same type of machine learning algorithm or different types of machine learning algorithms trained to detect and classify different cells. For example, the computing environment 1000 may include a first machine learning algorithm (e.g., U-Net) for detecting and classifying PD1. The computing environment 1000 may also include a second machine learning algorithm (e.g., 2DCNN) for detecting and classifying cluster of differentiation 68 ("CD68"). The computing environment 1000 may also include a third machine learning algorithm (e.g., U-Net) for combined detection and classification of PD1 and CD68. The computing environment 1000 may also include a fourth machine learning algorithm (e.g., HMM) for diagnosis of a therapeutic disease or for prognosis of a subject such as a patient. In other examples according to the present disclosure, other types of machine learning models may also be implemented.

[0167] The training process of the machine learning algorithm 1055 includes selecting hyperparameters of the machine learning algorithm 1055 from the parameter data storage area 1063, inputting the image subset 1040a (e.g., the label 1045 and the corresponding pre-processed image 1040) into the machine learning algorithm 1055, and performing an iterative operation to learn a set of parameters (e.g., one or more coefficients and / or weights) of the machine learning algorithm 1055. Hyperparameters are settings that can be adjusted or optimized to control the behavior of the machine learning algorithm 1055. Most algorithms explicitly define hyperparameters that control different aspects of the algorithm (such as memory or execution cost). However, additional hyperparameters can be defined to adapt the algorithm to specific scenarios. For example, hyperparameters can include the number of hidden units of the algorithm, the learning rate of the algorithm (e.g., 1e-4), the convolution kernel width, or the number of kernels of the algorithm. In some cases, the number of model parameters for each convolution and deconvolution layer and / or the number of convolution kernels for each convolution and deconvolution layer is reduced by half compared to a typical CNN.

[0168] The image subsets 1040a may be input into the machine learning algorithm 1055 as batches of a predetermined size. The batch size limits the number of images displayed to the machine learning algorithm 1055 before parameter updates can be made. Alternatively, the image subsets 1040a may be input into the machine learning algorithm 1055 as a time series or sequentially. In either case, where enhanced or synthesized images 1052 are included within the pre-processed images 1040a, the number of original images 1035 and the number of synthesized images 1052 included within each batch, or the manner in which the original images 1035 and the synthesized images 1052 are fed into the algorithm (e.g., every other batch or image is a batch of original images or original images) may be defined as hyperparameters.

[0169] Each parameter is an adjustable variable so that the value of the parameter is adjusted during training. For example, the cost function or objective function can be configured to optimize the accurate classification of the depicted representation, optimize the characterization of a given type of feature (e.g., characterizing shape, size, uniformity, etc.), optimize the detection of a given type of feature, and / or optimize the accurate positioning of a given type of feature. Each iteration can involve learning a parameter group of the machine learning algorithm 1055 that minimizes or maximizes the cost function of the machine learning algorithm 1055 so that the value of the cost function using the parameter group is less than or greater than the value of the cost function using another parameter group in the previous iteration. The cost function can be constructed to measure the difference between the output predicted using the machine learning algorithm 1055 and the markers 1045 contained in the training data. For example, for a supervised learning-based model, the goal of training is to learn a function "h()" (sometimes also called a hypothesis function) that maps the training input space X to the target value space Y, h:X→Y, so that h(x) is a good predictor of the corresponding value of y. A variety of different techniques can be used to learn the hypothesis function. In some techniques, as part of deriving the hypothesis function, a cost or loss function can be defined to measure the difference between the true value of an input and the predicted value of that input. As part of training, this cost or loss function is minimized using techniques such as backpropagation, random feedback, direct feedback alignment (DFA), indirect feedback alignment (IFA), Hebbian learning, etc.

[0170] The training iterations continue until a stop condition is met. The training completion condition may be configured to be met when, for example, a predefined number of training iterations have been completed, a statistic generated based on testing or validation exceeds a predefined threshold (e.g., a classification accuracy threshold), a statistic generated based on a confidence metric (e.g., an average or median confidence metric or a percentage of confidence metrics above a specific value) exceeds a predefined confidence threshold, and / or a user device that has participated in a training audit closes the training application executed by the training controller 1065. Once the model parameter set is identified via training, the machine learning algorithm 1055 has been trained, and the training controller 1065 uses the image subset 1040b (test or validation data set) to perform an additional process of testing or validation. The validation process may include inputting images from the image subset 1040b into the machine learning algorithm 1055 using validation techniques (such as K-fold cross validation, leave-one-out cross validation, leave-one-out cross validation, nested cross validation, etc.) to adjust hyperparameters and ultimately find an iterative operation of the optimal set of hyperparameters. Once the optimal set of hyperparameters is obtained, a holdout test set of images from the image subset 1040b is input into the machine learning algorithm 1055 to obtain an output, and correlation techniques such as the Bland-Altman method and the Spearman rank correlation coefficient are used to evaluate the relationship of the output to the true value and calculate performance metrics such as error, accuracy, precision, recall, receiver operating characteristic curve (ROC), etc. In some cases, a new training iteration can be initiated in response to receiving a corresponding request or trigger condition from a user device (e.g., initial model development, model update / adaptation, continuous learning, determining drift within the trained machine learning model 1060, etc.).

[0171] It should be understood that other training / validation mechanisms are also contemplated and can be implemented within the computing environment 1000. For example, the machine learning algorithm 1055 can be trained and hyperparameters can be adjusted on images from the image subset 1040a, and images from the image subset 1040b can be used only to test and evaluate the performance of the machine learning algorithm 1055. In addition, although the training mechanisms described herein focus on training new machine learning algorithms 1055. However, these training mechanisms can also be used for initial model development, model updates / adaptation, and continuous learning of existing machine learning models 1060 trained from other data sets, as described in detail herein. For example, in some cases, the machine learning model 1060 may have been pre-processed using images of other objects or biological structures or images of slices from other subjects or studies (e.g., human trials or rodent experiments). In these cases, the machine learning model 1060 can be used for initial model development, model updates / adaptation, and continuous learning using pre-processed images 1040.

[0172] The trained machine learning model 1060 can then be used (during the result generation phase 1025) to process new pre-processed images 1040 to generate predictions or inferences, such as predicting cell center and / or position probabilities, classifying cell types, generating cell masks (e.g., pixel-by-pixel segmentation masks of an image), predicting a diagnosis of a disease or a prognosis of a subject such as a patient, or a combination thereof. In some cases, the mask identifies the locations of depicted cells associated with one or more biomarkers. For example, given a tissue stained for a single biomarker, the trained machine learning model 1060 can be configured to: (i) infer the center and / or position of a cell, (ii) classify cells based on features of a staining pattern associated with the biomarker, and (iii) output a cell detection mask for positive cells and a cell detection mask for negative cells. As another example, given a tissue stained for two biomarkers, the trained machine learning model 1060 can be configured to: (i) infer the center and / or position of cells, (ii) classify cells based on features of the staining patterns associated with the two biomarkers, and (iii) output a cell detection mask for cells that are positive for the first biomarker, a cell detection mask for cells that are negative for the first biomarker, a cell detection mask for cells that are positive for the second biomarker, and a cell detection mask for cells that are negative for the second biomarker. As another example, given a tissue stained for a single biomarker, the trained machine learning model 1060 can be configured to: (i) infer the center and / or position of cells, (ii) classify cells based on features of the cells and the staining patterns associated with the biomarkers, and (iii) output a cell detection mask for positive cells and a cell detection mask for negative cell codes and mask cells classified as tissue cells.

[0173] In some cases, the analysis controller 1080 generates an analysis result 1085, which is used for the entity requesting the processing of the underlying image. The analysis result 1085 may include a mask superimposed on the new pre-processed image 1040 output from the trained machine learning model 1060. In addition or alternatively, the analysis result 1085 may include information calculated or determined according to the output of the trained machine learning model, such as a whole slide tumor score. In an exemplary embodiment, the automated analysis of the tissue slide uses the FDA-approved 510 (k) approved algorithm of the assignee VENTANA. Alternatively or additionally, any other automated algorithm can be used to analyze the selected area of ​​the image (e.g., mask image) and generate a score. In some embodiments, the analysis controller 1080 may further respond to instructions received from a computing device for a pathologist, physician, researcher (e.g., related to a clinical trial), subject, medical professional, etc. In some cases, the communication from the computing device includes an identifier for each of a specific subject group, which corresponds to a request for an analysis iteration for each subject represented in the group. The computing device may further perform analysis and / or provide a recommended diagnosis / treatment for the subject based on the output of the machine learning model and / or analysis controller 1080 .

[0174] It should be understood that computing environment 1000 is exemplary, and computing environments 1000 having different phases and / or using different components are contemplated. For example, in some cases, the network may omit pre-processing phase 1010 such that images used to train the algorithm and / or images processed by the model are raw images (e.g., from an image data store). As another example, it should be understood that each of pre-processing phase 1010 and training phase 1020 may include a controller to perform one or more actions described herein. Similarly, although labeling phase 1015 is depicted as being associated with labeling controller 1050, and although result generation phase 1025 is depicted as being associated with analysis controller 1080, the controllers associated with each phase may further or alternatively facilitate other actions described herein in addition to generating markers and / or generating analysis results. As yet another example, for Fig.10The depiction of the computing environment 1000 shown lacks the depicted representations of: devices associated with a programmer (e.g., who selects an architecture for the machine learning algorithm 1055, defines how various interfaces will operate, etc.); devices associated with a user who provides initial tags or tag reviews (e.g., during the tagging period 1015); and devices associated with a user who requests model processing for a given image (the user may be the same or a different user than the user who provided the initial tags or tag reviews). Although these devices are not depicted, the computing environment 1000 may involve the use of one, more than one, or all of the devices, and may in fact involve the use of multiple devices associated with a respective plurality of users who provide initial tags or tag reviews and / or multiple devices associated with a respective plurality of users who request model processing for various images.

[0175] VI. Techniques for Training Machine Learning Algorithms Using Adaptive Learning Frameworks

[0176] Fig.11 A flow chart is shown illustrating a process 1100 for training a machine learning algorithm using a training image set in accordance with various embodiments. Fig.11 The depicted process 1100 may be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores), hardware, or a combination thereof of a corresponding system. The software may be stored on a non-transitory storage medium (e.g., on a memory device). Fig.11 The process 1100 presented and described below is intended to be illustrative and non-limiting. Fig.11 Various processing steps are depicted as occurring in a particular order or sequence, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in Figure 4 and 10 In the depicted embodiment, Fig.11 The described processing may be performed as part of a training period (e.g., algorithm training 1020) to train a machine learning algorithm using a training image set to generate an algorithm configured to detect, characterize, classify, or a combination thereof, some or all regions or objects within an image.

[0177] Process 1100 begins at block 1105 where a first annotated training set of images is obtained for training a machine learning algorithm to detect, characterize, classify, or a combination thereof, some or all regions or objects within the image. The first annotated training set of images is in a first image domain (e.g., about Fig.10In some cases, the first annotated training set of images is a digital pathology image including one or more types of cells. The first annotated training set of images can depict cells having a staining pattern associated with a biomarker. In some cases, the first annotated training set of images depicts cells having multiple staining patterns associated with multiple biomarkers. The first annotated training set of images can be annotated with tags for (e.g., supervised, semi-supervised, or weakly supervised) training.

[0178] At block 1110, a first annotated training set of images is split, by a data processing system, into image mini-sets, each mini-set representing a different modeling subtask and including a finite number of examples. In some cases, the splitting includes: when only one image mini-set is available, for each different modeling subtask, selecting a subset of categories to be part of or match the number of categories trained in the second stage, and selecting the finite number of examples based on the selected subset of categories; and when multiple image mini-sets are available, for each different modeling subtask, either: (i) mixing examples from multiple image mini-sets, selecting a subset of categories to be part of or match the number of categories trained in the second stage, and selecting the finite number of examples from the mixed examples based on the selected subset of categories, or (ii) selecting an image mini-set from multiple image mini-sets, selecting a subset of categories to be part of or match the number of categories trained in the second stage, and selecting the finite number of examples from the selected image mini-set based on the selected subset of categories.

[0179] At box 1115, in the first stage, the machine learning algorithm is trained using the image mini-set to generate a pre-conditioned machine learning model, which is configured to detect, characterize, classify, or a combination thereof, some or all regions or objects within the new image. In some cases, the first stage further includes: an inner learning loop, in which the machine learning algorithm updates the model weights or parameters on one subtask using a predefined or flexible number of rounds to initialize the conditioned machine learning model to adapt to the target dataset, which generates a loss for the image validation set after the model update, denoted as L-subtask-i for the i-th subtask; and an outer learning loop, in which the purpose is to search for a set of model initializations that, when used to update all subtasks, generates a pre-conditioned machine learning model, each of which has only the finite number of examples, by finding a model initialization that minimizes the sum of all losses denoted as the sum of L-subtask-i, where i ranges from 1 to the number of subtasks, the number of subtasks being calculated based on the subtasks relative to the image validation set for the model initialization.

[0180] In some cases, the first stage of training includes performing an iterative operation to learn a set of parameters to detect, characterize, classify, or a combination thereof, for some or all regions or objects within the image mini-set, the set of parameters maximizing or minimizing a cost function, wherein each iteration involves finding a set of parameters for the machine learning algorithm such that a value of the cost function using the set of parameters is greater or less than a value of the cost function using another set of parameters in a previous iteration, and wherein the cost function is constructed to measure the difference between predictions made for some or all regions or objects using the machine learning algorithm and true value labels provided for the image mini-set.

[0181] At block 1120, a limited number of images from the target dataset are labeled to generate a second annotated training set of images for training a machine learning algorithm to detect, characterize, classify, or a combination thereof, some or all regions or objects within the images. The second annotated training set of images is in a second image domain (different from the first image domain). In some cases, the limited number of images is less than 50 images, 30 images, or 20 images.

[0182] At box 1125, the pre-processed machine learning model is trained in the second stage using the second annotated training set of images to generate a target machine learning model, which is configured to detect, characterize, classify, or a combination thereof, some or all regions or objects within the new image. The number of categories trained in the first stage is a fraction of or matches the number of categories trained in the second stage. In some cases, the second stage further includes: applying the pre-processed machine learning model to generate a feature vector representation for each example in the second annotated training set of images; combining feature vector representations of examples from the same category to generate one representation for each target category, each target category using one representation as a prototype for the target category; generating feature vector representations for the remaining images or image regions in the unlabeled images from the target data set; and comparing each feature vector representation from the unlabeled image to the prototype for the target category based on the distance between the feature vector representation from the unlabeled image and the prototype for the target category.

[0183] In some embodiments, the second stage of training includes performing an iterative operation to learn a set of parameters to detect, characterize, classify, or a combination thereof, for some or all regions or objects within a second annotated training set of images, the set of parameters maximizing or minimizing a cost function, wherein each iteration involves finding a set of parameters for a pre-treated machine learning model such that a value of the cost function using the set of parameters is greater or less than a value of the cost function using another set of parameters in a previous iteration, and wherein the cost function is constructed to measure the difference between predictions made for some or all regions or objects using the pre-treated machine learning model and true value labels provided for the second annotated training set of images.

[0184] At optional block 1130, a target machine learning model is provided. Fig.10 As described, a target machine learning model can be deployed for execution in an image analysis environment.

[0185] At block 1135, a digital pathology scenario is identified. The digital pathology scenario may be a data increment scenario, a domain increment scenario, a category increment scenario, or a task increment scenario.

[0186] At box 1140, an adaptive continuous learning method is selected for updating the target machine learning model for a given digital pathology scenario. In some cases, the adaptive continuous learning method is selected from the group consisting of elastic weight consolidation (EWC), learning without forgetting (LWF), incremental learner and representation learning (iCaRL), continuous prototype evaluation (CoPE), A-GEM, and parameter isolation method. In other cases, the adaptive continuous learning method includes: EWC, LWF, iCaRL, CoPE, A-GEM, parameter isolation method, similar continuous learning method, or any combination thereof.

[0187] At box 1145, the target machine learning model is updated based on the adaptive continuous learning method to generate an updated machine learning model.

[0188] At optional block 1150, an updated machine learning model is provided. Fig.10 As described, the updated machine learning model can be deployed for execution in an image analysis environment.

[0189] At block 1155, a new image is received. The new image can be divided into image tiles of a predetermined size. For example, whole slide images are typically of random size, and machine learning algorithms (such as modified CNNs) learn normalized image sizes more efficiently (e.g., parallel computation on batches of images of the same size; memory constraints), and thus the image can be divided into image tiles of a specific size to optimize analysis. In some embodiments, the image is divided into image tiles of a predetermined size of 64 pixels by 64 pixels, 128 pixels by 128 pixels, 256 pixels by 256 pixels, or 512 pixels by 512 pixels.

[0190] At block 1160, the new image or image patch is input into the target machine learning model or the updated machine learning model. At block 1165, the target machine learning model or the updated machine learning model detects, characterizes, classifies, or a combination thereof, some or all regions or objects within the new image or image patch and outputs an inference based on the detection, characterization, classification, or a combination thereof.

[0191] At optional block 1170, a diagnosis of the subject associated with the image or image patch is determined based on the inferences output by the modified machine learning model.

[0192] At optional block 1175, a treatment is administered to a subject associated with the image or image block. In some cases, the treatment is administered based on (i) an inference output by the machine learning model or a modified machine learning model and / or (ii) a diagnosis of the subject determined at block 1170.

[0193] VII. Examples

[0194]

[0046] Systems and methods implemented in various embodiments may be better understood by reference to the following examples.

[0195] data

[0196] CRC: The following experiments used 100,000 non-overlapping patches of H&E-stained histology images from human colorectal cancer (CRC), consisting of nine tissue classes for training: adipose (ADI), background (BACK), debris (DEB), lymphocytes (LYM), mucus (MUC), smooth muscle (MUS), normal colon mucosa (NORM), cancer-associated stroma (STR), and colorectal adenocarcinoma epithelium (TUM). Fig.12 Some example images are shown in . The test set consists of 7180 image patches with no overlap with the training data. All images are color normalized using the Macenko method.

[0197] The CRC data set is enhanced by changing the staining intensity, color and saturation individually or in combination to simulate the data collected from different dyeing machines, scanners and chromogens. Non-negative matrix decomposition is used to unmix the image. Four settings of color, saturation and intensity are applied to each dyeing from the non-overlapping subsets of the original data set. Each synthetic setting is combined with the original data set for creating different adaptive learning scenarios.

[0198] 1. Each enhancement setting is considered as a domain transformation from the original dataset. Therefore, the dataset can be divided into five data streams, each representing a different enhancement or domain setting, resulting in five empirical domain increment scenarios (five are used in the experiments disclosed in this article, but it should be understood that the dataset can be split into any number of different enhancements or domain settings).

[0199] “n” data streams, resulting in “n” experience domain increment scenarios). The model leverages every experience it has gained from training the model to learn to classify images in the new domain setting.

[0200] 2. Also mix the different augmentations evenly across categories. Use this mixed dataset to create five data streams or experiences of equal size with representations from all categories (five were used in the experiments disclosed herein, but it should be understood that the dataset can be split into any number "n" of data streams or experiences of equal size with representations from all categories). This constitutes a data increment scenario, where each subsequent experience is added to the training data for the model.

[0201] 3. The uniformly mixed dataset is also split into subsets or experiences that each contain different categories to form category increment scenarios. Depending on how the data is split, the model is exposed to different categories in each experience and trained on all categories during the training process.

[0202] Continuous Learning Scenario: Each synthetic setting can be used in combination with the original dataset to create a different continuous learning scenario. Each augmentation setting is a transformation of the domain from the original dataset. The scenarios are used individually as individual data streams or experiences in the domain increment setting. Images from different augmentation settings are mixed together evenly across categories and divided into experiences with equal category representation to form data increment scenarios. Evenly mixed datasets are also split into experiences each containing different categories to form category increment scenarios.

[0203] PatchCam: The PatchCam benchmark dataset consists of 327,680 patches extracted from 400 H&E-stained whole slide images (96x96 pixels in size, 10x magnification) of lymph node sections from breast tissue selected using a hard negative mining mechanism, with a training / validation / test split of 75 / 12.5 / 12.5%. This dataset has two classes, normal and tumor, to indicate the presence of metastatic tissue. This dataset is also normalized using Macenko's method to be consistent with the CRC dataset and easy to compare. The normalized examples from both classes are as shown in Fig.13 The top row includes samples from the normal class, while the bottom row includes samples from the tumor class.

[0204] Continuous learning scenario: Significant domain shifts in the data stream are evaluated by training the model with raw CRC images (stain-normalized) in the first experience and the normalized PatchCam dataset in the second experience.

[0205] method

[0206] The following continual learning methods were experimented to cope with these three scenarios: EWC and online EWC, LwF, iCaRL, CoPE, and A-GEM. All methods were compared with two baselines: 1) training from scratch (upper bound), where the same network architecture (18-layer ResNet) was trained on all data available from all experiences seen so far; and 2) transfer learning or fine-tuning (i.e., lower bound), where the model was trained with the same design as continual learning under exposure only to data available during a specific experience, but the model did not use strategies to mitigate forgetting, but was only fine-tuned to adapt to new categories. For all experiments, the training epochs were set to 15 and the batch size was 16. When testing with A-GEM, the same ResNet architecture was used in a multi-head setting based on the findings of each head, where each head was used for a different task. The stochastic gradient descent optimizer was used starting with a learning rate of 0.1, a momentum of 0.9, and a weight decay of 0.00001 applied after epochs 10 and 13.

[0207] Example 1.-Data increment setting

[0208] The first experiment involves a data delta scenario, where more data is sequentially fed into the model, which is then updated based only on the latest data, without accessing any older datasets on which the model was previously trained. The newer data stream will still have the same categories as the older data stream, but may be shifted in distribution. The mixed dataset used in this experiment should have a uniform distribution between experiences.

[0209] Models with this setting are trained using a method called Continuous Prototype Evaluation (CoPE). CoPE is an online data increment algorithm that uses prototypes to represent the most important features from the data. The prototypes continue to evolve as the model learns to keep up with the data changes and is able to make predictions accurately. CoPE also incorporates balanced replay to ensure that all categories are well represented in the replay population. The data is fed in an online manner as mini-batches or mini-experiences, that is, the model sees each data sample only once and is therefore trained in a single round. A mini-experience size of 128 samples (i.e., each mini-experience has only 128 samples) is used to train with a batch size of 10 and a momentum of 0.99 to reduce forgetting. Therefore, for the data increment scenario created for the enhanced dataset, each of the 5 experiences has 99 mini-experiences.

[0210] At the end of training, the test stream, which utilizes samples from different experiences, has an average accuracy of 76%. Fig.14 The accuracy of the test stream at the end of each main experience under the data increment setting of the enhanced CRC data set using CoPE is shown. The experience is split into a user-defined size of 128 images, and 99 mini experiences are generated in each experience to be fed into the model in an online manner. The numbers shown in the figure are the numbers at the end of each 99th mini experience or each main experience. The accuracy of the classification gradually improves, indicating that the model benefits from more data. As the model learns from the newer training stream, the accuracy of each test stream also improves, indicating that the model has not forgotten what it has learned previously. The equal representation from all synthetic domains in each experience enables transfer learning and test streams to benefit from all data. It is also worth noting that according to subsequent experience, the model performs equally well on the test stream. For example, at the end of main experience 1, the model has similar performance on test stream 1 and test streams 2, 3 and 4, indicating a certain transfer learning. The mixed enhanced dataset used here has equal representation from all synthetic domains, producing similar distributions between experiences. Between experience 3 and experience 4, the accuracy is also improved by less than 1%, indicating that the model performs equally well for 80% of the data.

[0211] Example 2.- Field increment setting

[0212] The second experiment utilized a domain increment scenario with 5 experiences, where the hue, saturation, and intensity values ​​of two stains (eosin and hematoxylin) were varied to varying degrees, mimicking images acquired using different stainers, scanners, and reagents. Fig.15Examples from 5 experiences are shown in . Each row corresponds to one of the nine tissue types, and three example images are drawn from each enhancement setting (i.e., domain). Domain 0 (columns 1 to 3): CRC data set with normalized staining. Domain 1 (columns 4 to 6): increased eosin staining intensity, simulating the scene of increased concentration of eosin solution or extended staining time. Domain 2 (columns 7 to 9): reduced eosin intensity, simulating the scene of aged slides with faded staining. Domain 3 (columns 10 to 12): differently changed chromaticity of hematoxylin and eosin. Domain 4 (columns 13 to 15): changed chromaticity and increased saturation level of eosin and hematoxylin. In the domain increment scenario, each of the three column segments can be regarded as a separate domain. These domains can be mixed together, and each domain can be divided into experiences with representations of all categories from the data increment scenario, or divided into mixed domain experiences with non-overlapping category subsets for each category increment scenario.

[0213] For this scenario, a method called learning without forgetting (LwF) is used. LwF is a combination of fine-tuning and refinement. LwF uses only the minimal data corresponding to the current task to learn task-specific parameters for the new / current task without compromising performance on old tasks. Unlike traditional regularization that penalizes changes in parameters based on their importance, LwF penalizes changes in the mapping from input to output. The loss function consists of two terms: a cross entropy loss for the current task and a refinement loss that prevents previously acquired knowledge from being forgotten.

[0214] LwF performs well for 3 out of 5 predefined domains with over 86% accuracy. Although the evaluation accuracy for domains 1 and 2 during the experience when the model was trained on the corresponding domains was 88% and 93%, respectively, the knowledge retention is not satisfactory. However, when the domain-specific data is no longer available, there is a forgetting of nearly 28% of the acquired domain-specific knowledge, especially for domains 1 and 2. The results are shown in Fig.16 In Figure 2, accuracy (left) and forgetting (right) of the test stream at the end of each primary experience in the domain increment setting using LwF with the augmented CRC dataset. At the end of training, the model performed with an accuracy of 86% to 94% on 3 of the 5 predefined domains. Although the model performed well on domain 1 (88%) and domain 2 (93%), the acquired domain-specific knowledge was forgotten during subsequent experiences when the specific domain data was no longer available. At the end of training, the test stream forgetting metric was about 28%, indicating that there was a loss of accuracy from domain 1 and domain 2 during the tire training process. This experiment demonstrates that a model presented with data from different domains in successive data streams can be trained to adapt to new domains fairly well while still maintaining performance on previous domains that are no longer accessible to it.

[0215] Example 3.- Category increment setting

[0216] In a third experiment, the model was trained using a class increment setting with 3 experiences and 6 categories as described herein so that the model had access to data from only two categories during each experience, and newer categories were gradually added at each experience. Incremental Class and Representation Learning (iCaRL) is the adaptive learning strategy used here. iCaRL dynamically selects samples from the data stream, and each category has its own set of samples. iCaRL updates both parameters and samples when it sees new data. iCaRL classifies with the closest mean of the samples. iCaRL involves representation learning using refinement and prototyping, where the augmented data set includes data from the current task and stored samples, and model parameters are updated based on the cross entropy loss of the newer categories and the refinement loss of the previously learned categories.

[0217] There are two baselines to which the iCaRL algorithm is compared: 1) training from scratch, where the same neural net is trained on all available data until a certain experience (upper bound). That is, the iCaRL algorithm is trained on two categories during the first experience, on four categories during the second experience, and on all six categories during the third experience; and 2) transfer learning or fine-tuning (lower bound), where the model is trained using the same design as adaptive learning while being exposed to only two categories during each of the three experiences, but the model does not use strategies to mitigate forgetting and is simply fine-tuned to adapt to the newer categories. Results are shown in Figure 2. Fig.17 As shown in . As shown, it is observed that iCaRL performs on par with the upper bound of training from scratch, which only utilizes a small fraction of the data, thus also providing a significant computational advantage. Transfer learning performs well on experience 0, but with exposure and learning of newer categories, previously acquired knowledge is forgotten. Despite being computationally comparable to iCaRL, transfer learning performance is unsatisfactory due to catastrophic forgetting.

[0218] Example 4.- Comparison of methods

[0219] Continuous learning using enhanced CRC datasets. For fair comparison, domain and data increment experiments have 5 experiences, and class increment experiments have 4 experiences, where the first three experiences each have 2 categories, and the last experience has the remaining 3 categories. It was found that A-GEM provided the best results with task descriptors, so the method was considered to be a task increment method, where each experience introduces a new set of categories to the model with a task ID. The hyperparameters of each method are determined by grid search. CoPE and A-GEM are considered to be online few-shot methods and are trained using only 1 round. iCaRL was experimented under three settings. The first setting has 4 experiences, where the first three experiences each have 2 categories, and the last experience has the remaining three categories. The second setting also has 4 experiences, but has 3 categories in the first experience, and the remaining experiences each have 2 categories. The final setting has 3 experiences with 3 categories each. The order of categories is the same in all settings and is in ascending order. LwF is experimented with to continuously learn on the original CRC dataset and the normalized PatchCam dataset in a domain-increment setting, where one tumor type is considered as one domain.

[0220] Fig.18 Figure 2 shows the evaluation accuracy of the continual learning methods at the end of training for three designed scenarios (data, domain, and category increment). In the data and domain increment scenarios, LwF and iCaRL are comparable to the upper-bound baselines. Overall, the category increment scenario is a more difficult task to learn. iCaRL has the highest accuracy of 83%. A-GEM is the only method that utilizes the task increment scenario for testing and evaluation.

[0221] Data Incremental Scenario: At the end of training, LwF has an overall accuracy of 93%, with <1% forgetting of previously acquired knowledge. This is 4% higher than the lower bound and within 0.5% of the upper baseline. The accuracy figures for each experience are shown in Fig.19 middle. Specifically, Fig.19 Data increment experiences tested with different CL methods listed in the legend are shown, including two baselines. Each experience consists of its own test stream, which contains examples from smaller data batches belonging only to that experience. Each sub-graph shows how the model performs on the test stream evaluated at the end of training on all experiences. The gray area shows experiences for which the model has not yet been trained, where the model is expected to perform poorly on these experiences. However, once the model has been trained on a particular experience, ideally the model should not forget what it has learned and retain the knowledge for the rest of the training process, that is, the accuracy should remain high in the non-gray area for all test streams. LwF has the highest overall accuracy at the end of training.

[0222] The accuracy of the classification gradually improves, indicating that the model benefits from more data. As the model learns from the newer training streams, the accuracy of each test stream also improves, indicating that the model has not forgotten what it has previously learned. Another observation is the performance of the iCaRL method. iCaRL is designed as a category increment method, but the concept of storing samples representing categories in each domain should theoretically be more efficient than EWC and LwF. The largest memory size tested may not only be insufficient to store "n" categories (as in category increment), but also insufficient to store "n" categories times "d" domains.

[0223] Domain increment scenario: Fig. 20 As shown in Figure 2, iCaRL outperforms EWC and LwF, where the best hyperparameters are selected from grid search. Fig. 20 As shown, 5 domain incremental experiences, including two baselines, were tested using different CL methods as listed in the legend. Each experience consists of its own test stream, which contains examples from domains belonging only to that experience. Each sub-graph shows how the model performs on the test stream evaluated at the end of training on all experiences. The gray area shows the experience on which the model has not been trained, where the model is expected to perform poorly on these experiences. However, once the model has been trained on a particular experience, ideally the model should not forget what it has learned and retain the knowledge for the rest of the training process, that is, the accuracy should remain high in the non-gray area for all test streams. It can be seen that iCaRL is comparable to the upper bound baseline, while the performance of EWC and LwF is closer to the lower bound baseline. Note that due to the small number of training examples (20% of the total examples), the upper bound of the first experience is lower (accuracy of 0.57).

[0224] Interestingly: 1) the model retains knowledge from some domains better than others. Specifically, the model retains knowledge from some domains better than others. Fig.15 4 to 6 in the column) or reduce (field 2; Fig.15 It is more challenging to retain the learned knowledge on the datasets with columns 7 to 9 in Fig.15 This is not the case for changes in hue and / or saturation of the coloring (domains 3 and 4). There is some transfer learning between the domains. Even after training only on domain 0, the model performs well on the test stream corresponding to domain 4. Performance on both domains remains high at the end of the training process. Since the fourth domain is generated with changes in hue of the coloring, these results show that the model can cope with a range of hue changes while continuing to learn and retain knowledge from a sequential data stream.

[0225] Category Incremental Scenario: iCaRL significantly outperforms other methods. All tested methods including the lower bound baseline fail to retain knowledge about categories learned from previous experience. iCaRL performs with an overall accuracy of 88%, which is about 6% lower than the joint training upper bound. This is still beneficial considering the smaller load on data storage and resources. Fig.21 Four categories of incremental experiences, including two baselines, are shown, tested using different CL methods as listed in the legend. Each experience consists of its own test stream, which contains examples from categories that belong only to that experience. Each sub-figure shows how the model performs on the test stream evaluated at the end of training on all experiences. The gray area shows experiences for which the model has not yet been trained - the model is expected to perform poorly on these experiences. However, once the model has been trained on a particular experience, ideally the model should not forget what it has learned and retain the knowledge for the rest of the training process, that is, the accuracy should remain high in the non-gray area for all test streams. It can be seen that iCaRL is the only CL method that performs well on the current experience and does not completely forget previous knowledge. Experience 0, on which the model was first trained, is the most easily forgotten experience, with an accuracy of 64% at the end of the training process.

[0226] Few-shot online continual learning: The biggest gain for both CoPE and A-GEM comes from the amount of training data per experience. A-GEM is tested using a type incremental setting with task IDs, where model updates are based on only 128 randomly selected examples stored in memory at 1 epoch, while producing comparable results to iCaRL, which is not an online method and is trained for 15 epochs. The overall accuracy at the end of training is 79%, which is more than 50% higher than the lower bound baseline. Detailed results are shown in Fig. 22 The dataset was initially designed as a category-incremental scenario, where each experience is assigned a separate task ID and trained in an online fashion on a multi-head architecture. The overall accuracy is about 79%, while the baseline lower bound is 27%. A-GEM also performs significantly worse when used without task IDs. A-GEM is unable to retain any prior knowledge gained.

[0227] CoPE was tested in a domain increment scenario, where the dataset was divided into mini-experiences each with the same number of examples as the mini-batch size to simulate online training. It has an overall accuracy that is 67% to 11% higher than the lower bound baseline. It is interesting to compare the results from CoPE with the LwF and EWC results from the domain increment scenario. For the latter two methods, the models performed poorly on the test streams from domains 1 and 2. While CoPE was unable to help retain knowledge from domain 1 (accuracy < 25%), the model had an accuracy of 60% from domain 2. The online setting may help retain more information. CoPE was also found to be sensitive to the softmax temperature. In contrast to using a temperature > 1 in other refinement methods, it was tested at a lower temperature than for a harder Softmax distribution as recommended in the literature. A finer sweep of the hyperparameters may produce better results. Another point to keep in mind is how each experience is split into mini-batches or mini-experiences in CoPE. Each experience in the tested setting has 128 samples or examples. Not all categories were equally represented in each of the mini-experiences, which may also have affected overall accuracy.

[0228] The comparison between CoPE and the baseline is shown in Fig.23 The experience is split into user-defined sizes of 128 images, producing 99 mini-experiences within each experience that are fed into the model in an online manner. The numbers shown in the left figure are the numbers at the end of every 99th mini-experience or each main experience. On the right are the results from the fine-tuned / naive baseline. For both CoPE and the baseline, equal representations from all synthetic domains in each experience allow transfer learning and the test stream to benefit from all data, but for CoPE, the overall accuracy is higher.

[0229] The impact of category grouping on continuous learning

[0230] Results from a third experiment using a different class grouping setting are shown in Fig.24Three settings were tested: A. 4 experiences with 2 categories in the first three and 3 categories in the last; B. 4 experiences with 3 categories in the first experience and 2 categories in the remaining experiences; C. 3 experiences with 3 categories each. Comparing the first two subgraphs (both with 4 experiences), there is a drop of about 6.5% in overall accuracy. The difference between the two settings is how many categories the model was trained on in the first experience. The assumption here is based on curriculum learning, i.e. knowledge is better captured if harder tasks are followed by easier ones. Here, starting with three categories may make it harder for the model to learn and retain knowledge, which is reflected in the drop in accuracy, even though the rest of the experiences were trained on only two categories in each of the two settings. The same assumption can also be applied to the third setting. Despite having fewer experiences, the model has more to learn in each experience - 3 categories vs. 2 categories in setting 1, which in turn leads to a drop in accuracy of nearly 8%.

[0231] Continuous learning from multiple tumor types

[0232] Both EWC and LwF were evaluated using this experiment, and LwF produced the results presented in Fig.25 CRC was designed as the first domain, and PatchCam was designed as the second domain. The model started out well, with over 90% accuracy on the CRC test stream, but by the end of training, the model did forget some of the knowledge acquired in the first experience. At the end of the training process, the model had an accuracy of about 70% on the CRC test stream and about 76% on the PatchCam test stream.

[0233] in conclusion

[0234] This systematic study characterizes the performance of various continual learning methods for different scenarios using augmented digital pathology images and evaluates them when the models are presented with different tumor types. The datasets were evaluated according to regularization and replay methods. Although EWC and LwF performed relatively well in data and domain increment scenarios, reshaping methods (iCaRL and A-GEM) were required to prevent catastrophic forgetting in the more challenging category increment scenario. The few-shot online methods tested require additional fine-tuning of hyperparameters and experimental settings to fully understand their effectiveness. It would also be interesting to investigate how changes in the images (from a clinical perspective) and shifting patient populations, disease progression, and / or disease types (subtypes) affect the performance of these CL methods, providing insights into the feasibility of applying these methods in clinical settings. In these experiments, knowledge related to staining intensity was found to be difficult to retain, and the models seemed to be less sensitive to chromatic changes in the range tested. Although some of these results suggest difficulties in learning tumor classification from DP images, this study shows the potential of continual learning in adapting to changes in clinical histopathology image acquisition factors.

[0235] VIII. Other considerations

[0236] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions, which when executed on the one or more data processors causes the one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, which includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein.

[0237] The terms and expressions that have been adopted are used as terms of description and not of limitation, and when such terms and expressions are used, there is no intention to exclude any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the claimed invention. Therefore, it should be understood that although the claimed invention has been specifically disclosed through embodiments and optional features, modifications and variations of the concepts disclosed herein may be adopted by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims.

[0238] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the preferred exemplary embodiments will provide a feasible description for implementing various embodiments for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope set forth in the appended claims.

[0239] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it should be understood that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components can be shown as parts in block diagram form to avoid obscuring the embodiments in unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques can be shown without unnecessary details to avoid obscuring the embodiments.

Claims

1. A computer-implemented method comprising: obtaining, at a data processing system, a first annotated training set of images for training a machine learning algorithm to detect, characterize, classify, or a combination thereof, some or all regions or objects within the images, wherein the first annotated training set of images is in a first image domain; splitting, by the data processing system, the first annotated training set of images into mini-sets of images, each mini-set representing a different modeling subtask and including a finite number of examples; training, by the data processing system in a first phase, the machine learning algorithm using the image mini-set to generate a preconditioned machine learning model, the preconditioned machine learning model being configured to detect, characterize, classify, or a combination thereof, some or all regions or objects within a new image; labeling, by the data processing system, a limited number of images from the target dataset to generate a second annotated training set of images for training a machine learning algorithm to detect, characterize, classify, or a combination thereof, some or all regions or objects within the images, wherein the second annotated training set of images is in a second image domain; and The pre-processed machine learning model is trained by the data processing system in a second stage using a second annotated training set of the images to generate a target machine learning model, wherein the target machine learning model is configured to detect, characterize, classify, or a combination thereof, some or all regions or objects within the new image, wherein the number of categories trained in the first stage is a fraction of or matches the number of categories trained in the second stage. 2 . The computer-implemented method of claim 1 , wherein the first annotated training set of images are digital pathology images comprising one or more types of cells.

3. The computer-implemented method of claim 1 or 2, wherein the splitting comprises: When only one image mini-set is available, for each different modeling subtask, selecting a subset of categories to be a fraction of or to match the number of categories trained in the second stage, and selecting the limited number of examples based on the selected subset of categories; and When multiple image mini-sets are available, for each different modeling subtask, either: (i) mixing examples from the multiple image mini-sets, selecting a subset of categories to be part of or match the number of categories trained in the second stage, and selecting the limited number of examples from the mixed examples based on the selected subset of categories, or (ii) selecting an image mini-set from the multiple image mini-sets, selecting a subset of categories to be part of or match the number of categories trained in the second stage, and selecting the limited number of examples from the selected image mini-set based on the selected subset of categories.

4. The computer-implemented method of claim 1 , 2 or 3, wherein the second phase further comprises: applying the pre-trained machine learning model to generate a feature vector representation for each example in a second annotated training set of images; combining the feature vector representations of examples from the same category to generate one representation per target category, each target category using one representation as a prototype for the target category; generating a feature vector representation of an image or image region for a remaining portion of the unlabeled images from the target dataset; as well as Each feature vector representation from the unlabeled image is compared to the prototype for the target class based on a distance between the feature vector representation from the unlabeled image and the prototype for the target class.

5. The computer-implemented method of claim 1 , 2 or 3, wherein the first stage further comprises: an inner learning loop, wherein the machine learning algorithm updates model weights or parameters for a subtask using a predefined or flexible number of rounds for initializing the preconditioned machine learning model to adapt to the target dataset, which generates a loss for an image validation set after the model update, denoted as L-subtask-i for the i-th subtask; as well as an outer learning loop, wherein the objective is to search for a set of model initializations that, when used to update all subtasks, generates the pretrained machine learning model by finding a model initialization that minimizes the sum of all losses, denoted as the sum L-subtask-i, where i ranges from 1 to the number of subtasks, the number of subtasks being calculated based on a validation set of images of the subtasks with respect to the model initialization.

6. A computer-implemented method according to claim 1 or 2, wherein the training of the first stage includes iterative operations to learn a set of parameters to detect, characterize, classify, or a combination thereof, some or all regions or objects within the image mini-set, the set of parameters maximizing or minimizing a cost function, wherein each iteration involves finding the set of parameters for the machine learning algorithm such that the value of the cost function using the set of parameters is greater or less than the value of the cost function using another set of parameters in a previous iteration, and wherein the cost function is constructed to measure the difference between the predictions made for some or all of the regions or objects using the machine learning algorithm and the true value labels provided for the image mini-set.

7. A computer-implemented method according to claim 1 or 2, wherein the training of the second stage includes iterative operations to learn a parameter set to detect, characterize, classify or a combination thereof for some or all regions or objects within a second annotated training set of the image, the parameter set maximizes or minimizes a cost function, wherein each iteration involves finding the parameter set for the pre-treated machine learning model so that the value of the cost function using the parameter set is greater or less than the value of the cost function using another parameter set in a previous iteration, and wherein the cost function is constructed to measure the difference between the predictions made for some or all of the regions or objects using the pre-treated machine learning model and the true value labels provided for the second annotated training set of the image.

8. The computer-implemented method of any one of claims 1 to 7, further comprising: Identify digital pathology scenarios; selecting an adaptive continuous learning method for updating the target machine learning model given the digital pathology scenario; as well as The target machine learning model is updated based on the adaptive continuous learning method to generate an updated machine learning model. 9 . The computer-implemented method of claim 8 , wherein the digital pathology scenario is a data increment scenario, a domain increment scenario, a category increment scenario, or a task increment scenario.

10. A computer-implemented method according to claim 8, wherein the adaptive continuous learning method is selected from the group including: elastic weight consolidation (EWC), learning without forgetting (LWF), incremental learner and representation learning (iCaRL), continuous prototype evaluation (CoPE), A-GEM and parameter isolation method.

11. The computer-implemented method of claim 1 or 8, further comprising providing the target machine learning model and / or the updated machine learning model.

12. The computer-implemented method of claim 11, wherein the providing comprises deploying the target machine learning model and / or the updated machine learning model in a digital pathology system.

13. The computer-implemented method of any one of claims 1 to 12, further comprising: receiving, by the data processing system, a new image; inputting the new image into the target machine learning model or the updated machine learning model; detecting, characterizing, classifying, or a combination thereof, some or all regions or objects within the new image by the target machine learning model or the updated machine learning model; as well as An inference is output by the target machine learning model or the updated machine learning model based on the detection, characterization, classification, or a combination thereof.

14. The computer-implemented method of claim 13, further comprising: A diagnosis of the subject associated with the new image is determined by a user, wherein the diagnosis is determined based on the inference output by the target machine learning model or the updated machine learning model.

15. The computer-implemented method of claim 14, further comprising administering treatment to the subject by the user based on (i) inferences output by the target machine learning model or the updated machine learning model and / or (ii) the diagnosis of the subject.

16. A computer-implemented method according to claim 1, wherein training the machine learning algorithm includes implementing meta-learning principles to enable the first stage to generate the pre-conditioned machine learning model using the limited number of examples.

17. A computer-implemented method according to claim 1, wherein training the pre-conditioned machine learning model includes implementing meta-learning principles to enable the second stage to generate the target machine learning model using the limited number of images.

18. A system comprising: one or more data processors; as well as A non-transitory computer-readable storage medium comprising instructions which, when executed on the one or more data processors, cause the one or more data processors to perform any of the steps of the method according to any one of claims 1 to 17.

19. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform any of the steps of the method according to any one of claims 1 to 17.

Images