Adaptive Learning Framework for Digital Pathology

Abstract

This provides an adaptive learning framework for efficient initial model development, model updates, and / or adaptation to different image domains. [Solution] The adaptive learning framework consists of initial model development, model updating, and / or adaptation to different image domains. Initial model development involves stage 1, model pre-adjustment, improving learning skills through training using existing annotated datasets; and stage 2, target model training, extending the learning skills learned in stage 1 to different image domains (target domains) that require fewer annotations in the target domain than conventional learning methods. Model updating and adaptation to new datasets involve identifying digital pathology scenarios, selecting an adaptive learning method based on the scenarios, updating the model using the adaptive learning method, and adapting it to the new datasets.

Description

Technical Field

[0001] The present disclosure relates to digital pathology, and more particularly, to techniques for efficient development of initial models, as well as efficient model updating and / or adaptation to different image domains using an adaptive learning framework.

Background Art

[0002] Digital pathology involves scanning slides (e.g., histopathology or cytopathology glass slides) into digital images interpretable on a computer screen. Tissues and / or cells within the digital images can then be examined by digital pathology image analysis and / or interpreted by a pathologist for various reasons, including disease diagnosis, assessment of response to treatment, and drug development for combating diseases. To examine tissues and / or cells (substantially transparent) within the digital image, pathology slides can be prepared using various staining assays (e.g., immunohistochemistry) that selectively bind to tissue and / or cell components. Immunofluorescence (IF) is a technique for analyzing assays that bind fluorescent dyes to antigens. Multiple assays responsive to different wavelengths can be utilized on the same slide. These multiplexed IF slides enable an understanding of the complexity and heterogeneity of the immune landscape of the tumor microenvironment, as well as the potential impact on the tumor's response to immunotherapy. In some assays, the target antigen for a stain in the tissue may be called a biomarker. Digital pathology image analysis can then be performed on the digital image of the stained tissue and / or cells to identify and quantify the staining for antigens (e.g., biomarkers indicating various cells such as tumor cells) in the biological tissue.

[0003] Artificial intelligence and machine learning-based methods and / or techniques show great potential in digital pathology image analysis, including cell detection, counting, localization, classification, and patient prognosis. Many computing systems with machine learning techniques, including convolutional neural networks (CNNs), have been proposed for image classification and digital pathology image analysis, such as cell detection and classification. For example, a CNN can have a series of convolutional layers as hidden layers, and this network structure enables the extraction of representational features for object / image classification and digital pathology image analysis. In addition to object / image classification, machine learning techniques have also been implemented for image segmentation. Image segmentation is the process of dividing a digital image into multiple segments (sets of pixels, also known as image objects). The purpose of segmentation is to simplify and / or modify the representation of an image to make it more meaningful and easier to analyze. For example, image segmentation is typically used to find objects such as cells and boundaries (lines, curves, etc.) within an image. To perform image segmentation on large datasets (e.g., full slide pathology images), the image is first divided into many small patches. A computing system equipped with machine learning techniques is trained to classify each pixel of these patches, and all pixels of the same class are combined into one segmented area of ​​each patch, and then all segmented patches are combined into one segmented image (e.g., a segmented whole-slide pathology image). Subsequently, machine learning techniques are further implemented to predict or further classify segmented areas (e.g., cells positive for a given biomarker, cells negative for a given biomarker, or cells without staining expression) based on the expression features associated with the segmented areas. [Overview of the Initiative]

[0004] Artificial intelligence and machine learning-based methods have achieved excellent performance in digital pathology. However, developing such models is extremely time-consuming and resource-intensive. Not only are hundreds of thousands of annotations required to build a reliable model from scratch in the initial development stage, but once developed, the model's generalizability to intangible data is limited, leading to unavoidable continuous investment in developing new models, even for related tasks. This specification discloses a framework for reducing resource requirements throughout the entire development process of AI-based digital pathology algorithms, including initial model development and subsequent model updates, improvements, and adaptation to different datasets. Specifically, disclosed herein is a model preconditioning stage using existing annotated datasets that are relevant to, but not necessarily similar to, the target dataset for building the model, thereby requiring only a small number of annotations to generate a model with reasonable accuracy in the initial model development. In the subsequent model update and adaptation stages, an adaptive learning workflow is used for multiple digital pathology scenarios and strategies to select the best learning method for efficient model updates without having to train all the data from scratch.

[0005] In various embodiments, a computer implementation method is provided, which includes: obtaining a first annotated training set of images in a data processing system for training a machine learning algorithm to detect, characterize, classify, or combine parts or all of the regions or objects in an image, wherein the first annotated training set of images is located in a first image domain; dividing the first annotated training set of images into minisets of images, each miniset representing a distinct modeling subtask and containing a limited number of examples; and using the minisets of images to generate a pre-configured machine learning model configured to detect, characterize, classify, or combine parts or all of the regions or objects in a new image, in a first step... The data processing system labels a limited number of images from a target dataset in order to train a rugorism and generate a second annotated training set of images for training a machine learning algorithm to detect, characterize, classify or combine some or all regions or objects in an image, wherein the second annotated training set of images is in a second image domain, and the data processing system trains a second pre-prepared machine learning model using the second annotated training set of images in order to generate a target machine learning model configured to detect, characterize, classify or combine some or all regions or objects in a new image, wherein the number of classes targeted in the first stage is part of or matches the number of classes targeted in the second stage.

[0006] In some embodiments, the first annotated training set of images is a digital pathology image containing one or more types of cells.

[0007] In some embodiments, the partitioning includes, when only one miniset of images is available, for each separate modeling subtask, selecting a subset of classes that is part of or matches the number of classes targeted in the second stage, and selecting a limited number of examples based on the selected subset of classes; and, when multiple minisets of images are available, for each separate modeling subtask, (i) mixing examples from multiple minisets of images, selecting a subset of classes that is part of or matches the number of classes targeted in the second stage, and selecting a limited number of examples from the mixed examples based on the selected subset of classes; or (ii) selecting one miniset of images from multiple minisets of images, selecting a subset of classes that is part of or matches the number of classes targeted in the second stage, and selecting a limited number of examples from the selected miniset of images based on the selected subset of classes.

[0008] In some embodiments, the second step further includes applying a machine learning model pre-configured to generate a feature vector representation for each example in a second annotated training set of images; combining feature vector representations from examples of the same class to generate one representation for each target class and use one representation for each target class as a prototype in the target class; generating feature vector representations for the remaining unlabeled images or image regions from the target dataset; and comparing each feature vector representation from the unlabeled images to the prototype in the target class based on the distance between the feature vector representation from the unlabeled images and the prototype for the target class.

[0009] In some embodiments, the first stage further includes an internal learning loop in which a machine learning algorithm updates model weights or parameters on one subtask in a predetermined or flexible number of epochs to initialize a machine learning model pre-conditioned for adaptation to a target dataset, which generates a loss in a validation set of images after the model update, indicated as L-subtask-i with respect to the i-th subtask; and an external learning loop whose purpose is to find a set of model initializations that generate a machine learning model pre-conditioned when used to update all subtasks, each with only a limited number of examples, by finding a model initialization that minimizes the sum of all losses, indicated as the sum of L-subtask-i, where i ranges from 1 to the number of subtasks computed from the validation set of images of the subtask with respect to the model initialization.

[0010] In some embodiments, the first stage of training involves performing iterative operations to learn a set of parameters to detect, characterize, classify, or combine, some or all of the regions or objects within a miniset of images, each iteration involving finding a set of parameters for a machine learning algorithm such that the value of the cost function using the set of parameters is greater than or less than the value of the cost function using a different set of parameters in the previous iteration, the cost function being constructed to measure the difference between the predictions made with some or all of the regions or objects using the machine learning algorithm and the ground truth labels given to the miniset of images.

[0011] In some embodiments, the second stage of training involves performing iterative operations to learn a set of parameters to detect, characterize, classify, or combine, some or all of the regions or objects in a second annotated training set of images, each iteration involving finding a set of parameters for a pre-defined machine learning model such that the value of the cost function using the set of parameters is greater than or less than the value of the cost function using a different set of parameters in the previous iteration, the cost function being constructed to measure the difference between the predictions made for some or all of the regions or objects using the pre-defined machine learning model and the ground truth labels given to the second annotated training set of images.

[0012] In some embodiments, the computer implementation method further includes identifying a digital pathology scenario, selecting an adaptive continuous learning method for updating a target machine learning model based on the digital pathology scenario, and updating the target machine learning model based on the adaptive continuous learning method to generate an updated machine learning model.

[0013] In some embodiments, the digital pathology scenario is a data-incremental scenario, a domain-incremental scenario, a class-incremental scenario, or a task-incremental scenario.

[0014] In some embodiments, the adaptive continuous learning method is selected from the group including elastic weight reinforcement (EWC), learning without forgetting (LWF), incremental learner and representation learning (iCaRL), continuous prototype evaluation (CoPE), A-GEM, and parameter separation methods.

[0015] In some embodiments, the computer implementation method further includes providing a target machine learning model and / or an updated machine learning model.

[0016] In some embodiments, providing includes deploying a target machine learning model and / or an updated machine learning model to a digital pathology system.

[0017] In some embodiments, the computer implementation method further includes: receiving a new image by a data processing system; inputting the new image into a target machine learning model or an updated machine learning model; detecting, characterizing, classifying, or a combination thereof of some or all regions or objects within the new image by the target machine learning model or the updated machine learning model; and outputting inferences based on the detection, characterization, classification, or a combination thereof by the target machine learning model or the updated machine learning model.

[0018] In some embodiments, the computer implementation method further includes a user determining a diagnosis of a subject associated with a new image, the diagnosis being determined based on inferences output by a target machine learning model or an updated machine learning model.

[0019] In some embodiments, the computer implementation method further includes the user performing a procedure on the subject based on (i) inferences output by the target machine learning model or an updated machine learning model, and / or (ii) a diagnosis of the subject.

[0020] In some embodiments, training a machine learning algorithm involves implementing a meta-learning principle so that the first stage generates a pre-conditioned machine learning model using a limited number of examples.

[0021] In some embodiments, training a pre-conditioned machine learning model involves implementing a meta-learning principle so that a second stage can generate a target machine learning model using a limited number of images.

[0022] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed by the one or more data processors, cause the one or more data processors to execute some or all of the one or more methods disclosed herein.

[0023] In some embodiments, a computer program product is provided that is tangibly embodied on a non-transitory machine-readable storage medium and contains instructions configured to cause one or more data processors to execute some or all of the one or more methods disclosed herein.

[0024] The terms and expressions used are used as terms for explanation and not for limitation, and the use of such terms and expressions is not intended to exclude any equivalents of the features shown and described, and it is recognized that various changes are possible within the scope of the claimed invention. Therefore, although the invention described in the claims is specifically disclosed by embodiments and any features, changes and modifications of the concepts disclosed herein may be reclassified by those skilled in the art, and it should be understood that such changes and modifications are considered to be within the scope of the invention as defined by the appended claims.

Brief Description of the Drawings

[0025] Aspects and features of various embodiments will become more apparent from the description of the examples in connection with the accompanying drawings.

[0026] [Figure 1] Illustrates the challenges of artificial intelligence-based digital pathology algorithms for efficiently adapting to the increasingly complex data.

[0027] [Figure 2] Illustrates efficient model development using an adaptive learning framework according to various embodiments.

[0028] [Figure 3] Shows a comparison of resource requirements for training from scratch, transfer learning, and adaptive learning.

[0029] [Figure 4] Shows a typical network for generating digital pathology images according to some embodiments.

[0030] [Figure 5] Shows an overview of an adaptive learning framework according to various embodiments.

[0031] [Figure 6] Shows examples of IHC images and ground truth for cell detection according to various embodiments.

[0032] [Figure 7] Shows an example of adaptively learning an increasing number of data having similarities for hematoxylin and eosin (H&E) tissue type classification (data incremental) according to various embodiments.

[0033] [Figure 8] Shows an example of adaptively learning an increasing number of data sources for H&E tissue type classification (domain incremental) according to various embodiments.

[0034] [Figure 9] Shows an example of adaptive learning of an increasing number of tissue types for H&E tissue type classification (class incremental) according to various embodiments.

[0035] [Figure 10] Shows a typical computing environment for processing digital pathology images using a machine learning / deep learning model according to various embodiments.

[0036] [Figure 11]A flowchart illustrating the process for training and using machine learning models according to various embodiments of this disclosure is provided.

[0037] [Figure 12] An example of images from a CRC training dataset according to various embodiments is shown.

[0038] [Figure 13] This shows an example of an image from the PatchCam dataset according to various embodiments.

[0039] [Figure 14] This shows the accuracy of the test stream at the end of each major experience under a class incremental setting using an enhanced CRC dataset with COPE having two baselines according to different embodiments.

[0040] [Figure 15] Examples of images from augmented training datasets according to various embodiments are shown.

[0041] [Figure 16] The accuracy (left) and forgetting (right) of the test stream at the end of each main experience under a domain incremental setting using an augmented CRC dataset with LwF according to various embodiments are shown.

[0042] [Figure 17] This shows the accuracy of the test stream at the end of each major experience under a data incremental setting using an enhanced CRC dataset with iCaRL according to various embodiments.

[0043] [Figure 18] The overall accuracy at the end of training data, domain, and class incremental scenarios from different sequential learning methods according to various embodiments is shown, along with two baselines.

[0044] [Figure 19] This document demonstrates the performance of continuous learning methods in data incremental scenarios according to various embodiments.

[0045] [Figure 20] This document demonstrates the performance of continuous learning methods in domain incremental scenarios according to various embodiments.

[0046] [Figure 21] This document demonstrates the performance of a continuous learning method in class incremental scenarios according to various embodiments.

[0047] [Figure 22] The results from A-GEM, which has extended CRC data set as task incremental scenarios according to various embodiments, are shown.

[0048] [Figure 23] This shows the accuracy of the final test stream of each main experience under a domain incremental setting using an augmented CRC dataset with CoPE according to various embodiments.

[0049] [Figure 24] This paper demonstrates the effects of class grouping and experience settings on class incremental performance using iCaRL according to various embodiments.

[0050] [Figure 25] This demonstrates continuous learning from different tumor types in domain incremental settings using LwF according to various embodiments. [Modes for carrying out the invention]

[0051] While specific embodiments are described, these embodiments are presented only as examples and are not intended to limit the scope of protection. The apparatus, methods, and systems described herein may be embodied in various other forms. Furthermore, various omissions, substitutions, and modifications of the exemplary methods and systems described herein may be made without departing from the scope of protection.

[0052] I. Overview Artificial intelligence and machine learning-based methods have achieved unprecedented performance in solving complex problems in digital pathology-based analysis, thereby automatically suggesting potential lesions and thus improving diagnostic reliability and reducing subjectivity, thereby lowering the workload of pathologists. A common strategy to ensure the performance of reliable and dependable digital pathology algorithms is to collect and train models from the outset with numerous annotations for each target image domain in digital pathology, both during (a) the initial model training phase and (b) the model update phase for performance improvement or adaptation to different image domains. However, fundamental tasks for histopathology image analysis, such as image classification, semantic segmentation, and object detection, require manual creation and curation of annotations by pathologists. Such time-consuming and resource-intensive model development has presented two major challenges in digital pathology.

[0053] Firstly, there is the challenge of efficiently and effectively establishing artificial intelligence models for targeted digital pathology datasets. Due to their nature of requiring large amounts of data and having limited generalizability to invisible data distributions, most deep learning models take months to develop, especially when a large set of annotations specific to the target dataset needs to be generated and validated by certified experts. However, such time-consuming and resource-intensive development processes face challenges in meeting the ever-increasing demand for such models. Considering the emerging multiplexing technologies in the market, a diverse range of new assays will be developed in the future, along with deep learning-based digital pathology analysis. Furthermore, constantly evolving patient populations and unpredictable disease outbreaks necessitate efficient model development strategies to address changing needs in diagnosis.

[0054] Secondly, there is the challenge of efficiently adapting existing model systems to related but different datasets. Image digitization conditions are constantly evolving, especially with technological advancements. The use of different staining pigments, variations in staining, and the platforms of digital scanners and vendors result in changes in the appearance of digitized images. The proliferation and availability of digital pathology also results in a nearly continuous stream of data with biological variance due to heterogeneous disease samples to be analyzed. The ever-increasing volume and complexity of data necessitates the development of artificial intelligence algorithms that can adapt and maintain performance under various conditions. Traditional methods of retraining with newer data batches using manual task-specific annotations rapidly increase the demands on data storage, training time, and computing power, subsequently increasing costs, delaying product releases, and ultimately reaching a point where it becomes prohibitively expensive for organizations, as shown in Figure 1.

[0055] Active learning-based methods have been designed since the pre-deep learning era to reduce the number of annotations required for model development. These methods utilize heuristic scoring strategies to query a small subset of the most informative unlabeled examples within a dataset. By iteratively adding only such a small set of selected examples for model retraining, active learning aims to gradually improve model performance and thus avoid the need to annotate many potentially less informative examples. However, models developed using active learning are only applicable to test images with the same or identical distribution as the training images. For example, a model trained by active learning on one immunohistochemistry (IHC) assay (domain 1) cannot generalize to another domain, e.g., another IHC assay (domain 2). As another example, a model trained on one tissue type cannot be easily transferred to other tissue types. Thus, unlike the adaptive learning framework disclosed herein, such methods cannot fully address the challenges of model inflexibility and lack of model adaptability. Furthermore, active learning cannot help reduce the demands on computational power and memory space because it requires iterative model retraining using all existing and newly annotated examples in each development iteration.

[0056] Alternatively, transfer learning and domain adaptation have been used to improve model generalization to some extent. In transfer learning, all or part of the model weights are targeted on a large annotated dataset applied to a new image domain, while domain adaptation aims to use only a few annotations from the target domain, or no annotations at all, for model development. However, both frameworks can suffer from catastrophic forgetting, where a model targeted in one image domain (source domain) is used to train on another image domain (target domain), resulting in a significant decrease in its performance on the source domain and thus "forgetting" the knowledge learned from the previous training process on the source domain data. Domain adaptation algorithms aim to produce models that perform well on both the source and target domains, even without annotations from the target domain (unsupervised domain adaptation), but current practice still relies on a validation set from the target domain for model selection, which inevitably tends to overfit the validation set.

[0057] To address these and other challenges, various embodiments disclosed herein relate to methods, systems, and computer-readable storage media for reducing the resource requirements for subsequent iterations of model development, which aim to (1) reduce the resource requirements for developing artificial intelligence models for the invisible distribution of digital pathology data in the initial model development stage, and (2) improve or adapt the initial artificial intelligence model constructed in accordance with (1) to related but not entirely identical datasets.

[0058] To reduce resource demands during the initial model development phase, techniques are employed to pre-condition artificial intelligence systems for learning useful features from existing digital pathology data. Such designs leverage existing annotated digital pathology datasets that are relevant to but not necessarily similar to the target data, thereby allowing the AI ​​system to distill its learning skills through a pre-training phase using these relevant datasets. Herein, “learning skills” include one or more of the following: the best set of model initialization, the best set of model weights that can generalize to unseen data, the best set of model architecture, etc. These learning skills enable the pre-conditioned model to require only a small set of annotations to achieve reasonable performance. For example, a 75% accuracy can be achieved with <50 annotated images for classifying tissue types from tumor types that the model has not trained on, and >2000 images for training a model specific to this tumor type using a conventional AI model.

[0059] To reduce resource requirements in subsequent model development or model adaptation to different datasets, sequential learning techniques and algorithms are implemented to enable model updates using sequentially acquired data without training the model from scratch on all existing and new datasets. While sequential learning algorithms provide solutions for learning from a series of data streams, the challenge lies in preventing the loss of old knowledge (catastrophic forgetting). To select the most effective sequential learning algorithm for diverse digital pathology data, a set of strategies targeting various model update requirements commonly encountered in digital pathology applications is designed, and the corresponding algorithms are implemented to learn sequentially without training from scratch and without degrading model performance on previously encountered data (e.g., catastrophic forgetting by machine learning models).

[0060] These various techniques and algorithms are implemented within an adaptive learning framework that includes the following features and advantages.

[0061] (1) Effectively reduce resource requirements throughout the entire development process of AI-based digital pathology algorithms. The adaptive learning framework addresses the challenges of a process that requires resources for both initial model development and subsequent model updates / adaptations, as shown in Figure 2. Compared to conventional AI strategies (training from scratch dots), the adaptive learning framework enables efficient initial model development with a small number of annotations (e.g., less than 50) and faster model improvement (adaptive learning dots) to reach convergence to good model performance. The arrows illustrate the model update process, where various methods can be applied, including self-supervised learning, continuous adaptive learning, pre-supervised continuous adaptive learning (with a small set of annotations), and conventional training (with sufficient annotations).

[0062] (2) The adaptive learning framework can be applied to the development of models that target images from different image domains in digital pathology. In this specification, an image domain refers to a set of images having a specific sample distribution. Examples of different image domains include different image modalities (IHC images vs. H&E images), IHC assays (Ki67 vs. CK7 vs. IHC targeting PDL1-CK7), tumor types and subtypes (breast cancer vs. lung cancer), scanners from different vendors, etc.

[0063] (3) The adaptive learning framework leverages existing digital pathology datasets (from one or more image domains) during the preconditioning phase for initial model development, and the corresponding preconditioned models can be applied to the same or related but different image domains as the image domains in the preconditioning phase.

[0064] (4) The algorithm designed for selecting a model update strategy enables the effective extension of existing models to adapt to different image domains after initial model development.

[0065] (5) Users such as developers can flexibly apply only the initial model development strategy, only the model update strategy, or both, depending on the availability of the initial model and the need to update the model to a new image dataset or a new domain. Alternatively, self-managed pre-training can be combined with preconditioning and model update strategies in one or all of the model development iterations.

[0066] (6) The adaptive learning framework is domain-independent and can be applied to other imaging modalities as well as other computational studies such as multimodal analysis, gene sequencing signal analysis, and survival modeling.

[0067] Figure 3 shows a comparison of resource requirements for training from scratch, transfer learning, and the adaptive learning framework described herein. Resources (y-axis) refer to the total number of images that need to be computed for training various subsequent model versions (x-axis), where if one image is processed N times, it is counted as N images. Training from scratch consumes most resources, and the demand increases rapidly. Transfer learning requires only resources to compute new batches of data. Adaptive learning requires similar computational resources to transfer learning, but its increase is negligible.

[0068] In one exemplary embodiment, a computer implementation process is provided, which in a data processing system obtains a first annotated training set of images for training a machine learning algorithm to detect, characterize, classify, or combine parts or all of the regions or objects in an image, wherein the first annotated training set of images is located in a first image domain; the data processing system divides the first annotated training set of images into minisets of images, each miniset representing a distinct modeling subtask and containing a limited number of examples; and the data processing system uses the minisets of images in a first step to generate a pre-configured machine learning model configured to detect, characterize, classify, or combine parts or all of the regions or objects in a new image. The present invention comprises training a machine learning algorithm, labeling a limited number of images from a target dataset to generate a second annotated training set of images for training the machine learning algorithm to detect, characterize, classify, or combine some or all regions or objects in an image, wherein the second annotated training set of images is in a second image domain, and training a second-stage pre-facing machine learning model using the second annotated training set of images to generate a target machine learning model configured to detect, characterize, classify, or combine some or all regions or objects in a new image, wherein the number of classes targeted in the first stage is part of or matches the number of classes targeted in the second stage.

[0069] In some embodiments, the computer implementation process further includes identifying a digital pathology scenario, selecting an adaptive sequential learning method for updating a target machine learning model based on the digital pathology scenario, and updating the target machine learning model based on the adaptive sequential learning method to generate an updated machine learning model.

[0070] Preferably, the various techniques described herein can improve the robustness of machine learning models (for example, improve the accuracy of cell classification).

[0071] II. Definition As used herein, when an act is “based on” something, this means that the act is at least partially based on at least a part of something.

[0072] As used herein, the terms “substantially,” “approximately,” and “about” are defined as largely specified but not necessarily fully specified (and fully specified) as understood by those skilled in the art. In any disclosed embodiment, the terms “substantially,” “approximately,” or “about” may be replaced by “within [percentage]” of the specified, the percentages including 0.1, 1, 5, and 10%.

[0073] As used herein, the terms “sample,” “biological sample,” “tissue,” or “tissue sample” refer to any sample containing biomolecules (e.g., proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (e.g., veterinary animals such as humans, cats, dogs, horses, cattle, and pigs, as well as laboratory animals such as mice, rats, and primates), insects, annelids, spiders, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections and needle biopsies), cell samples (such as cytological smears, including Pap smears or blood smears, or cell samples obtained by microdissection), or cell fractions, fragments, or organelles (such as those obtained by lysing cells and separating their components by centrifugation). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical biopsy or needle biopsy), nipple aspirate, earwax, milk, vaginal fluid, saliva, swabs (such as oral swabs), or any material containing biomolecules derived from the initial biological sample. In certain embodiments, the term “biological sample” as used herein refers to a sample (such as a homogenized or liquefied sample) prepared from a tumor or a portion thereof obtained from the subject.

[0074] As used herein, the terms “biological material,” “biological structure,” or “cellular structure” refer to natural materials or structures that include all or part of a living structure (e.g., cell nucleus, cell membrane, cytoplasm, chromosomes, DNA, cell, cell mass, etc.).

[0075] As used herein, “digital pathology image” refers to a digital image of a stained specimen.

[0076] As used herein, the term “cell detection” refers to the detection of the location and characteristics of a cell or cellular structure (e.g., cell nucleus, cell membrane, cytoplasm, chromosomes, DNA, cell, cell mass, etc.) pixel.

[0077] As used herein, the term “target region” refers to a region of an image containing image data intended to be evaluated in an image analysis process. The target region includes any region of an image, such as a tissue region, that is intended to be analyzed in an image analysis process (e.g., tumor cells or staining expression).

[0078] As used herein, the terms “tile” or “tiled image” refer to a single image corresponding to a portion of an entire image or to an entire slide. In some embodiments, a “tile” or “tiled image” refers to an area of ​​the entire slide scan or an area of ​​interest having (x,y) pixel dimensions (e.g., 1000 pixels × 1000 pixels). For example, consider an entire image divided into M columns of tiles and N rows of tiles, where each tile in the M × N mosaic contains a portion of the entire image, i.e., the tile at position M1, N1 contains a first portion of the image, the tile at position M1, N2 contains a second portion of the image, and the first and second portions are distinct. In some embodiments, each tile may have the same dimensions (pixel size × pixel size). In some examples, tiles may partially overlap and represent overlapping areas of the entire slide scan or the area of ​​interest.

[0079] As used herein, the terms “patch,” “image patch,” or “mask patch” refer to a container of pixels corresponding to an entire image, an entire slide, or a portion of an entire mask. In some embodiments, a “patch,” “image patch,” or “mask patch” refers to an area of ​​an image or mask, or an area of ​​interest having (x,y) pixel dimensions (e.g., 256 pixels × 256 pixels). For example, a 1000-pixel × 1000-pixel image divided into 100-pixel × 100-pixel patches would contain 10 patches (each patch containing 1000 pixels). In other embodiments, a patch has (x,y) pixel dimensions and overlaps with each other, sharing one or more pixels with another “patch,” “image patch,” or “mask patch.”

[0080] III. Generation of Digital Pathological Images Digital pathology involves the interpretation of digitized images to accurately diagnose subjects and guide therapeutic decisions. Digital pathology solutions establish image analysis workflows that can automatically detect or classify biological objects of interest, such as positive and negative tumor cells. An exemplary digital pathology solution workflow includes acquiring tissue slides, scanning a pre-selected area or the entirety of the tissue slides with a digital image scanner (e.g., a whole-slide image (WSI) scanner) to acquire digital images, performing image analysis on the digital images using one or more image analysis algorithms, and potentially detecting and quantifying each object of interest based on the image analysis (e.g., quantitative or semi-quantitative scoring such as positive, negative, moderate, weak, etc.) (e.g., counting or identifying the object-specific or cumulative area of ​​each object of interest).

[0081] Figure 4 shows an exemplary network 400 for generating digital pathological images. The fixation / embedding system 405 fixes and / or embeds tissue specimens (e.g., specimens containing at least a portion of at least one tumor) using a fixative (e.g., a liquid fixative such as a formaldehyde solution) and / or an embedding material (e.g., a histological wax such as paraffin wax, and / or one or more resins such as styrene or polyethylene). Each specimen may be fixed by exposing the specimen to the fixative for a predetermined period (e.g., at least 3 hours) and then dehydrating the specimen (e.g., via exposure to an ethanol solution and / or a clearing intermediate). The embedding material can be impregnated when the specimen is in a liquid state (e.g., when heated).

[0082] Fixation and / or embedding of specimens are used to preserve specimens and slow their degradation. In histology, fixation generally refers to an irreversible process that uses chemicals to preserve chemical composition, maintain the natural structure of a specimen, and protect cellular structures from degradation. Fixation may also harden cells or tissues for sectioning. Fixatives may enhance the preservation of specimens and cells by using cross-linking proteins. Fixatives may bind to and cross-link some proteins, and denature other proteins by dehydration, which can harden tissues and inactivate enzymes that would normally degrade specimens. Fixatives may also kill bacteria.

[0083] Fixatives can be administered, for example, by perfusion and immersion of the prepared sample. Various fixatives can be used, including methanol, Buin fixatives and / or formaldehyde fixatives, such as neutral buffered formalin (NBF) or paraffin-formalin (paraformaldehyde-PFA). If the sample is a liquid sample (e.g., a blood sample), the sample may be smeared onto a slide and dried before fixation. While the fixation process can help preserve the structure of the sample and cells for histological examination, fixation can mask tissue antigens, thereby reducing antigen detection. Therefore, since formalin can crosslink antigens and mask epitopes, fixation is generally considered a limiting factor in immunohistochemistry. In some cases, additional processes are performed to reverse the crosslinking effect, including treating the fixed sample with anhydrous citraconic acid (a reversible protein crosslinking agent) and heating.

[0084] Embedding may involve impregnating a specimen (e.g., a fixed tissue specimen) with a suitable histological wax, such as paraffin wax. Histological waxes may be insoluble in water or alcohol, but soluble in paraffin solvents such as xylene. Therefore, it may be necessary to replace the water in the tissue with xylene. To do this, the specimen may first be dehydrated by gradually replacing the water in the specimen with alcohol, which can be achieved by passing the tissue through ethyl alcohol of increasing concentration (e.g., 0 to about 100%). After replacing the water with alcohol, the alcohol may be replaced with xylene, which is miscible with alcohol. Since histological wax may be soluble in xylene, the molten wax can be filled with xylene, filling the spaces that were previously filled with water. The wax-filled specimen may be cooled to form a hardened block, which can then be clamped to a microtome, vibratome, or compressstorm to cut sections. In some cases, deviations from the exemplary procedure described above may result in paraffin wax penetration, which may inhibit the penetration of antibodies, chemicals, or other fixatives.

[0085] Next, a tissue slicer 410 may be used to section the fixed and / or embedded tissue sample (e.g., a tumor sample). Sectioning is the process of cutting thin slices (e.g., 4-5 μm thick) of the sample from a tissue block for mounting onto microscope slides for examination. Sectioning may be performed using a microtome, vibratome, or compressstorm. In some cases, the tissue can be rapidly frozen in dry ice or isopentane and then cut with a cold knife in a refrigerated cabinet (e.g., a cryostat). Other types of coolants, such as liquid nitrogen, can be used to freeze the tissue. Sections for use in bright-field and fluorescence microscopy are generally about 4-10 pm thick. In some cases, the sections can be embedded in epoxy or acrylic resin, which may allow for the cutting of thinner sections (e.g., <2 μm). These sections may then be mounted on one or more glass slides. A coverslip may be placed on top to protect the sample sections.

[0086] Since tissue sections and the cells within them are substantially transparent, slide preparation typically further involves staining the tissue sections (e.g., automated staining) to make the relevant structures more visible. In some cases, staining is performed manually. In some cases, staining is performed semi-automatically or automatically using staining system 415. The staining process involves exposing sections of tissue or liquid-fixed samples to one or more different stains (e.g., sequentially or simultaneously) to express different characteristics of the tissue.

[0087] For example, staining may be used to mark specific types of cells and / or flag specific types of nucleic acids and / or proteins to aid in microscopic examination. The staining process generally involves adding a dye or stain to a sample to confirm or quantify the presence of specific compounds, structures, molecules, or features (e.g., intracellular features). For example, staining may help identify or highlight specific biomarkers from tissue sections. In other examples, staining can be used to identify or highlight biological tissues (e.g., muscle fibers or connective tissue), cell populations (e.g., different blood cells), or organelles within individual cells.

[0088] One exemplary type of tissue staining is histochemical staining, which stains tissue structures using one or more chemical dyes (e.g., acid dyes, basic dyes, pigment sources). Histochemical staining may be used to show general aspects of tissue morphology and / or cellular microanatomy (e.g., distinguishing the cell nucleus from the cytoplasm, showing lipid droplets, etc.). An example of histochemical staining is H&E. Other examples of histochemical stains include trichrome stains (e.g., Masson's trichrome), Schiff periodate (PAS), silver stains, and iron stains. The molecular weight of histochemical staining reagents (e.g., dyes) is generally about 500 kilodaltons (kD) or less, although some histochemical staining reagents (e.g., Alcian blue, phosphomolybdic acid (PMA)) may have molecular weights of up to 2000 or 3000 kD. An example of a high molecular weight histochemical staining reagent is α-amylase (about 55 kD), which is sometimes used to show glycogen.

[0089] Another type of tissue staining is IHC, also known as "immunostaining," which uses a primary antibody that specifically binds to a target antigen of interest (also called a biomarker). IHC can be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or fluorophore). In indirect IHC, the primary antibody first binds to the target antigen, and then a secondary antibody conjugated to a label (e.g., a chromophore or fluorophore) binds to the primary antibody. Because antibodies have a molecular weight of approximately 150 kD or more, the molecular weight of IHC reagents is much larger than that of histochemical staining reagents.

[0090] Various types of staining protocols may be used to perform staining. For example, an exemplary IHC staining protocol includes using a hydrophobic barrier line around the sample (e.g., tissue section) to prevent reagent leakage from the slide during incubation; treating the tissue section with reagents to block endogenous sources of nonspecific staining (e.g., enzymes, free aldehyde groups, immunoglobulins, and other irrelevant molecules that may mimic specific staining); incubating the sample with permeabilizing buffer to facilitate the penetration of antibodies and other staining reagents into the tissue; incubating the tissue section with primary antibody at a specific temperature (e.g., room temperature, 6-8°C) for a certain period (e.g., 1-24 hours); rinsing the sample with washing buffer; then incubating the sample (tissue section) with secondary antibody at another specific temperature (e.g., room temperature) for another period; rinsing the sample again with water buffer; incubating the rinsed sample with a chromogen (e.g., DAB: 3,3'-diaminobenzidine); and washing away the chromogen to stop the reaction. In some cases, counterstaining is then used to identify the entire "landscape" of the sample and serves as the primary color reference used for detecting tissue targets. Counterstains may include, for example, hematoxylin (blue to purple stain), methylene blue (blue stain), toluidine blue (stain that makes nuclei deep blue and polysaccharides pink to red), nuclear fast red (also known as Kern-Echtroth dye, a red stain), methyl green (green stain), and non-nuclear coloriactive stains such as eosin (pink stain). As those skilled in the art will know, staining can be performed by carrying out other immunohistochemical staining techniques.

[0091] In another example, the H&E staining protocol can be performed for tissue section staining. The H&E staining protocol involves applying a hematoxylin stain mixed with a metal salt or mordant to the sample. The sample can then be rinsed with a weak acid solution to remove excess staining (differentiation), followed by bluing in weak alkaline water. After the application of hematoxylin, the sample can be counterstained with eosin. It will be understood that other H&E staining techniques can be performed.

[0092] In some embodiments, staining can be performed using various types of stains, depending on which features of interest are targeted. For example, DAB can be used for various tissue sections for IHC staining, and DAB yields a brown color that represents the features of interest in the stained image. In another example, alkaline phosphatase (AP) may be used for skin tissue sections for IHC staining, as the DAB color may be masked by melanin pigment. With respect to primary staining techniques, applicable stains may include, for example, basophilic and acidophilic stains, hematine and hematoxylin, silver nitrate, tricolor stains, etc. Acidic dyes may react with cationic or basic components in tissue or cells, such as proteins and other components in the cytoplasm. Basic dyes may react with anionic or acidic components in tissue or cells, such as nucleic acids. As described above, an example of a staining system is H&E. Eosin may be a negatively charged pink acidic dye, and hematoxylin may be a purple or blue basic dye containing hematine and aluminum ions. Other examples of staining may include periodic acid-Schiff (PAS) staining, Masson's tricolor, Alcian blue, funguison, and reticuline staining. In some embodiments, different types of stains may be used in combination.

[0093] The sections may then be mounted on corresponding slides, and the imaging system 420 may then scan or image them to generate raw digital pathological images 425a-n. To magnify the stained specimens, a microscope (e.g., an electron microscope or an optical microscope) may be used. For example, an optical microscope may have a resolution of less than 1 μm, such as about several hundred nanometers. An electron microscope may be used to observe finer details in the nanometer or sub-nanometer range. An imaging device (combined with or separate from the microscope) images the magnified biological specimen to acquire image data, such as a multi-channel image (e.g., multi-channel fluorescence) having several channels (e.g., 10-16 channels). The imaging device may include, but is not limited to, a camera (e.g., an analog camera, a digital camera), optical elements (e.g., one or more lenses, a sensor focus lens group, a microscope objective lens, etc.), an imaging sensor (e.g., a charge-coupled device (CCD), a complementary metal-oxide-semiconductor (CMOS) image sensor, etc.), photographic film, etc. In digital embodiments, the imaging device may include multiple lenses that cooperate to ensure on-the-fly focusing. An image sensor, such as a CCD sensor, can capture a digital image of a biological sample. In some embodiments, the imaging device is a bright-field imaging system, a multispectral imaging (MSI) system, or a fluorescence microscope system. The imaging device may utilize invisible electromagnetic radiation (e.g., UV light) or other imaging techniques to acquire images. For example, the imaging device may comprise a microscope and a camera configured to acquire images magnified by the microscope. Image data received by the analysis system may be identical to and / or derived from raw image data acquired by the imaging device.

[0094] Next, images of the stained sections may be stored in a storage device 425, such as a server. Images may be stored on local, remote, and / or cloud servers. Each image may be stored associated with a subject identifier and date (e.g., the date the sample was collected and / or the date the image was taken). Images may further be transmitted to another system (e.g., a system associated with a pathologist, an automated or semi-automated image analysis system, or a machine learning training and deployment system, as described in more detail herein).

[0095] It is understood that modifications to the process described for network 400 are intended. For example, if the sample is a liquid sample, embedding and / or sectioning may be omitted from the process.

[0096] IV. Adaptive Learning Framework Figure 5 shows that the adaptive learning framework includes two components: (505) initial model development, and (510) model updating and / or adaptation to different image domains. The two components 505;510 can be applied separately or together, such as initial model development and subsequent model updating and / or adaptation to different image domains.

[0097] Development of the initial model For efficient development of the initial model, a two-stage development strategy (shown in Figure 5) is as follows: Stage 1: Model preconditioning, in which the artificial intelligence system (e.g., the artificial intelligence system described in detail with respect to Figure 10) utilizes existing annotated datasets and improves its learning skills through training with these datasets. Stage 2: Target model training, in which the artificial intelligence system uses the learning skills learned in Stage 1 to extend itself to a different image domain (target domain) where fewer annotations are required in the target domain than in conventional learning methods.

[0098] The aforementioned "learning skills" include one or more of the following: the best set of model initialization, the best set of model weights that can generalize to invisible data, the best set of model architecture, etc. The best set is determined using one or more metrics to measure model performance, such as precision or area under the curve (AUC). The learning skills are then applied to the target image domain, resulting in fewer annotations being required on the target domain than in conventional machine learning.

[0099] To achieve model preconditioning, a meta-learning strategy may be employed, where an existing dataset (e.g., the training dataset) is split into minisets, each representing a distinct modeling subtask, containing only a small number of examples, and thus forming a large number of subtasks. The splitting can be performed during training time, and different data splits can be performed in each model training iteration. By training an artificial intelligence system with these subtasks, the system becomes capable of searching for superior solutions in the model optimization landscape, which can be generalized to any relevant small subtask without overfitting to a specific subtask, thus preconditioning the AI ​​system to stage 2 for unseen target domains. Generally, the number of classes in stage 1 training is set to match some of or the number of classes in stage 2 training. In a stage 2 class-incremental scenario, the number of classes in stage 1 can be increased, while in a stage 2 domain and data-incremental scenario, the number of classes from stage 1 must match. In specific examples, the number of classes in stage 2 can be greater than that in stage 1 (explained in detail in relation to Figure 9). In classification tasks (e.g., binary and multi-class classification), class labels are at the image level, for example, a class (or multiple classes) is assigned to the entire image. In prediction tasks (e.g., dense prediction), such as image segmentation and object detection, class labels are at a finer level, for example, in object detection, each distinct object in the image has its class and location labels. The training datasets used to pre-configure the model are related to each other but can have varying degrees of similarity to those of the target domain.

[0100] To sample existing annotated datasets, the following criteria may be employed: (1) If only one annotated dataset is available, for each subtask, a subset of classes may be randomly selected so that it is part of or matches the number of classes in Stage 2, and some examples may be randomly selected from the selected classes. In this scenario, there are many subtasks with different or partially different classes. (2) If multiple annotated datasets are available, for each subtask, examples may be mixed from multiple datasets and then the mixed dataset may be implemented in the same way as in (1), or a specific dataset may be randomly selected first, and then a subset of classes within that dataset may be randomly selected so that it is part of or matches the number of classes in Stage 2. In both scenarios, (a) sampling strategies other than complete randomness may also be employed, for example, some classes or some datasets may be sampled more frequently than others, and / or (b) the examples within each subtask may be divided into a training subset and a validation subset.

[0101] More specifically, in the case of image-level prediction, such as an image classification task, “example” refers to an image in the dataset that has that class label; that is, each subtask consists of a set of images, all belonging to a selected class. Within each subtask, the image class labels are redefined for the Stage 1 training process. For example, five images might be selected for each class, for example, 15 images in total across three classes, and regardless of the original class label of each image, a random ordering of the classes is generated for Stage 1 training, so that one of the three classes can be set as Class No. 0, another as Class No. 1, and the last class as Class No. 2. In the next subtask, another 15 images could be selected from a different set of classes, and then their class labels are reset in a random order so that they are again 0, 1, and 2. In this way, the model becomes class-independent in the sense that it does not focus on learning information from each particular class, but rather learns how to improve its learning skills for all possible subtasks it encounters.

[0102] For prediction tasks (e.g., dense prediction), one example might include all annotated entities (regions or objects) of the same class in an image along with their labels, and when sampling several examples from a selected subset of classes, several images containing at least one labeled entity from the selected class are selected first, and then labeled entities that are not from the selected class are set as background classes. For example, in an image segmentation task, class labels may include foreground and background classes (i.e., regions of no interest for modeling purposes), and if the selected classes are tumors and blood vessels, all regions in the image that do not belong to these two classes are relabeled as background classes, all regions belonging to tumors are randomly relabeled as class No. 0 or No. 1, and vascular regions are relabeled as any remaining class index after being relabeled as tumor regions.

[0103] The learning methods for the subtask can be thought of as follows: Metric-based methods that learn representations of existing datasets and distill from these datasets the skill to compare similarities between examples of any class from an unseen target domain. In these methods, the representations learned in stage 1 can be applied to stage 2 in the following ways: (a) select a small number of images in the target domain; (b) annotate these images and apply a pre-defined model to each example to generate a feature vector representation (e.g., the output of the last layer in a convolutional network before the classification layer); (c) combine representations from examples of the same class to generate one representation per target class; (d) use these processed representations as prototypes for the target classes; (e) generate feature vector representations of the remaining unlabeled images or image regions (or other entities in the case of a high-density prediction task) of the target domain; (f) compare each feature vector representation from the unlabeled target domain to the prototype by calculating the distance between these vectors, e.g., cosine distance, and then assign the class label in the unlabeled image or image region as the prototype class with the minimum distance (e.g., the most similar). Other techniques for learning subtasks may be used in combination with metric-based methods. For example, adversarial generative models may be used, which synthesize images based on the distribution of existing images and use the generated images to increase the number of class-specific examples in the target image domain.

[0104] Alternatively, optimization-based methods for learning the best model weights for initializing a model can efficiently adapt to an unseen target dataset with only a small number of examples. In these methods, training in stage 1 involves two model optimization loops: (a) an internal learning loop in which the artificial intelligence model updates its model weights on one subtask over a predetermined or flexible number of epochs, generating a loss in its validation set after the model update, indicated as L-subtask-i, with respect to the i-th subtask; and (b) an external learning loop in which the objective is to find a set of model initializations that, when used to update all subtasks, produces the best model, each subtask accompanied by only a small number of annotated examples, and this is achieved by finding a model initialization that minimizes the sum of all losses (L-subtask-i) calculated from the validation sets of the subtasks on their validation sets with respect to those model initializations, where i ranges from 1 to the number of subtasks. Other techniques for learning subtasks may be used in combination with optimization-based methods. For example, in addition to finding the best model initializations, one could also find the best model architecture (e.g., performing a neural architecture search).

[0105] The workflow described herein for the design of initial model development was applied to cell detection in bright-field IHC assays and tissue type classification in H&E assays as follows. However, similar workflows can also be applied to other staining methods, such as special staining in bright-field assays (e.g., Trichrome Masson assays that simultaneously stain muscle, collagen fibers, erythrocytes, and cell nuclei) and fluorescence IHC assays.

[0106] (i) Model preconditions for cell detection in IHC assays: The objective is to identify the staining phenotype, cell type, and cell location in each image. For example, in the DAB-Ki67 IHC assay, a cell detection model can be designed to identify tumor cells stained positively with Ki67 (Ki67+ tumors), tumor cells stained negatively with Ki67 (Ki67- tumors), and all other cell types / staining types, along with the location of each cell nucleus center in a single pixel, or along with the bounding box of each cell nucleus (e.g., the rectangular pixel location circumscribing each cell nucleus), or along with the pixel of each cell nucleus (e.g., nuclear segmentation mask).

[0107] Generally, images from bright-field IHC assays are related and have a certain degree of similarity in their appearance; that is, hematoxylin staining is used in most of these assays, staining the cell nucleus and acting as a pointer to where the cells are located within the entire slide image, but one or more biomarkers are targeted by the IHC staining protocol, and accordingly, cells expressing these biomarkers show color upon application of the chromogen. "Staining pattern" refers to the appearance of positive image regions for a target biomarker with respect to (a) intracellular and / or intracellular localization of the biomarker, (b) stained cell type, (c) staining intensity, (d) incidence, and (e) spatial distribution of the positive region. For example, Ki67 has a nuclear staining pattern; for instance, positive staining signals in IHC images from a Ki67 assay are observed to be located in the cell nucleus, mainly in either scattered tumor cells or clustered tumor foci, and the positive staining signals range from low to very high staining intensity.

[0108] The preconditioning workflow may be designed using images from various IHC assays having various chromogens and biomarker staining patterns (see Table 1 for, e.g., assays). An exemplary workflow may include: (1) If one or more chromogens used in an existing annotated dataset are the same as those of one or more targeted IHC assays: (1.1) If annotations from only one assay are available, in each model training iteration, this dataset is divided into subtasks, one subtask is sampled for training, which has a small number of images with cell annotations from one or more classes, and it is confirmed that cells of all classes selected for this development stage are present in at least one of the images. For example, applicable to the preconditioning of the DAB-Ki67 assay and the DAB-PDL1 assay. Figure 6 shows an example of a DAB-Ki67 IHC image (A), in which the brown signal is the image region where the chromogen DAB produced color, indicating these cells in which Ki67 expressed in the cell nucleus was detected by this IHC assay, and the grayish-blue signal is hematoxylin-stained cell nucleus. An exemplary ground truth (B) is also shown for cell detection in the image of (A), where different colored dots overlaid on the cell nucleus center indicate the class label of each cell. (1.2) If annotations from multiple assays are available, in order to train an artificial intelligence system pre-conditioned to be as assay-independent as possible, in each model training iteration, an assay may first be selected from all assays, and then several examples may be sampled from this assay. For example, the artificial intelligence system may be pre-conditioned for the DAB-Ki67 and DAB-CK7 assays and may be applied to the DAB-PDL1 assay.

[0109] (2) If the chromogens used in the existing annotated dataset differ from those in the target IHC assay: a workflow similar to (1.1) or (1.2) can be performed. Furthermore, in order to match the chromogens in both step 1 and step 2, stain demixing can be performed to decompose the IHC image into each stain component, and these components can be remixed with the color vectors extracted from the target IHC assay. For example, an artificial intelligence system may be a preconditioner for the Tamra-PDL1 / Dabsyl-CK7 double-strand assay, or it may be applied to the DAB-Ki67 assay, and the training system or user may choose to decompose the Tamra-PDL1 / Dabsyl-CK7 double-strand image into PDL1, CK7, and hematoxylin-Gray scale intensity images, and remix them with the DAB color vectors to obtain two sets of images for the preconditioning step, namely a synthetic DAB-PDL1 single-strand image and a synthetic DAB-CK7 single-strand image. Figure 6 shows an exemplary Tamra-PDL1-Dabsyl-CK7 IHC image (C). Exemplary ground truth (D) is also shown in DAB-PR IHC images where different colored dots overlaid on the center of the cell nucleus indicate the class label of each cell.

[0110] The assays used in the prerequisite stage do not necessarily need to be highly similar to the target domain assays, but some level of similarity can be beneficial for accumulating both learned skills and learned knowledge in Stage 1. [Table 1]

[0111] (ii) Model preconditions for tissue type classification in H&E assays: In this case, the goal is to generate a model for identifying the tissue type of each image tile from all slide images. For example, a model can be built that can classify each image tile into tumor, stroma, normal tissue, and other types. Model preconditioning can be performed using existing datasets from other disease types outside the target domain (e.g., different tumor types), different disease stages, etc., using a similar workflow described herein. Each subtask can be sampled from the same dataset or from a mixture of different datasets, if available.

[0112] Model updates and adaptations To efficiently perform model updates and adaptation to new datasets after initial model development, we identified scenarios commonly encountered in digital pathology settings and developed techniques for selecting the most suitable adaptive learning method for each scenario. Tissue type classification of H&E images is used as an example to illustrate these techniques. However, it should be understood that the techniques described for model updates and adaptations can be applied to various other scenarios commonly encountered in digital pathology.

[0113] When a model is trained continuously, the data stream can differ in different ways. Common situations encountered in digital pathology were categorized into the following scenarios: 1. Data Incremental Scenario: Creating annotated datasets curated by pathologists is a time-consuming process, and it is preferable to train models in batches as data arrives and whenever data arrives. Incoming data, considered a new stream for training the model, should have minimal differences from previous data, typically originate from the same underlying distribution, and possess all classes as previously observed. 2. Domain Incremental: A continuous stream of data is considered a domain incremental scenario when it originates from different domains or distributions. This scenario is similar to the data incremental scenario in that all data streams have all the classes they represent. 3. Class Incremental: A class incremental scenario occurs when the model expands in terms of the number of classes. The clinical need for this scenario arises when a batch of annotated images contains different subsets of classes or when the model definition changes to classify the data into more tissue subtypes. It should be understood that each data stream may contain images from new invisible classes as well as images from visible classes introduced in previous data streams. 4. Task Incremental: Any of the above scenarios can be considered task incremental if each data stream is defined as a new task. This scenario uses a different architectural design, and the network has shared layers between tasks and between task-specific layers.

[0114] Each incremental data stream is called an experience. Each experience is then divided into training, validation, and test streams. The model is trained in the training stream, validated in the validation stream at the end of every epoch, and evaluated in the test stream at the end of each experience. Model performance was evaluated in the test stream from all experiences at the end of training all experiences to study forward and backward movement.

[0115] (i) Data incremental digital pathology model update: (1.1) Digital Pathology Scenario: After the initial development of an AI-based digital pathology algorithm, it may be necessary to incorporate more annotated data from the same domain (e.g., slight differences and nearly similar; from the same underlying data distribution) into the model. In the digital pathology setting, it is common to expect annotated data to arrive in batches, as professionally annotated and / or professionally validated annotations are time-consuming and may not be available during initial model development. In addition, due to the large number of parameters in AI models, especially deep neural networks, it is common to prevent model overfitting by learning from additional data. For example, the initial model may be trained to classify tumor tissue versus normal tissue from colon cancer images (shown in Figure 7 - the plot in the lower left corner shows the class labels and number of examples in each model version), and in subsequent model development / updates, similar examples of the same class may be incorporated into the model, such examples arriving in batches over time.

[0116] (1.2) Workflow for selecting the best adaptive learning method: (1.2a) To efficiently and effectively update the model without training from scratch using existing and newly arrived data, a benchmark dataset is set up from a given digital pathology domain, and the data assets are randomly divided into batches to simulate the case of sequentially arriving data. (1.2b) To evaluate how effective a candidate learning method is in the presence of small random variations in individual examples within each batch, an augmented dataset may be created to simulate such small variations, and the image order may be randomly shuffled so that each data batch contains images augmented in different ways. The various types and degrees of changes relative to the original data may be carefully balanced so as not to distort the distribution of the entire dataset. Data augmentation can be performed dynamically during training or in advance for each training iteration. Exemplary extensions include (i) altering the staining space by first performing stain decomposition (e.g., demixing), then remixing each stain with a given color vector to change the hue, saturation, and intensity of each stain; (ii) increasing image resolution by resizing the image and then returning it to its original size; (iii) similarly; or (iv) any combination thereof.

[0117] (1.3) Artificial intelligence-based methods for adaptive learning: A batch of benchmark data generated from (1.2b) can be run on candidate learning methods such as the following:

[0118] (1.3a) Normalization-based methods: These methods primarily focus on prioritizing privacy and reducing memory usage. Privacy is maintained by avoiding the storage of raw inputs. Of the different regularization-based methods, two common methods are Elastic Weight Reinforcement (EWC) and Learning Without Forgetting (LWF). EWC is a prior-focused method in which model parameters are used as priors when learning from new data, and the method estimates the distribution over the model parameters. LWF, on the other hand, is a data-focused method. The main design of data-focused methods is based on knowledge distillation from previous models to the current model targeted with new data. This concept is also introduced in LWF, which deals with forgetting in the knowledge transfer mechanism. These methods prevent the model from forgetting the knowledge it has learned by applying constraints to the current model weights, and as a result, when updating the current model, the model weights do not deviate significantly from their current version.

[0119] (1.3b) Replay-based methods: These methods retain the most useful examples or their feature representations (e.g., feature vectors extracted from the hidden layers of a neural network) from existing data batches and revisit them during training on newly arrived data. To overcome forgetting, previous samples of the task are replayed in the process of learning a new task. Among the various methods belonging to this category are Incremental Learner and Representation Learning (iCaRL), Continuous Prototype Evaluation (CoPE), and A-GEM. The iCaRL method memorizes a subset of the best examples for each class selected according to an approximate class means in the learned feature space. The upper limit of this method is determined by joint training of past and present tasks. CoPE is an online data incremental learner with prototypes that persistently represent the most prominent features of a class population. The rapidly evolving prototypes allow for learning and prediction at any point in time. CoPE is robust to class imbalances using replay and balanced memory population schemes. GEM is designed based on a task incremental setting. This method focuses only on the new task by limiting its updates and therefore does not interfere with previous tasks. This is achieved by projecting the computed gradient direction onto a feasible region outlined by the previous task gradient using a first-order Taylor series approximation. The A-GEM method is an improved version of the GEM method. A-GEM helps mitigate the problem of projecting in one direction estimated by a randomly selected sample from the previous task data buffer. A-GEM yields similar performance accuracy to GEM with similar computational and memory efficiency to regularization methods.

[0120] (1,3c) A method combining regularization and regeneration

[0121] (1.3d) A method that leverages meta-learning principles to enable model updates in several examples within each batch (see the "CoPE" method implemented in the Experiments section).

[0122] (1.3e) Parameter isolation methods: These methods assign different model parameters to each task to address forgetting. Since there are no constraints on the size of the architecture, there is no fixed architecture in these methods. Thus, new branches can grow for new tasks. This can be achieved by freezing the parameters of previous tasks or by making model copies dedicated to each task. These types of architectures are called dynamic architectures.

[0123] (1.3f) Other techniques applied in conjunction with the methods described above. For example, performing self-supervised learning on new, unlabeled data before applying the methods described above. Another example is running a generative adversarial model to generate similar examples from previous batches of data.

[0124] (ii) Updating domain incremental digital pathology models: (2.1) Digital Pathology Scenarios: After the initial development of artificial intelligence-based digital pathology algorithms, it may be necessary to incorporate annotated data from different domains (e.g., from similar but different underlying data distributions) into the model. In digital pathology settings, there is a practical need to generate models that are robust to (a) changes or differences in experimental settings, such as (i) staining reagents, (ii) staining protocols or equipment, (iii) scanner suppliers, and (iv) sample sources (e.g., different clinical sites or tissue banks), and (b) changes in other aspects, such as histological type, organ type, patient population, disease stage, and disease subtype (e.g., tumor type and subtype).

[0125] (2.2) Workflow for selecting the best adaptive learning method: (2.2a) To efficiently and effectively update the model without training from scratch using both existing and newly arrived data, a benchmark dataset is set up from the DP domain and synthetic images simulating realistic changes are generated as data from different sources / domains. (2.2b) To evaluate which candidate adaptive learning method is most effective in this scenario, augmented datasets may be created to simulate multiple types of domain shifts and how well the candidate methods perform with such changes may be evaluated. Data augmentation can be performed dynamically during training or in advance at each training iteration. Exemplary augmentation techniques include (i) changing the staining space by first performing stain destaining (e.g., demixing) and then remixing each stain with a given color vector to change the hue, saturation, and intensity of each stain; (ii) increasing image resolution by resizing the image and then returning it to its original size; (iii) etc.; or (iv) any combination thereof. One or more types of changes can be applied to each subset to simulate a series of consecutive domain shifts. For example, in the case of H&E images with the same tissue type for a classification task, the dataset may be randomly split to generate the following expanded subsets: (i) one stain becomes stronger due to a change in protocol; (ii) both stains fade (i.e., decrease in staining intensity) due to aging of the prepared slides; (iii) one stain becomes more saturated in the HSV color space than the other due to a change in stainer; and (iv) both stains change in hue in the HSV color space due to a change in scanner. Figure 8 illustrates the changes described in (i) and (ii) above (the plot in the lower left corner shows the class labels and number of examples in each model version).

[0126] (2.3) Artificial intelligence-based methods for adaptive learning: Batches of benchmark data generated from (2.2b) may be run on candidate learning methods such as: (2.3a) Normalization-based methods: These methods prevent the model from forgetting learned knowledge by applying constraints to the current model weights, so that when the current model is updated, the model weights do not deviate significantly from its current version. (2.3b) Replay-based methods: These methods preserve the most useful examples or their feature representations (e.g., feature vectors extracted from the hidden layers of a neural network) from existing data batches and revisit them during training of newly arrived data. (2.3c) Methods combining regularization and replay. (2.3d) Methods that leverage meta-learning principles to enable model updates on several examples from each batch (see the "CoPE" method implemented in the Experiments section). (2.3e) Parameter isolation methods. (2.3f) Other techniques applied in conjunction with the aforementioned methods. For example, performing self-supervised learning on new, unlabeled data before applying the aforementioned methods. Another example is running a generative adversarial model to generate similar examples from previous data batches.

[0127] (iii) Class incremental digital pathology model update: (3.1) Digital Pathology Scenarios: After the initial development of an artificial intelligence-based digital pathology algorithm, it may be necessary to expand the model in terms of the number of classes. In digital pathology settings, this scenario is commonly encountered for the following reasons: (i) changes in end-user needs, (ii) annotated data being delivered in batches, with each batch having a set of classes that is partially or completely different from the existing data classes, and / or (iii) model design choices, for example, a model initially designed to classify tumor versus normal may find it necessary to identify necrotic and lymphocyte clusters to ensure good model performance.

[0128] (3.2) Workflow for selecting the best adaptive learning method: (3.2a) To efficiently and effectively update the model without training both existing and new data from scratch, a benchmark dataset may be set up from the digital pathology domain, and the classes may be split into several different subsets, so that the corresponding examples are split according to their class labels. For example, Figure 9 shows such class splitting for each model version (the plot in the lower left shows the class labels and the number of examples in each model version). (3.2b) Next, candidate adaptive learning methods are evaluated to determine which adaptive learning method is most effective for learning the classes incrementally, and their performance in learning different sets of class orders for each data batch may be compared. (3.3) Artificial intelligence-based methods for adaptive learning: Batches of benchmark data generated from (3.2b) may be run against candidate learning methods such as: (3.3a) Normalization-based methods: These methods prevent the model from forgetting learned knowledge by applying constraints to the current model weights, so that when the current model is updated, the model weights do not deviate significantly from its current version. (3.3b) Replay-based methods: These methods preserve the most useful examples or their feature representations (e.g., feature vectors extracted from the hidden layers of a neural network) from existing data batches and revisit them during training of newly arrived data. (3.3c) Methods combining regularization and replay. (3.3d) Methods that leverage meta-learning principles to enable model updates on several examples from each batch (see the "CoPE" method implemented in the Experiments section). (3.3e) Parameter isolation methods. (3.3e) Other techniques applied in conjunction with the aforementioned methods. For example, performing self-supervised learning on new, unlabeled data before applying the aforementioned methods. Another example is running a generative adversarial model to generate similar examples from previous data batches.

[0129] (iv) Updating task-incremental digital pathology models: (4.1) In each of the above scenarios, a decision may be made as to whether to formulate each different data batch as a new task, and therefore whether to apply a task-incremental method. Such a method may specify separate model components for each task (e.g., separate sets of neurons or separate layers in a neural network model), and in each model iteration, only the model-specific components are trained, while the rest of the previously trained components remain unchanged. Thus, the model can adapt to changes in data with separate components and avoid forgetting the knowledge learned from previous tasks.

[0130] Applying Adaptive Learning Frameworks to Other Modalities The adaptive learning framework may be applied to other imaging modalities and other research fields. The adaptive learning framework is domain-independent in the following ways: (1) The preconditioning strategy for initial model training can be applied to other types of data that require numerous annotations to generate the initial model. (2) The three adaptive learning scenarios for model updating / adaptation have commonalities with scenarios encountered in other computational biomedical research, and therefore the adaptive learning method selection strategy can be utilized by these studies.

[0131] The adaptive learning framework may be applied to federated learning. Federated learning aims to update a global model without sharing data from individual data sources and without explicitly sharing local models. The adaptive learning framework can be leveraged by federated learning in the following ways: (1) pre-conditioning local and / or global models to enable more effective and efficient model updates with fewer annotations within the target image domain; and (2) updating local and / or global models through one or more adaptive learning methods to continuously update the model without retraining with previous data and to select the best learning method by applying the model selection workflow described herein.

[0132] The adaptive learning framework may be applied to multi-model learning. Multimodal learning aims to integrate knowledge learned from different modalities of data. The adaptive learning framework can be leveraged by multi-model learning in the following ways: (1) pre-conditioning models from one or more data modalities to enable more effective and efficient model updates with fewer annotations within the target image domain; (2) updating from one or more data modes via one or more adaptive learning methods to continuously update the model without retraining previous data and select the best learning method by applying the model selection workflow described herein; and (3) generating a model for adaptively integrating representations from data of multiple modalities by applying the adaptive learning framework for the first iteration of learning and / or subsequent updates / adaptations.

[0133] V. Exemplary Systems for Adaptive Learning Figure 10 shows a block diagram illustrating a computing environment 1000 for processing digital pathology images using an artificial intelligence system (e.g., one or more machine learning models). As further described herein, processing digital pathology images may include training a machine learning algorithm using digital pathology images, and / or transforming some or all of the digital pathology images into one or more results using a trained (or partially trained) version of the machine learning algorithm (i.e., a machine learning model).

[0134] As shown in Figure 10, the computing environment 1000 includes several stages, namely an image storage stage 1005, a preprocessing stage 1010, a labeling stage 1015, a data augmentation stage 1017, a training stage 1020, and a result generation stage 1025.

[0135] The image storage step 1005 includes one or more image data stores 1030 (e.g., storage devices 430 described in relation to Figure 4) that are accessed (e.g., by the pre-processing step 1010) to provide a set of digital images 1035 of a pre-selected area from a biological sample slide (e.g., a tissue slide) or of the entire biological sample slide. Each digital image 1035 stored in each image data store 1030 and accessed in the image storage step 1010 may include a digital pathology image generated according to some or all of the processes described with respect to the network 400 shown in Figure 4. In some embodiments, each digital image 1035 includes image data from one or more scanned slides. Each digital image 1035 may correspond to image data from a single specimen and / or image data from a single day on which the underlying image data corresponding to the image was collected.

[0136] Image data may include the image, as well as any information relating to the color channel or color wavelength channel, and details relating to the imaging platform on which the image was generated. For example, tissue sections may need to be stained by applying a staining assay that includes one or more different biomarkers associated with a chromogenic stain for bright-field imaging or a phosphor for fluorescence imaging. The staining assay can use a chromogenic stain for bright-field imaging, an organic phosphor for fluorescence imaging, a quantum dot, or a combination of an organic phosphor and a quantum dot, or any other combination of stain, biomarker, and observation or imaging device. Examples of biomarkers include estrogen receptor (ER), human epidermal growth factor receptor 2 (HER2), human Ki-67 protein, progesterone receptor (PR), and programmed cell death protein 1 (PD1), where the tissue sections are labeled detectably with their respective binders (e.g., antibodies). In some embodiments, digital image and data analysis operations such as classification, scoring, Cox modeling, and risk stratification depend on the type of biomarker used and the selection and annotation of the field of view (FOV). Furthermore, typical tissue sections are processed on an automated staining / assay platform that applies a staining assay to the tissue sections, thereby obtaining stained samples. A variety of commercially available products suitable for use as staining / assay platforms exist on the market, one example being the VENTANA® SYMPHONY® product from the acquirer, Ventana Medical Systems, Inc. The stained tissue sections may be fed into an imaging system of a microscope or whole-slide scanner having, for example, a microscope and / or imaging components, one example being the VENTANA® iScan Coreo® / VENTANA® DP200 product from the acquirer, Ventana Medical Systems, Inc. Multiple tissue slides may be scanned on an equivalent multiple-slide scanner system.Additional information provided by the imaging system may include any information regarding the staining platform, such as the concentration of chemicals used for staining, the reaction time of chemicals applied to the tissue during staining, and / or pre-analysis conditions of the tissue, including the age of the tissue, fixation method, duration, section embedding method, and cutting method.

[0137] In the preprocessing step 1010, each of one, more, or all of the set of digital images 1035 is preprocessed using one or more techniques to generate a corresponding preprocessed image 1040. Preprocessing may include cropping the image. In some examples, preprocessing may further include standardization or rescaling (e.g., normalization) to make all features the same scale (e.g., the same size scale or the same color scale or saturation scale). In certain cases, the image is resized to have a minimum size (width or height) of a predetermined number of pixels (e.g., 2500 pixels) or a maximum size (width or height) of a predetermined number of pixels (e.g., 3000 pixels), and optionally maintains the original aspect ratio. Preprocessing may further include noise reduction. For example, the image may be smoothed by applying a Gaussian function or Gaussian blur to remove unwanted noise.

[0138] The preprocessed images 1040 may include one or more training images, validation input images, and unlabeled images. It should be understood that the preprocessed images 1040 corresponding to the training group, validation group, and unlabeled group do not need to be accessed simultaneously. For example, the initial set of training and validation preprocessed images 1040 may be accessed and used first to train the machine learning algorithm 1055, and then the unlabeled input images may be accessed or received (e.g., once or multiple times thereafter) and used by the trained machine learning model 1060 to provide a desired output (e.g., cell classification).

[0139] In some examples, the machine learning algorithm 1055 is trained using supervised training, and some or all of the preprocessed images 1040 are partially or fully labeled manually, semi-automatically, or automatically in the labeling stage 1015 with labels 1045 that identify the "correct" interpretation (i.e., "ground truth") of various biological substances and structures within the preprocessed images 1040. For example, labels 1045 may identify features of interest (e.g.), cell classification, a binary representation of whether a given cell is of a particular type, a binary representation of whether the preprocessed image 1040 (or a particular region having the preprocessed image 1040) contains a particular type of representation (e.g., necrosis or artifact), a categorical feature of slide-level or region-specific representation (e.g., identifying a particular type of cell), a number (e.g., identifying the amount of a particular type of cell in a region, the amount of represented artifact, or the amount of necrotic area), the presence or absence of one or more biomarkers, etc. In some cases, labels 1045 include location. For example, label 1045 may identify the nuclear dot location of a particular type of cell, or the dot location of a particular type of cell (e.g., raw dot label). As another example, label 1045 may include borders or boundaries of depicted tumors, blood vessels, necrotic areas, etc. As yet another example, label 1045 may include one or more biomarkers identified based on biomarker patterns observed using one or more stains. For example, a tissue slide stained for a biomarker, e.g., programmed cell death protein 1 ("PD1"), may be observed and / or processed to label cells as either positive or negative cells, taking into account the expression level and pattern of PD1 in the tissue. Depending on the features of interest, a given labeled preprocessed image 1040 may be associated with a single label 1045 or multiple labels 1045. In the latter case, each label 1045 may be associated with (e.g.) instructions regarding the location or portion in the preprocessed image 1045 to which the label corresponds.

[0140] The labels 1045 assigned in the labeling stage 1015 may be identified based on input from a human user (e.g., a pathologist or image scientist) and / or an algorithm configured to define the labels 1045 (e.g., an annotation tool). In some examples, the labeling stage 1015 may include transmitting and / or presenting some or all of one or more preprocessed images 1040 to a computing device operated by the user. In some examples, the labeling stage 1015 may include utilizing an interface presented by the labeling controller 1050 (e.g., using an API) on the computing device operated by the user, the interface including input components for accepting input that identifies the labels 1045 for features of interest. For example, a user interface that allows selection of an image or region of an image (e.g., FOV) for labeling may be presented by the labeling controller 1050. A user operating the terminal may select an image or FOV using the user interface. Several image or FOV selection mechanisms may be provided, such as specifying known or irregular shapes or defining anatomical regions of interest (e.g., tumor regions). In one example, the image or FOV is the entire tumor region selected on an IHC slide stained with a combination of H&E staining. The selection of the image or FOV may be performed by the user or by an automated image analysis algorithm such as tumor region segmentation on an H&E tissue slide. For example, the user may choose that the image or FOV be the entire slide or the entire tumor, or that the entire slide or the entire tumor region be automatically designated as the image or FOV using a segmentation algorithm. The user operating the terminal may then select one or more labels 1045 to be applied to the selected image or FOV, such as point locations on cells, positive markers for biomarkers expressed by cells, negative biomarkers for biomarkers not expressed by cells, and boundaries around cells.

[0141] In some examples, the interface may identify which particular label 1045 is requested and / or to what extent, which may be communicated to the user via (e.g.) text instructions and / or visualizations. For example, a specific color, size, and / or symbol may indicate that a label 1045 is requested for a particular representation in an image (e.g., a specific cell or region or staining pattern) in relation to other representations. If labels 1045 are requested for multiple representations, the interface may identify each representation simultaneously, or identify each representation sequentially (such that providing a label for one identified representation triggers the identification of the next representation for labeling). In some examples, each image is presented until the user identifies a certain number (e.g., a specific type) of labels 1045. For example, a given entire slide image or a given patch of an entire slide image may be presented until the user identifies the presence or absence of three different biomarkers, at which point the interface may present images of different entire slide images or different patches (e.g., until a threshold number of images or patches are labeled). Therefore, in some examples, the interface is configured to request and / or accept labels 1045 for an incomplete subset of the features of interest, and the user can decide which of potentially many representations are labeled.

[0142] In some examples, the labeling stage 1015 includes a labeling controller 1050 that implements an annotation algorithm to semi-automatically or automatically label various features of an image or region of interest within an image. The labeling controller 1050 annotates the image or FOV on a first slide according to user input or an annotation algorithm and maps the annotations across the rest of the slide. Depending on the defined FOV, several methods for annotation and alignment are possible. For example, an entire tumor region annotated on an H&E slide from a series of consecutive slides may be selected automatically or by the user on an interface such as VIRTUOSO / VERSO(trademark). Since the other tissue slides correspond to consecutive sections from the same tissue block, the labeling controller 1050 performs inter-marker alignment operations to map the entire tumor annotation from the H&E slide and transfer it to each of the remaining IHC slides in the series. Exemplary methods for aligning markers are described in more detail in International Publication No. 2014140070, “Whole slide image registration and cross-image annotation devices, systems and methods,” filed March 12, 2014, by the same applicant, which is incorporated herein by reference in its entirety for all purposes. In some embodiments, any other methods for image registration and generation of whole-tumor annotations may be used. For example, a qualified radiologist, such as a pathologist, may annotate whole-tumor regions on any other IHC slide and run the labeling controller 1050 to map the whole-tumor annotations on the other digitized slide. For example, a pathologist (or an automated detection algorithm) may annotate whole-tumor regions on an H&E slide to trigger an analysis of all adjacent serially sectioned IHC slides to determine a whole-slide tumor score for the annotated regions on all slides.

[0143] In augmentation stage 1017, the training set of labeled or unlabeled images (original images) from preprocessed images 1040 is augmented with synthetic images 1052 generated using augmentation controls 1054 that execute one or more augmentation algorithms. Augmentation techniques are used to artificially increase the quantity and / or type of training data by adding slightly modified synthetic copies of existing training data, or newly created synthetic data from existing training data. As described herein, differences between scanners and laboratories can cause variability in intensity and color within digital images. Furthermore, poor scanning can result in gradient variations and blurring effects, assay staining can result in staining artifacts such as background washes, and different tissue / patient samples can lead to variability in cell size. These variability and disturbances can negatively impact the quality and reliability of deep learning and artificial intelligence systems. The augmentation techniques performed in augmentation stage 1017 can act as regularizers for these variability and disturbances, helping to reduce overfitting when training machine learning models. Examples of extension techniques include (i) altering within a staining space, where first a stain decomposition (e.g., demixing) is performed, and then each stain is remixed with a predetermined color vector to change the hue, saturation, and intensity of each stain; (ii) increasing image resolution by resizing an image and then returning it to its original size; (iii) the same; or (iv) any combination thereof.

[0144] In training stage 1020, the labels 1045 and the corresponding preprocessed images 1040 can be used by the training controller 1065 to train the machine learning algorithm 1055 according to the various workflows described herein. For example, to train the algorithm 1055, the preprocessed images 1040 may be split into a subset of training images 1040a (e.g., 90%) and a subset of validation images 1040b (e.g., 10%). The splitting may be performed randomly (e.g., 90 / 10% or 70 / 30%), or according to more complex validation techniques such as K-fold cross-validation, skip-a-image cross-validation, skip-group cross-validation, or nested cross-validation to minimize sampling bias and overfitting. The splitting may also be performed on the basis of including augmented or composite images 1052 in the preprocessed images 1040. For example, it may be beneficial to limit the number or proportion of composite images 1052 included in the subset of training images 1040a. In some examples, the ratio of the original image 1035 to the composite image 1052 is maintained at 1:1, 1:2, 2:1, 1:3, 3:1, 1:4, or 4:1.

[0145] In some examples, the machine learning algorithm 1055 includes a CNN, a modified CNN with a coding layer replaced by a residual neural network ("ResNet"), or a modified CNN with coding and decoding layers replaced by ResNet. In other examples, the machine learning algorithm 1055 may be any suitable machine learning algorithm configured to localize, classify, and / or analyze a preprocessed image 1040, or a combination of one or more such techniques, such as a visual transformer, CNN-HMM, or MCNN (multiscale convolutional neural network), such as a two-dimensional CNN ("2DCNN"), Mask R-CNN, U-Net, feature pyramid network (FPN), dynamic time stretching ("DTW") technique, hidden Markov model ("HMM"), or pure attention-based model. The computing environment 1000 may employ the same type of machine learning algorithm, or different types of machine learning algorithms trained to detect and classify different cells. For example, the computing environment 1000 may include a first machine learning algorithm (e.g., U-Net) for detecting and classifying PD1. Computing environment 500 may also include a second machine learning algorithm (e.g., 2DCNN) for detecting and classifying differentiation clusters 68 ("CD68"). Computing environment 1000 may also include a third machine learning algorithm (e.g., U-Net) for detecting and classifying PD1 and CD68 in combination. Computing environment 1000 may also include a fourth machine learning algorithm (e.g., HMM) for diagnosing diseases for treatment or prognosis of a subject, such as a patient. Further other types of machine learning algorithms may be implemented in other examples provided herein.

[0146] The training process for machine learning algorithm 1055 includes selecting hyperparameters for machine learning algorithm 1055 from parameter datastore 1063, inputting a subset of images 1040a (e.g., labels 1045 and corresponding preprocessed images 1040) into machine learning algorithm 1055, and performing iterative operations to learn a set of parameters for machine learning algorithm 1055 (e.g., one or more coefficients and / or weights). Hyperparameters are settings that can be tuned or optimized to control the behavior of machine learning algorithm 1055. Most algorithms explicitly define hyperparameters that control different aspects of the algorithm, such as memory or execution cost. However, additional hyperparameters may be defined to fit the algorithm to a particular scenario. For example, hyperparameters may include the number of hidden units of the algorithm, the learning rate of the algorithm (e.g., 1e-4), the convolution kernel width, or the number of kernels of the algorithm. In some examples, the number of model parameters decreases with each convolutional and deconvolutional layer, and / or the number of kernels decreases by half with each convolutional and deconvolutional layer compared to a typical CNN.

[0147] A subset of images 1040a can be input to the machine learning algorithm 1055 as a batch of a predetermined size. The batch size limits the number of images presented to the machine learning algorithm 1055 before parameter updates can be performed. Alternatively, the subset of images 1040a can be input to the machine learning algorithm 1055 as a time series or sequentially. In either case, if an augmented or composite image 1052 is included in the preprocessed image 1040a, the number of original images 1035 versus the number of composite images 1052 included in each batch, or the way in which the original images 1035 and phenotypic images 1052 are supplied to the algorithm (e.g., every other batch or image is the original batch or original image of the images) can be defined as hyperparameters.

[0148] Each parameter is a variable that can be adjusted so that its value is adjusted during training. For example, the cost function or objective function may be configured to optimize the accurate classification of the displayed representations, to optimize the characterization of features of a given type (e.g., characterization of shape, size, uniformity, etc.), to optimize the detection of features of a given type, and / or to optimize the accurate localization of features of a given type. Each iteration may include learning a set of parameters for the machine learning algorithm 1055 that minimizes or maximizes the cost function of the machine learning algorithm 1055, so that the value of the cost function using a set of parameters is less than or greater than the value of the cost function using a different set of parameters in the previous iteration. The cost function may be configured to measure the difference between the output predicted using the machine learning algorithm 1055 and the labels 1045 contained in the training data. For example, in the case of a supervised learning-based model, the goal of training is to learn a function "h()" (sometimes called a hypothesis function) that maps the training input space X to a target value space Y, h:X→Y, such that h(x) is a good predictor of the corresponding value in y. Various different techniques may be used to learn this hypothetical function. In some techniques, a cost function or loss function may be defined as part of deriving the hypothetical function, which measures the difference between the ground truth value for a given input and the predicted value for that input. As part of the training, techniques such as backpropagation, random feedback, direct feedback alignment (DFA), indirect feedback alignment (IFA), and Hebbs learning are used to minimize this cost or loss function.

[0149] Training iterations continue until a termination condition is met. Training completion conditions may be configured to be met when (for example) a predetermined number of training iterations are completed, when statistics generated based on testing or validation exceed a predetermined threshold (e.g., a classification accuracy threshold), when statistics generated based on confidence metrics (e.g., the mean or median of a confidence metric or a percentage of a confidence metric above a certain value) exceed a predetermined confidence threshold, and / or when a user device that was involved in the training review closes the training application executed by the training controller 1065. Once a set of model parameters is identified through training, the machine learning algorithm 1055 is trained, and the training controller 1065 performs an additional testing or validation process using a subset of images 1040b (test or validation dataset). The validation process may include iterative operations of inputting images from the subset of images 1040b into the machine learning algorithm 1055 using validation techniques such as K-fold cross-validation, miss-a-image cross-validation, skip-group cross-validation, and nested cross-validation to tune the hyperparameters and ultimately find the optimal set of hyperparameters. Once the optimal set of hyperparameters is obtained, a reserved test set of images from the subset of images 1040b is input to the machine learning algorithm 1055 to obtain an output, which is then evaluated against ground truth by calculating performance metrics such as error, accuracy, precision, recall, and receiver operating characteristic curve (ROC) using correlation techniques such as the Bland-Altman method and Spearman's rank correlation coefficient. In some cases, a new training iteration may be initiated in response to the reception of a corresponding request or trigger condition from the user device (e.g., initial model development, model update / adaptation, continuous learning, drift determined within the trained machine learning model 1060).

[0150] As can be understood, other training / validation mechanisms may be intended and implemented within the computing environment 1000. For example, the machine learning algorithm 1055 may be trained on images from a subset of images 1040a, and its hyperparameters may be tuned, while images from a subset of images 1040b may be used solely to test and evaluate the performance of the machine learning algorithm 1055. Furthermore, the training mechanisms described herein focus on training new machine learning algorithms 1055. These training mechanisms can also be used for developing initial models, updating / adapting models, and continuous learning of existing machine learning models 1060 trained on other datasets, as will be described in detail herein. For example, in some cases, the machine learning model 1060 may be pre-conditioned using images of other objects or biological structures, or from sections from other subjects or studies (e.g., human trials or mouse experiments). In those cases, the machine learning model 1060 can be used for developing initial models, updating / adapting models, and continuous learning using pre-processed images 1040.

[0151] Next, (in the result generation stage 1025) the trained machine learning model 1060 can be used to process the new preprocessed image 1040 to generate predictions or inferences, such as predicting cell center and / or location probabilities, classifying cell types, generating cell masks (e.g., pixel-by-pixel segmentation masks for the image), predicting the diagnosis or prognosis of a target disease such as a patient, or a combination thereof. In some examples, the mask identifies the location of displayed cells associated with one or more biomarkers. For example, given tissue stained for a single biomarker, the trained machine learning model 1060 may be configured to (i) infer the center and / or location of cells, (ii) classify cells based on the characteristics of the staining pattern associated with the biomarker, and (iii) output cell detection masks for positive cells and cell detection masks for negative cells. As another example, given tissue stained for two biomarkers, the trained machine learning model 1060 may be configured to (i) infer the center and / or location of cells, (ii) classify cells based on the characteristics of the staining patterns associated with the two biomarkers, and (iii) output cell detection masks for cells positive for the first biomarker, cell detection masks for cells negative for the first biomarker, cell detection masks for cells positive for the second biomarker, and cell detection masks for cells negative for the second biomarker. As yet another example, given tissue stained for a single biomarker, the trained machine learning model 1060 may be configured to (i) infer the center and / or location of cells, (ii) classify cells based on the characteristics of the cells and the staining patterns associated with the biomarker, and (iii) output cell detection masks for positive cells and cell detection masks for negative cell coding, as well as mask cells classified as tissue cells.

[0152] In some embodiments, the analysis controller 1080 generates an analysis result 1085 for use by the entity that requested the processing of the underlying image. The analysis result 1085 may include a mask output from a trained machine learning model 1060 overlaid on a new preprocessed image 1040. Additionally or alternatively, the analysis result 1085 may include information calculated or determined from the output of the trained machine learning model, such as a full-slide tumor score. In exemplary embodiments, the automated analysis of tissue slides uses the FDA-approved 510(k) approved algorithm of the assignee, VENTANA. Alternatively or additionally, any other automated algorithm may be used to analyze selected areas of the image (e.g., masked images) to generate scores. In some embodiments, the analysis controller 1080 may further respond to instructions received from a computing device, such as those from a pathologist, physician, investigator (e.g., associated with a clinical trial), subject, or medical professional. In some embodiments, communication from a computing device includes an identifier for each of a particular set of subjects and corresponds to a request to perform an iterative analysis for each subject represented in that set. The computing device can perform further analysis based on the machine learning model and / or the output of the analysis controller 1080, and / or provide recommended diagnoses / actions.

[0153] Computing environment 1000 is illustrative, and it will be understood that computing environment 1000 may have different stages and / or use different components. For example, in some examples, the network may omit the preprocessing stage 1010, thereby making the images used to train the algorithm and / or the images processed by the model raw images (e.g., from an image data store). In another example, it will be understood that each of the preprocessing stage 1010 and the training stage 1020 may include a controller for performing one or more operations described herein. Similarly, the labeling stage 1015 is shown in relation to the labeling controller 1050, and the results generation stage 1025 is shown in relation to the analysis controller 1080, but the controllers associated with each stage may further or alternatively facilitate other operations described herein other than the generation of labels and / or the generation of analysis results. As yet another example, the representation of the computing environment 1000 shown in Figure 10 lacks a displayed representation of devices associated with a programmer (e.g., selecting an architecture for a machine learning algorithm 1055 that defines how various interfaces function), devices associated with a user providing an initial label or label review (e.g., in the labeling stage 1015), and devices associated with a user requesting model processing of a given image (which may be the same user or a different user as the one providing the initial label or label review). Despite the absence of these device representations, the computing environment 1000 may include the use of one, more, or all of these devices, and in fact may include the use of multiple devices associated with multiple corresponding users providing initial labels or label reviews, and / or multiple devices associated with multiple corresponding users requesting model processing of various images.

[0154] VI. Techniques for training machine learning algorithms using adaptive learning frameworks Figure 11 is a flowchart of process 1100 for using a training set of images to train a machine learning algorithm according to various embodiments. Process 1100 shown in Figure 11 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system, hardware, or combination thereof. The software may be stored in a non-temporary storage medium (e.g., a memory device). Process 1100 presented in Figure 11 and described below is intended to be illustrative and non-limiting. Figure 11 shows, but is not limited to, various processing steps performed in a particular sequence or order. In certain alternative embodiments, the steps may be performed in several different orders, or several steps may be performed in parallel. In certain embodiments, such as the embodiments shown in Figures 4 and 10, the processing shown in Figure 11 may be performed as part of a training phase (e.g., algorithm training 1020) in which a machine learning algorithm is trained using a training set of images to produce a machine learning model configured to detect, characterize, classify, or perform a combination thereof, some or all of a region or object in an image.

[0155] Process 1100 begins in block 1105, in which a first annotated training set of images is acquired to train a machine learning algorithm to detect, characterize, classify, or combine some or all of the regions or objects within the images. The first annotated training set of images is located in a first image domain (e.g., preprocessed image 1040 of the computing environment 1000 described with respect to Figure 10). In some cases, the first annotated training set of images is a digital pathology image containing one or more types of cells. The first annotated training set of images may depict cells having staining patterns associated with biomarkers. In some cases, the first annotated training set of images may show cells having multiple staining patterns associated with multiple biomarkers. The first annotated training set of images may be annotated with training labels (e.g., supervised, semi-supervised, or weakly supervised).

[0156] In block 1110, the first annotated training set of images is divided into minisets of images, each miniset representing a separate modeling subtask and containing a limited number of examples. In some cases, the division includes, when only one miniset of images is available, selecting a subset of classes for each separate modeling subtask that is part of or matches the number of classes targeted in the second stage, and selecting a limited number of examples based on the selected subset of classes; and, when multiple minisets of images are available, for each separate modeling subtask, (i) mixing examples from multiple minisets of images, selecting a subset of classes that is part of or matches the number of classes targeted in the second stage, and selecting a limited number of examples from the mixed examples based on the selected subset of classes; or (ii) selecting one miniset of images from multiple minisets of images, selecting a subset of classes that is part of or matches the number of classes targeted in the second stage, and selecting a limited number of examples from the selected miniset of images based on the selected subset of classes.

[0157] In block 1115, the machine learning algorithm generates a pre-defined machine learning model that is trained in a first stage using a miniset of images and configured to detect, characterize, classify, or perform a combination thereof for some or all of the regions or objects in a new image. In some cases, the first stage further includes an internal learning loop in which the machine learning algorithm updates the model weights or parameters on one subtask in a predetermined or flexible number of epochs to initialize a pre-defined machine learning model for adaptation to a target dataset, which is represented as L-subtask-i with respect to the i-th subtask, and an external learning loop whose purpose is to find a set of model initializations that generate a pre-defined machine learning model when each is used to update all subtasks, each with only a limited number of examples, by finding a model initialization that minimizes the sum of all losses, where i ranges from 1 to the number of subtasks computed from the validation set of images of the subtask with respect to the model initialization.

[0158] In some cases, the first stage of training involves performing iterative operations to learn a set of parameters to detect, characterize, classify, or combine some or all of the regions or objects within a miniset of images, each iteration involving finding a set of parameters for a machine learning algorithm such that the value of the cost function using the set of parameters is greater than or less than the value of the cost function using a different set of parameters in the previous iteration, the cost function being constructed to measure the difference between the predictions made for some or all of the regions or objects using the machine learning algorithm and the ground truth labels given to the miniset of images.

[0159] In block 1120, a limited number of images from the target dataset are labeled to generate a second annotated training set of images for training a machine learning algorithm to detect, characterize, classify, or do a combination of these for some or all regions or objects within an image. The second annotated training set of images resides in a second image domain (different from the first image domain). In a particular example, the limited number of images is less than 50, 30, or 20.

[0160] In block 1125, the pre-conditioned machine learning model is trained in a second stage using a second annotated training set of images to produce a target machine learning model configured to detect, characterize, classify, or perform combinations thereof for some or all of the regions or objects in the new images. The number of classes targeted in the first stage is some of the number of classes targeted in the second stage, or matches the number of such classes. In some cases, the second stage further includes applying the pre-conditioned machine learning model to generate a feature vector representation for each example in the second annotated training set of images; combining the feature vector representations from examples of the same class to generate one representation for each target class and use one representation for each target class as a prototype in the target class; generating feature vector representations for the remaining unlabeled images or image regions from the target dataset; and comparing each feature vector representation from the unlabeled images to the prototype in the target class based on the distance between the feature vector representations from the unlabeled images and the prototypes for the target classes.

[0161] In some embodiments, the second stage of training involves performing iterative operations to learn a set of parameters to detect, characterize, classify, or combine, some or all of the regions or objects in a second annotated training set of images, each iteration involving finding a set of parameters for a pre-defined machine learning model such that the value of the cost function using the set of parameters is greater than or less than the value of the cost function using a different set of parameters in the previous iteration, the cost function being constructed to measure the difference between the predictions made for some or all of the regions or objects using the pre-defined machine learning model and the ground truth labels given to the second annotated training set of images.

[0162] In an optional block 1130, a target machine learning model is provided. For example, the target machine learning model may be deployed for execution in an image analysis environment, as described with respect to Figure 10.

[0163] In block 1135, a digital pathology scenario is identified. The digital pathology scenario may be a data incremental scenario, a domain incremental scenario, a class incremental scenario, or a task incremental scenario.

[0164] In block 1140, an adaptive sequential learning method is selected to update the target machine learning model in light of a digital pathology scenario. In some examples, the adaptive sequential learning method is selected from a group that includes Elastic Weight Reinforcement (EWC), Learning Without Forgetting (LWF), Incremental Learner and Representation Learning (iCaRL), Continuous Prototype Evaluation (CoPE), A-GEM, and parameter separation methods. In other examples, the adaptive sequential learning method includes EWC, LWF, iCaRL, CoPE, A-GEM, parameter separation methods, similar sequential learning methods, or any combination thereof.

[0165] In block 1145, the target machine learning model is updated based on an adaptive continuous learning method to generate an updated machine learning model.

[0166] In an optional block 1150, an updated machine learning model is provided. For example, the updated machine learning model may be deployed for execution in an image analysis environment, as described with respect to Figure 10.

[0167] In block 1155, a new image is received. The new image may be divided into image patches of a predetermined size. For example, a complete slide image typically has random sizes, and machine learning algorithms such as modified CNNs learn more efficiently with normalized image sizes (e.g., parallel computing on batches of images of the same size, memory constraints). Therefore, the image may be divided into image patches of a specific size to optimize the analysis. In some embodiments, the image is divided into image patches of a predetermined size of 64 pixels × 64 pixels, 128 pixels × 128 pixels, 256 pixels × 256 pixels, or 512 pixels × 512 pixels.

[0168] In block 1160, a new image or image patch is input to the target machine learning model or the updated machine learning model. In block 1165, the target machine learning model or the updated machine learning model detects, characterizes, classifies, or performs combinations thereof for some or all regions or objects within the new image or image patch, and outputs inferences based on the detection, characterization, classification, or combination thereof.

[0169] In the optional block 1170, the diagnosis of the object associated with the image or image patch is determined based on the inferences output by the revised machine learning model.

[0170] In the optional block 1175, the treatment is applied to the subject associated with the image or image patch. In some examples, the treatment is applied based on (i) the inference output of a machine learning model or a revised machine learning model, and / or (ii) the diagnosis of the subject determined in block 1170.

[0171] VII. Examples Systems and methods implemented in various embodiments can be better understood by referring to the following examples.

[0172] data CRC: In the following experiment, 100,000 non-overlapping patches from H&E-stained histological images of human colorectal cancer (CRC), consisting of nine tissue classes including adipose tissue (ADI), background (BACK), debris (DEB), lymphocytes (LYM), mucus (MUC), smooth muscle (MUS), normal colonic mucosa (NORM), cancer-associated stroma (STR), and colorectal adenocarcinoma epithelium (TUM), were used for training. Several exemplary images are shown in Figure 12. The test set contained 7,180 image patches that did not overlap with the training data. All images were color-normalized using Macenko's method.

[0173] The CRC dataset was augmented by individually and combinedly varying staining intensity, color, and saturation to simulate data collected from different staining machines, scanners, and pigments. Images were not mixed using non-negative matrix factorization. The four settings of color, saturation, and intensity were applied to individual stains from non-overlapping subsets of the original dataset. Different adaptive learning scenarios were created using each composite setting along with the original dataset. 1. Each extension was recognized as a domain shift from the original dataset. Thus, the dataset can be divided into five data streams, each representing a different extension or domain setting resulting in five experience domain incremental scenarios (although five were used in the experiments disclosed herein, it should be understood that the dataset can be divided into any number of "n" data streams, each representing a different extension or domain setting resulting in "n" experience domain incremental scenarios). The model learns to classify images in the new domain setting for all experiences the model is targeted. 2. Different extensions were also uniformly mixed across the classes. Using this mixed dataset, we created five data streams or experiences of equal size with representations from all classes (although five were used in the experiments disclosed herein, it should be understood that the dataset can be divided into any number "n" data streams or experiences of equal size with representations from all classes). This constitutes a data incremental scenario in which each subsequent experience is added to the model's training data. 3. The uniformly mixed datasets were further divided into subsets or experiences, each containing a different class, forming class-incremental scenarios. Depending on how the data was divided, the models were exposed to different classes in each experience and were targeted at all classes during the training process.

[0174] Sequential Learning Scenarios: Each composite setting, along with the original dataset, can be used to create different sequential learning scenarios. Each augmentation setting is a domain shift from the original dataset. They were used separately as separate data streams or as experiences in domain incremental settings. Images from different augmentation settings were uniformly blended across classes and split into experiences with equal class representations to constitute a data incremental scenario. The uniformly blended datasets were also split into experiences, each containing a different class, to form a class incremental scenario.

[0175] The PatchCam benchmark dataset consists of 327,680 patches extracted from 400 H&E-stained full slide images of lymph node sections from breast tissue, at 10x magnification and 96x96 pixel size. 75 / 12.5 / 12.5% ​​training / validation / test splits were selected using a hard-negative mining regime. The dataset has two classes (normal and tumor) to indicate the presence of metastatic tissue. For consistency and easy comparison with the CRC dataset, this dataset was also normalized using Macenko's method. Normalized examples from both classes are shown in Figure 13. The upper panel contains samples from the normal class, and the lower panel contains examples from the tumor class.

[0176] Continuous learning scenario: Dramatic domain shifts in the data stream were evaluated by training a model using the original CRC images (stain-normalized) in the first experience and the normalized PatchCam dataset in the second experience.

[0177] method The following sequential learning methods were experimented with to handle three scenarios: EWC and online EWC, LwF, iCaRL, CoPE, and A-GEM. All methods were compared to two baselines: 1) training from scratch (upper bound), where the same network architecture, an 18-layer ResNet, was targeted with all available data from all experiences seen so far; and 2) transfer learning or fine-tuning (i.e., lower bound), where the model was trained with the same design as sequential learning, exposing only to data available during a particular experience, but instead of using strategies to mitigate forgetting, the model was simply fine-tuned to adapt to new classes. For all experiments, the training epochs were set to 15 and the batch size to 16. The same ResNet architecture was used in a multi-head configuration, where each head was used for different tasks when testing with A-GEM according to their findings. A stochastic gradient descent optimizer was used, starting with a learning rate of 0.1, momentum of 0.9, and weight decay of 0.00001 applied after epochs 10 and 13.

[0178] Example 1 - Data Incremental Setting The first experiment involved a data incremental scenario in which more data was sequentially fed to the model, and then the model was updated based only on the latest data without accessing any of the older datasets in which it had previously been trained. Newer data streams may have the same classes as older streams, but with shifts in distribution. The mixed dataset used in this experiment should have a uniform distribution across experiences.

[0179] To train the model in this configuration, a method called Continuous Prototype Evaluation (CoPE) was used. CoPE is an online data incremental algorithm that uses prototypes to represent the most important features from the data. The prototypes evolve continuously as the model learns to keep up with changes in the data, enabling accurate predictions. CoPE also incorporates balanced replays to ensure that all classes are well represented in the replay population. The data was supplied online as mini-batches or mini-experiences, meaning the model sees each data sample only once and is therefore trained in a single epoch. To reduce forgetting, training with a batch size of 10 and momentum of 0.99 used a mini-experience size of 128 samples (i.e., each mini-experience had only 128 samples). Thus, for the data incremental scenario created for the augmented dataset, each of the five experiences has 99 mini-experiences.

[0180] At the end of training, the test streams, which contained samples from different experiences, had an average accuracy of 76%. Figure 14 shows the accuracy of the test streams at the end of each major experience under a data incremental setup using an augmented CRC dataset with CoPE. The experiences were divided into user-defined sizes of 128 images, each experience yielding 99 mini-experiences, which were fed to the model in an online manner. The numbers shown in the plots were the end of the 99th mini-experience or the end of all major experiences. The accuracy of the classification gradually increased, indicating that the model benefited from more data. The accuracy of individual test streams also increased as the model trained from newer training streams, indicating that the model had not forgotten what it had previously learned. Equal presentations from all synthetic domains in each experience resulted in transfer learning and test streams that benefited from all data. It is also noteworthy that the model ran similarly on test streams from subsequent experiences. For example, at the end of major experience 1, the model ran similarly on test stream 1 as well as test streams 2, 3, and 4, showing some transfer learning. The mixed augmentation dataset used here has equal representation from all synthetic domains, resulting in a similar distribution across experiences. Accuracy also improved by less than 1% between experiences 3 and 4, indicating that the model performed similarly on 80% of the data.

[0181] Example 2 - Domain Incremental Configuration The second experiment involved a domain incremental scenario with five experiences in which the hue, saturation, and intensity values ​​of two stains, eosin and hematoxylin, were varied to different degrees to mimic images acquired using different stains, scanners, and reagents. Examples from the five experiences are shown in Figure 15. Each row corresponds to one of nine tissue types, and three exemplary images are plotted from each extended setting (i.e., domain). Domain 0 (columns 1-3): Stain-normalized CRC dataset. Domain 1 (columns 4-6): Simulates scenarios of increasing eosin stain intensity, increasing eosin solution concentration, or extending staining time. Domain 2 (columns 7-9): Simulates scenarios of decreasing eosin intensity and aging slides with faded stains. Domain 3 (columns 10-12): Differences in hue changes between hematoxylin and eosin. Domain 4 (columns 13-15): Hue changes and increased saturation levels for both eosin and hematoxylin. Each of the three column sections can be considered a separate domain in a domain-incremental scenario. The domains can be mixed and divided into experiences, each having representations from all classes for a data-incremental scenario, or into mixed-domain experiences, each having a non-overlapping subset of classes for a class-incremental scenario.

[0182] This scenario employs a method called learning without forgetting (LwF). LwF is a combination of fine-tuning and distillation. LwF learns task-specific parameters for the new / current task using only the most recent data corresponding to the current task, without impairing performance on older tasks. Unlike traditional regularization, which penalizes parameter changes based on their importance, LwF penalizes changes in the input-to-output mapping. The loss function consists of two terms: the cross-entropy loss for the current task and a distillation loss to prevent previously acquired knowledge from being forgotten.

[0183] LwF performed well with over 86% accuracy in three of the five given domains. While the evaluation accuracy for domains 1 and 2 during the experience was 88% and 93%, respectively, the model targeted the corresponding domains, but knowledge retention was insufficient. However, nearly 28% of the acquired domain-specific knowledge was forgotten, particularly in domains 1 and 2, when specific domain data was no longer available. The results are shown in Figure 16, with test stream accuracy (left) and forgetting (right) at the end of each major experience under a domain incremental setting using the augmented CRC dataset with LwF. At the end of training, the model ran with 86–94% accuracy in three of the five given domains. The model performed well in domain 1 (88%) and domain 2 (93%), but the acquired domain-specific knowledge was forgotten during the experience after specific domain data was no longer available. The test stream forgetting metric was approximately 28% at the end of training, indicating accuracy loss from domains 1 and 2 throughout the training tire process. This experiment demonstrated that a model presented with data from different domains within a contiguous data stream can be trained to adapt reasonably well to a new domain while still maintaining its performance in the previous domain, which it no longer has access to.

[0184] Example 3. - Class Incremental Setting In the third experiment, the model was trained with three experiences and six classes in the class incremental setup described herein, so that the model could access data from only two classes during each experience, and newer classes were gradually added in each experience. Incremental Class & Representation Learning (iCaRL) was the adaptive learning strategy used here. iCaRL dynamically selects examples from the data stream, and each class has its own set of examples. iCaRL updates both parameters and examples when it sees new data. iCaRL performs classification by nearest neighbor mean of examples. iCaRL included representation learning by distillation and prototype rehearsal, where the augmented dataset included data from the current task, and the memorized examples and model parameters were updated based on the cross-entropy loss of newer classes and the distillation loss of previously learned classes.

[0185] There are two baselines for comparison with the iCaRL algorithm. One is (-1) training from scratch (upper bound), where the same neural network was targeted with all available data up to a specific experience. That is, the iCaRL algorithm was targeted with two classes during the first experience, four classes during the second experience, and all six classes during the third experience. The other baseline is (2) transfer learning or fine-tuning (lower bound), where the model was trained with the same design as adaptive learning, exposing only two classes between each of the three experiences, but instead of using a strategy to mitigate forgetting, the model was simply fine-tuned to adapt to newer classes. The results are shown in Figure 17. As shown, iCaRL was observed to perform comparably to the upper bound of training from scratch using only a portion of the data, and therefore also provide a significant computational advantage. Transfer learning performed well in experience 0, but for exposure to and learning of newer classes, it forgot previously acquired knowledge. While computationally comparable to iCaRL, its transfer learning performance was insufficient for catastrophic forgetting.

[0186] Example 4. - Comparison of Methods Sequential training using the extended CRC dataset. For fair comparison, the domain and data incremental experiments had 5 experiences, while the class incremental experiments had 4 experiences, with the first 3 experiences each having 2 classes and the last experience having the remaining 3 classes. A-GEM was treated as a task incremental method, having each experience introducing a new set of classes to the model with a task ID, as it was found to provide the best results with the task descriptor. Hyperparameters for each method were determined by grid search. CoPE and A-GEM were treated as online, small-shot methods and trained in just 1 epoch. iCaRL was experimented with in 3 settings. The first setting had 4 experiences, with the first 3 each having 2 classes and the last experience having the remaining 3 classes. The second setting also had 4 experiences, but the first experience had 3 classes and the remaining experiences each had 2 classes. The final setting had 3 experiences, each with 3 classes. The class order was the same across settings and was ascending. LwF was experimented with to continuously train on the original CRC dataset and the normalized PatchCam dataset in a domain-incremental setting, where one tumor type is considered one domain.

[0187] Figure 18 shows the evaluation accuracy at the end of training for three designed scenarios (data, domain, and class incremental) for sequential learning methods. In the data and domain incremental scenarios, LwF and iCaRL were comparable to the upper baseline. The class incremental scenario was a more difficult task overall to learn. iCaRL had the highest accuracy at 83%. A-GEM was the only method tested and evaluated in the task incremental scenario.

[0188] Data Incremental Scenario: LwF had an overall accuracy of 93% at the end of training, forgetting <1% of previously acquired knowledge. This was 4% better than the lower bound and within 0.5% of the upper bound. The accuracy per experience is shown in Figure 19. Specifically, Figure 19 shows data incremental experiences tested using different CL methods, as listed in the legend, which includes two baselines. Each experience consists of its own test stream containing examples from smaller batches of data that belong exclusively to that experience. Each subplot shows how the model performed against the test streams evaluated at the end of training for all experiences. The gray areas indicate experiences that the model has not yet targeted, and the model is expected to perform less well for these experiences. However, once targeted for a particular experience, ideally it should not forget what it learned and should retain the knowledge throughout the rest of the training process, i.e., accuracy should remain high in the areas not grayed out for all test streams. LwF had the highest overall accuracy at the end of training.

[0189] The classification accuracy gradually increased, indicating that the model benefited from more data. The accuracy of individual test streams also increased as the model trained on newer training streams, demonstrating that the model had not forgotten what it had previously learned. Another observation is the performance of the iCaRL method. Although iCaRL was designed as a class-incremental method, the concept of storing examples representing classes within each domain should theoretically have worked better than EWC and LwF. The maximum memory size tested was not sufficient to store only "n" classes like class-incremental methods, but was capable of storing "n" classes × "d" domains.

[0190] Domain Incremental Scenario: As shown in Figure 20, iCaRL performed better than EWC and LwF with the best hyperparameters selected from grid search. As shown in Figure 20, five domain incremental experiences were tested using different CL methods, as enumerated in the legend, which includes two baselines. Each experience consists of its own test stream containing examples from domains exclusively belonging to that experience. Each subplot shows how the model performed against the test streams evaluated at the end of training for all experiences. The gray areas indicate experiences that the model has not yet targeted, and the model is expected to perform less well for these experiences. However, once targeted for a particular experience, ideally it should not forget what it learned and should retain that knowledge throughout the rest of the training process; i.e., accuracy should remain high in the areas not grayed out for all test streams. It can be seen that iCaRL is comparable to the upper baseline, while the performance of EWC and LwF was closer to the lower baseline. Note that in the first experience, the training examples were few (20% of all examples), resulting in a low upper limit (accuracy of 0.57).

[0191] Interestingly, 1) the model retained knowledge from several domains more than others. Specifically, it was more difficult to retain learned knowledge about datasets where eosin intensity increased (domain 1; columns 4-6 in Figure 15) or decreased (domain 2; columns 7-9 in Figure 15), although there was some transfer learning between domains (changes in staining hue and / or saturation (domains 3 and 4) are not shown in Figure 15). Even after training only in domain 0, the model performed well on the test stream corresponding to domain 4. Performance in these two domains remained high even at the end of the training process. Since the fourth domain was generated using hue changes for staining, these results suggest that the model can handle a range of hue changes when continuously learning and retaining knowledge from a continuous data stream.

[0192] Class Incremental Scenario: iCaRL performed significantly better than the other methods. None of the tested methods, including the lower bound baseline, were able to retain knowledge about classes learned in previous experiences. The overall accuracy of iCaRL was 88%, which is about 6% lower than the upper bound of indirect training. This is still beneficial considering the lower load on data storage and resources. Figure 21 shows four-class incremental experiences tested with different CL methods, as listed in the legend, including two baselines. Each experience consists of its own test stream containing examples from classes exclusively belonging to that experience. Each subplot shows how the model performed against the test streams evaluated at the end of training for all experiences. Grayed-out areas indicate experiences for which the model has not yet been trained, and the model is expected to perform poorly for these experiences. However, once the model is targeted in a particular experience, ideally the model should not forget what it learned and should not retain knowledge throughout the rest of the training process. That is, accuracy should remain high in the areas that are not grayed out for all test streams. iCaRL is found to be the only CL method that performs well with the current experience and does not completely forget previous knowledge. Experience 0, the first experience the model targets, is the most forgotten experience with 64% accuracy at the end of the training process.

[0193] Few-shot online sequential learning: The greatest gains for both CoPE and A-GEM are proportional to the amount of training data each experience has. A-GEM was tested using a class incremental setup with task IDs, and model updates were based only on 128 randomly selected examples stored in memory in one epoch, producing results comparable to iCaRL trained over 15 epochs, rather than using an online method. The overall accuracy at the end of training was 79%, which was more than 50% better than the lower bound baseline. Detailed results are shown in Figure 22 - the dataset was initially designed as a class incremental scenario, with each experience assigned a separate task ID, and targeted in a multi-head architecture using an online method. The overall accuracy was approximately 79%, with a baseline lower bound of 27%. A-GEM also deteriorates significantly when used without task IDs. A-GEM was unable to retain any of the prior knowledge gained.

[0194] CoPE was tested in a domain incremental scenario, simulating online training by splitting the dataset into mini-experiences, each with the same number of examples as the mini-batch size. This resulted in overall accuracy 67%–11% better than the lower bound baseline. It is interesting to compare the results from CoPE with those from LwF and EWC from the domain incremental scenario. In both of the latter methods, the model did not perform well for test streams from domains 1 and 2. CoPE did not help retain knowledge from domain 1 (less than 25% accuracy), while the model had 60% accuracy from domain 2. The online setting may have helped retain more information. CoPE was also found to be sensitive to softmax temperature. In contrast to using temperatures >1 as other distillation methods, we tested temperatures lower than the stiffer softmax distribution, as recommended in the literature. Finer sweeps of hyperparameters may yield better results. Another point to note is how each experience is split into mini-batches or mini-experiences in CoPE. Each experience in the tested setting had 128 samples or examples. Not all classes are represented equally in each mini-experience, which can also affect overall accuracy.

[0195] Figure 23 shows a comparison between CoPE and the baseline. The experience was divided into user-defined sizes of 128 images, yielding 99 mini-experiences within each experience, which were fed to the model online. The numbers shown in the left plot represent the end of the 99th mini-experience or the end of all major experiences. The right side shows the results from the fine-tuned / naive baseline. Equal presentations from all synthetic domains in each experience resulted in transfer learning and test streams that benefited from all data for both CoPE and the baseline, but overall accuracy was better with CoPE.

[0196] The impact of class grouping on continuous learning Figure 24 shows the results of a third experiment using different class grouping settings. Three settings were tested: A. four experiences with two of the first three classes and the last three classes; B. four experiences with three classes in the first experience and two classes in the remaining experiences; and C. three experiences, each with three classes. Comparing the first two subplots, both with four experiences, the overall accuracy decreases by approximately 6.5%. The difference between the two settings is the number of classes the model targeted in the first experience. The hypothesis here is based on curriculum learning, where knowledge is better captured when more difficult tasks are followed by easier tasks. Starting with three classes here may make it more difficult for the model to learn and retain the knowledge, which is reflected in the decrease in accuracy despite the remaining experiences targeting only two classes in each setting. The same applies to the third setting. Despite fewer experiences, the model has to learn -3 more classes in each experience compared to 2 classes in setting 1, resulting in an almost 8% decrease in accuracy.

[0197] Continuous learning from multiple tumor types Both EWC and LwF were evaluated in this experiment, with LwF yielding slightly better results, as shown in Figure 25. The model was designed with CRC as the first domain and PatchCam as the second domain. The model started well with over 90% accuracy in the CRC trial stream, but towards the end of training, it forgot some of the knowledge gained from its initial experience. At the end of the training process, the model produced approximately 70% accuracy in the CRC trial stream and approximately 76% accuracy in the PatchCam trial stream.

[0198] conclusion This systematic study characterized the performance of various sequential learning methods for different scenarios using augmented digital pathology images and evaluated models when different tumor types were presented. The datasets were evaluated using regularization and rehearsal methods. EWC and LwF performed relatively well in data and domain incremental scenarios, but the rehearsal methods iCaRL and A-GEM were necessary to prevent catastrophic forgetting in more challenging class incremental scenarios. The few online methods tested require additional hyperparameter fine-tuning and experimental setup to fully understand their effectiveness. Furthermore, it is interesting to investigate how changes in images from a clinical perspective due to shifted patient populations, disease progression, and / or disease (sub)types affect the performance of these CL methods, providing insights into the feasibility of applying these methods in clinical practice. These experiments found that while retaining knowledge regarding staining intensity was difficult, the models appeared to be less sensitive to hue changes within the tested range. While some results indicate the difficulty of learning tumor classification from DP images, this study demonstrates the potential for continuous learning when adapting to changes in clinicopathological image acquisition factors.

[0199] VIII. Further Considerations Some embodiments of this disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-temporary computer-readable storage medium containing instructions, which, when executed by one or more data processors, cause one or more data processors to execute some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of this disclosure include a computer program product tangibly embodied in a non-temporary machine-readable storage medium, which includes instructions configured to cause one or more data processors to execute some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0200] The terms and expressions used are for illustrative purposes only, not limitation, and the use of such terms and expressions is not intended to exclude any equivalents of the illustrated features or parts thereof, and it should be recognized that various modifications are possible within the scope of the claimed invention. Accordingly, although the invention as described in the claims is specifically disclosed by embodiments and optional features, it should be understood that modifications and variations of the concepts disclosed herein may be reclassified by those skilled in the art, and such modifications and variations should be considered to fall within the scope of the invention as defined by the appended claims.

[0201] The subsequent description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the Disclosure. Rather, the subsequent description of preferred exemplary embodiments provides a possible description for implementing various embodiments for those skilled in the art. It will be understood that the function and arrangement of the elements may be varied without departing from the idea and scope described in the appended claims.

[0202] In the following description, specific details are given to provide a comprehensive understanding of the embodiments. However, it will be understood that embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in the form of block diagrams to avoid obscuring the embodiments with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments.

Claims

1. In a data processing system, obtaining a first annotated training set of images for training a machine learning algorithm to detect, characterize, classify, or combine parts or all of the regions or objects within an image, wherein the first annotated training set of images is located within a first image domain. The data processing system divides the first annotated training set of images into minisets of images, each miniset representing a separate modeling subtask and containing a limited number of examples. The data processing system trains the machine learning algorithm in a first step using the miniset of images in order to generate a pre-tuned machine learning model configured to detect, characterize, classify, or perform combinations thereof for some or all of the regions or objects within a new image. Labeling a limited number of images from a target dataset by the data processing system in order to generate a second annotated training set of images for training a machine learning algorithm to detect, characterize, classify, or combine some or all regions or objects within an image, wherein the second annotated training set of images is located in a second image domain. The data processing system trains the pre-tuned machine learning model in a second stage using the second annotated training set of images to generate a target machine learning model configured to detect, characterize, classify, or combine some or all of the regions or objects in the new image, wherein some of the classes targeted in the first stage are part of or match some of the classes targeted in the second stage. Computer implementation methods including

2. The computer implementation method according to claim 1, wherein the first annotated training set of images is a digital pathology image containing one or more types of cells.

3. The division described above means If only one miniset of images is available, for each separate modeling subtask, select a subset of classes that are part of or match the classes targeted in the second stage, and select the limited number of examples based on the selected subset of classes, and If multiple minisets of images are available, for each distinct modeling subtask, either (i) mix examples from the multiple minisets of images, select a subset of classes that are part of or match the classes targeted in the second stage, and select the limited number of examples from the mixed examples based on the selected subset of classes, or (ii) select one miniset of images from the multiple minisets of images, select a subset of classes that are part of or match the classes targeted in the second stage, and select the limited number of examples from the selected miniset of images based on the selected subset of classes, A computer implementation method according to claim 1 or 2, including the method described in claim 1 or 2.

4. The second stage described above is, Applying the pre-tuned machine learning model to generate a feature vector representation for each example in the second annotated training set of images, This involves generating one representation for each target class and combining the feature vector representations from examples of the same class to use one representation for each target class as a prototype for the target class, To generate a feature vector representation for the remaining unlabeled portion of the image or image region from the target dataset, Based on the distance between the feature vector representation from the unlabeled image and the prototype for the target class, each feature vector representation from the unlabeled image is compared to the prototype for the target class. A computer implementation method according to any one of claims 1 to 3, further comprising:

5. The first stage is, An internal learning loop in which the machine learning algorithm updates the model weights or parameters on one subtask in a predetermined or flexible number of epochs to initialize the pre-tuned machine learning model for adaptation to the target dataset, generating a loss in a validation set of model-updated images, denoted as L-subtask-i with respect to the i-th subtask, An external learning loop, represented as L-subtask-i, where i ranges from 1 to the number of subtasks calculated from the validation set of images for the subtasks with respect to model initialization, aims to find a set of model initializations that produce the pre-tuned machine learning model, when each subtask is used to update all subtasks, each with only a limited number of examples, by finding the model initialization that minimizes the sum of all losses. A computer implementation method according to any one of claims 1 to 3, further comprising:

6. The computer implementation method according to claim 1 or 2, wherein the first stage of training includes performing iterative operations to learn a set of parameters to detect, characterize, classify, or combine any part or all of any region or object within the miniset of images, each iteration involving finding the set of parameters for the machine learning algorithm such that the value of the cost function using the set of parameters is greater than or less than the value of the cost function using a different set of parameters in a previous iteration, the cost function being constructed to measure the difference between a prediction made with respect to some or all of the region or object using the machine learning algorithm and a ground truth label given to the miniset of images.

7. The computer implementation method according to claim 1 or 2, wherein the second stage of training comprises performing iterative operations to learn a set of parameters to detect, characterize, classify, or combine thereof some or all of the regions or objects in the second annotated training set of images that maximize or minimize a cost function, each iteration involving finding the set of parameters for the pre-tuned machine learning model such that the value of the cost function using the set of parameters is greater than or less than the value of the cost function using a different set of parameters in the previous iteration, the cost function being constructed to measure the difference between predictions made with respect to some or all of the regions or objects using the pre-tuned machine learning model and ground truth labels given to the second annotated training set of images.

8. Identifying digital pathology scenarios, Based on the aforementioned digital pathology scenario, an adaptive continuous learning method is selected to update the target machine learning model, The process involves updating the target machine learning model based on the adaptive continuous learning method described above to generate an updated machine learning model. A computer implementation method according to any one of claims 1 to 7, further comprising:

9. The computer implementation method according to claim 8, wherein the digital pathology scenario is a data incremental scenario, a domain incremental scenario, a class incremental scenario, or a task incremental scenario.

10. The adaptive continuous learning method is selected from the group including elastic weight reinforcement (EWC), learning without forgetting (LWF), stepwise learner and display learning (iCaRL), continuous prototype evaluation (CoPE), A-GEM, and parameter separation methods. The computer implementation method according to claim 8.

11. The computer implementation method according to claim 1 or 8, further comprising providing a target machine learning model and / or an updated machine learning model.

12. The computer implementation method according to claim 11, wherein providing the above includes deploying the target machine learning model and / or the updated machine learning model to a digital pathology system.

13. The aforementioned data processing system receives a new image, Inputting the aforementioned new image into the target machine learning model or the updated machine learning model, The target machine learning model or the updated machine learning model is used to detect, characterize, classify, or perform combinations thereof for some or all of the regions or objects within the new image. The target machine learning model or the updated machine learning model outputs inference based on the detection, characterization, classification, or a combination thereof. A computer implementation method according to any one of claims 1 to 12, further comprising:

14. The computer implementation method according to claim 13, further comprising the user determining a diagnosis of a subject associated with the new image, wherein the diagnosis is determined based on the inference output by the target machine learning model or the updated machine learning model.

15. The computer implementation method according to claim 14, further comprising the user performing a procedure on the subject based on (i) inferences output by the target machine learning model or the updated machine learning model, and / or (ii) the diagnosis of the subject.

16. The computer implementation method according to claim 1, wherein training the machine learning algorithm includes implementing a meta-learning principle so that the first step can generate the pre-tuned machine learning model using the limited number of examples.

17. The computer implementation method according to claim 1, wherein training the pre-tuned machine learning model includes implementing a meta-learning principle so that the second step can generate a target machine learning model using the limited number of images.

18. One or more data processors, A non-temporary computer-readable storage medium containing instructions, wherein, when executed by one or more data processors, the instructions cause the one or more data processors to perform any of the steps of the method described in any one of claims 1 to 17; A system equipped with these features.

19. A computer program product tangibly embodied in a non-temporary machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform any of the steps of the method according to any one of claims 1 to 17.

Images