Out-of-distribution generalization for deep learning models

Abstract

The present disclosure relates to use of normalization layers or techniques in deep learning models to enhance generalization and robustness of the model against domain shift or out-ofdistribution test data. More specifically, architecture of a deep learning model may be modified to include various normalization layers (e.g., of same type, different type, or a weighted combination) to improve accuracy of the model on unseen test data that is drawn from a different distribution as compared to the distribution of training dataset of the model. The normalization techniques may include instance normalization, layer normalization, batch normalization, group normalization, or switchable normalization. The disclosed approach provides a computationally or resource efficient method to improve model generalization without compromising (test) data integrity and eliminating a need to collect annotations and retraining of the model on a target domain (or shifted domain).

Description

OUT-OF-DISTRIBUTION GENERALIZATION FOR DEEP LEARNING MODELSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the priority to and the benefit of U.S. Provisional Application Number 63 / 548,781, filed on February 1, 2024, which is hereby incorporated by reference in its entirety for all purposes.BACKGROUND

[0002] Digital pathology (DP) is often used to guide a medical professional in diagnosis, prognosis and treatment assessment of a patient's disease. Stained biological samples (e.g., stained tissue slices) are mounted on glass microscope slides and are digitally imaged. A human pathologist or computational algorithm analyzes individual sections (referred to as patches) and characterizes cells in each patch. For example, for each patch, whether tumor cells are depicted in the patch, a quantity of tumor cells, or a spatial distribution of tumor cells may be determined. Such results can inform a diagnosis, treatment selection, treatment efficacy assessment, and so on.

[0003] Technological advances have enabled the digitization of histopathology slides into high-resolution images, providing opportunities to develop computer vision tools for a wide range of clinical applications. High-resolution, digital images of tissue slides make it possible to use computer-based analysis of slides. For example, machine learning models have shown promise as a tool in medical diagnostic applications and in predicting treatment outcomes.

[0004] Traditionally, machine learning models are trained with a data set that is representative of data that will be fed to the trained machine learning model. However, in clinical applications, there is often a domain shift where unseen test data comes from different sites with distribution shifts. This may result in suboptimal performance of the model. Thus, it is important to make machine learning models robust against these domain shifts or variations in appearance of images to generate accurate results. Therefore, methods, system, and / or techniques for analyzing histopathological images are needed that can effectively and accurately generate assessments to predict diseases using data received from multiple sites and are of different distributions.SUMMARY

[0004] Some embodiments of the present disclosure relate to use of normalization techniques to enhance generalization of deep learning models and to generate accurate results on unseen data with varying distribution or domain shift as compared to distribution of training dataset of the model. A computer-implemented method includes accessing a digital pathology (DP) image. The DP image may correspond to one or more particular imaging modalities or one or more stains. The DP image may be divided into smaller portions or tiles (e.g., 256x256 pixels or 512x512 pixels).

[0005] A label may be generated for each portion of a set of portions of the digital pathology image. The label corresponding to each portion may predict whether the portion depicts one or more tumor cells. The label can be generated by processing the portion using a trained deep learning model that includes multiple layers. The multiple layers may include a first convolution layer, a second convolution layer, and a normalization layer that is between the first convolution layer and the second convolution layer. A result may be generated based on the labels for the set of portions and the result may be output. The result may include a diagnosis of a disease, e.g., tumor.

[0006] In some instances, the normalization layer may be configured to perform an instance normalization. In some other instances, the normalization layer may be configured to perform a weighted combination of the instance normalization and a layer normalization. In some other instances, the normalization layer may be configured to perform a weighted combination of the instance normalization, the layer normalization, and a batch normalization.

[0007] In some embodiments, the multiple layers of the trained deep learning model may further include a third convolution layer and a second normalization layer. In some instances, one of the (first) normalization layer and the second normalization layer can be configured to perform the instance normalization and another of the (first) normalization layer and the second normalization layer can be configured to perform a different type of normalization. The different types of normalization include layer normalization, batch normalization, and group normalization.

[0008] Further, the trained deep learning model may include a residual network (ResNet), a densely connected convolutional network (DenseNet), or a convolutional neural network (CNN).The DP image may be drawn from a distribution that can be different (domain shift) from the distribution of training dataset of the model.

[0009] The techniques disclosed in the present disclosure can be utilized to develop robust and generalized deep learning models for various clinical applications.

[0010] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0011] In some embodiments, a computer-program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and that includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein.

[0012] In some embodiments, a system is provided that includes one or more means to perform part or all of one or more methods or processes disclosed herein.

[0013] The terms and expressions which have been employed are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention claimed. Thus, it should be understood that although the present invention as claimed has been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosed may be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this invention as defined by the appended claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. The present disclosure is described in conjunction with the appended figures.

[0015] FIG. 1 illustrates an example of a traditional method or architecture to address domain shift or stain variations problem in digital pathology.

[0016] FIG. 2 is a block diagram illustrating an example overview of a system to generate and utilize robust and generalized deep learning models for out-of-distribution (OOD) dataset in accordance with some embodiments of the present disclosure.

[0017] FIG. 3 illustrates an exemplary network of an image generation system of FIG. 2 to generate digital pathology images.

[0018] FIG. 4 illustrates an example architecture of a classification model that is based on normalization layers to mitigate domain shift problem in accordance with some embodiments of the present disclosure.

[0019] FIG. 5 shows an example flowchart of a system performing classification on the digital pathology images using the classification model in accordance with some embodiments of the present disclosure.

[0020] FIG. 6 shows illustrative examples of sample tiles extracted from three different sites including GOYA, CAVALLI, and ISTS to demonstrate domain shift across sites.

[0021] FIG. 7 illustrates 2D embedding visualizations for batch normalization and instance normalization techniques in accordance with an example implementation of the present disclosure.DETAILED DESCRIPTION

[0022] Some embodiments of the present disclosure relate to use of normalization techniques in deep learning models to enhance models’ robustness and generalization capabilities to obtain accurate results even with out-of-distribution (OOD) inputs. More specifically, architecture of a deep learning model may be modified to include various normalization layers (e.g., of same type, different type, or a weighted combination) to improve accuracy of the model on unseen test data that is drawn from a different distribution as compared to the distribution of training dataset of the model. According to some embodiments, a technical solution is provided in the present disclosure to a technical problem of improving robustness and generalization of machine learning models for OOD dataset or domain shift.

[0023] Machine learning models used in clinical applications often use training and testing data that have similar distributions of properties. For example, a model trained solely with black- and-white images is typically used to process other black-and-white images (not color images). In clinical applications, a domain shift may frequently be present in the test data. The domain shift may correspond to unseen test data that is drawn from different distributions relative to that of training data, which may result in suboptimal performance of the model. Re-training the model once new data comes from a different distribution adds to the computational complexity of the model. Thus, the need to employ generalized machine learning models in clinical applications is imperative to increase the performance of the models. Generalization refers to the model’s ability to adapt to new, previously unseen data, drawn from a different distribution as the one used to train the model.

[0024] In some embodiments of the present disclosure, the term “domain” may refer to a representation of an image using a specific stain or imaging modality. The domain may correspond to different medical imaging modalities (e.g., radiology imaging, brightfield imaging etc.) or different stains in digital pathology (e.g., hematoxylin and eosin H&E staining, multiplex staining etc.). Exemplary domains include H&E staining of digital pathology slides, multiplex staining of digital pathology slides (with any combination of stains), singleplex staining of digital pathology slides (with any given stain), immunohistochemistry IHC (using any given antibody). The exemplary domains may further include imaging modalities such as darkfield microscopy, brightfield microscopy, fluorescent microscopy, or radiology imaging (e.g., magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), etc.). For a given domain, domain shift may correspond to potential changes or variations in the appearance (e.g., colors, pixel intensity values, etc.) of the images of test dataset as compared to training dataset of the machine learning model. For example, in digital pathology, stain variation is a common phenomenon that is observed when distinct pathology laboratories stain tissue slides that exhibit similar but not identical color appearance within or across the slides of distinct pathology laboratories.

[0025] In the present disclosure, the term “sample” may be understood as material derived from a biological organism, comprising but not limited to hair, skin samples, tissue samples, cultured cells, cultured cell media, and biological fluids. The term “tissue” refers to a mass of interconnected cells (e.g., central nervous system (CNS) tissue, liver tissue, or eye tissue) derivedfrom a human or other animal. Samples may include the connecting material and the liquid in association with the cells, such as blood samples. In the context of histopathology, the term “slide” refers to a glass microscope slide carrying a thin section of tissue that has been stained for microscopic examination. The term sample may also include media containing isolated cells. One skilled in the art may determine the quantity of samples required to obtain a reaction by standard laboratory techniques. In the present disclosure, the term “singleplex” may refer to an image that displays a single staining component or a marker. The term is often used in contrast to a “multiplex” or “MPX” image, which involves the simultaneous visualization of multiple staining components within a single cell or tissue sample.

[0026] One approach to improve model generalization against domain shift or stain variations is to use stain normalization models as shown in Figure 1. Figure 1 illustrates an example of a traditional method or architecture to address domain shift or stain variations problem in digital pathology. An input image or its patch (or portion or tile) may initially be sent to a stain normalizing model. The stain normalizing model may align the color distributions between the training and test data to mitigate differences in stain appearance and may output a normalized stain image. The stain normalization model can be a deep learning model that is trained (unsupervised, self-supervised, or supervised manner) to perform an image-to-image translation task, for example a UNet architecture-based model (also referred hereinafter as ‘StainNorm model’). Afterwards, the normalized stain image or its patch can be fed into a machine learning model to generate output. The machine learning model may include classification, regression, or segmentation model including but not limited to tumor segmentation, cell segmentation, disease classification, and the like. For example, the machine learning model can be a tumor classifier and may utilize ResNetl8 architecture to classify each normalized stain image into ‘tumor’ or ‘normal’ category.

[0027] While techniques such as stain normalizing model (e.g., StainNorm) as a preprocessing tool can be effective in addressing staining inconsistencies, these techniques (i.e., stain normalizing models) introduce added complexity (e.g., 2x model size), longer training times, and higher computational costs. Additionally, the stain normalizing model may focus solely on stain variation, leaving other domain shift factors unaddressed, and may also affect the integrity of the original dataset (input images) by introducing artifacts in the normalized stain images. Other traditional methods or approaches to make robust machine learning modelsinclude data augmentation, transfer learning, domain adaptations, or continual learning. Data augmentation may simulate a range of potential stain variations during training, helping to improve model’s generalizability. The effectiveness of data augmentation approach may diminish when dealing with complex and unknown variations, which is a common challenge in medical imaging. Transfer learning, domain adaptation, and continual learning may demand retraining on target domains, requiring access to target domain dataset and annotations, and thus contradicting the objective of minimal retraining. Moreover, access to target domain data and annotations may not be feasible in all scenarios. Furthermore, fine-tuning on the target domain can degrade performance on the source data (or training data used initially), especially when the source data and the target data have significant discrepancy between their distributions. Due to these observations, the present disclosure focuses on resource efficient approach and utilizes normalization layers or techniques in the deep learning model architecture to improve robustness or generalizability of the model.

[0028] In some embodiments, techniques are provided to train, use, and / or avail a deep learning model for digital pathology, where the deep learning model includes one or more normalization techniques across the layers of the model architecture in the digital pathology domain. More specifically, the deep learning model is configured to include one or more normalization layers within a neural network, and the model can be utilized with the unseen test data that includes images with domain shift. The accompanying data illustrates that the disclosed technique generates clinically meaningful accurate results and achieving generalized applicability, despite extreme domain shifts and the disclosed technique does not demand retraining on the target domain data.

[0029] In some instances, the machine learning model (or the deep learning model) may be implemented on a cloud platform and can be accessed via a cloud-app or a service by various client sites (or remote DP labs). In some instances, a digital pathology image may be received from a client system associated with a client site. The digital pathology image may be preprocessed. The preprocessing may include standardizing the digital pathology image and / or applying an affine transformation. The affine transformation facilitates correcting geometric distortions or deformations in the images that occur with non-ideal camera angles. In some instances, a pre-processing normalization is applied (e.g., by normalizing each channel such as red, green, blue (RBG) channels of an image with its mean and standard deviation).

[0030] The digital pathology image (e.g., in its original or pre-processed form) can be fed to the machine learning model. The machine learning model may include a neural network, such as a deep-learning network (e.g., with at least 3 layers and / or at least 3 convolutional layers). The machine-learning model may include a convolutional neural network (CNN), a deep neural network, a transformer-based model, etc. CNN-based models may further include architectures such as ResNet, DenseNet, EfficientNet, or InceptionNet (e.g., InceptionV2, V3, V4), and the like.

[0031] The machine-learning model may further include one or more, two or more, or three or more normalization layers. At least one, at least two, at least three, or each of the normalization layers may perform instance normalization (IN). At least one, at least two, at least three, or each of the normalization layers may perform group normalization (GN). At least one, at least two, at least three, or each of the normalization layers may perform batch normalization (BN). At least one, at least two, at least three, or each of the normalization layers may perform switchable normalization (SN). At least one, at least two, at least three, or each of the normalization layers may perform layer normalization (LN).

[0032] One or more normalization techniques (or two or more normalization techniques) can be integrated across different layers of the deep learning model. For example, to avoid diminishing the content discrimination in the deep layers of the model, instance normalization can be integrated into some layers (e.g., in the shallow layers) or all layers of the deep learning model. Feature divergence caused by appearance is exhibited in the shallow layers of the deep learning models. Instance normalization can contrast to batch statistic synchronization across nodes in the shallow layers and can be performed on a single instance of the data. In some embodiments, batch normalization is performed on batch data in the deeper layers of the model. The batch normalization may be absent and / or partially present in shallow layers to preserve content in these layers. Use of multiple normalization techniques across different layers of the deep learning model may enhance the generalization of the deep learning model which may enable it to seamlessly work across domain shifts and may improve the learning rate of the model.

[0033] Integration of one or more normalization layers (or techniques) can support a model that was trained on a training data from a first distribution (of a particular domain) to be used togenerate accurate predictions when data fed from a second (or more) distribution(s) (of the particular domain). The disclosed technique provides a simple, computationally efficient, and resource efficient approach that does not need additional modules (that may add to the complexity of the model), avoids the use or need for target data (which is not available during training the model in realistic settings), and eliminates the need of frequent retraining of the deployed model, for example, at clinical sites, or as a cloud-app, thereby supporting widespread efficient and useful utilization of the model. Given the high distribution incongruence that is observed across sites (e.g., imaging centers, laboratories, medical facilities, etc.), the multiple normalization layers can facilitate generalized use of a model across different digital-pathology domains. This approach can reduce computational resources both during training the model (as extra modules are not needed), as well as during the deployment of the model at clinical sites or on the cloud (as models does not need retraining or fine-tuning for domain shifts due to data heterogeneity across client sites) and may improve accuracy.

[0034] In some instances, a single type of normalization is performed at one or more layers of a machine-learning model (e.g., a deep learning model). For example, instance normalization may be performed within one or more or two or more layers of a neural network. This may include performing an instance normalization at a normalization that feeds to a convolution layer. In some other instances, at least one instance normalization is performed between multiple convolutional layers.

[0035] In some instances, during the training process, the deep learning model may learn to select a normalization technique for a normalization of the model. The model may switch across the normalization techniques by learning the weights of each normalization technique across the nodes of each layer during the backpropagation of the training process of the deep learning model.

[0036] Further, a trained deep learning model may predict the classification label in clinical applications such as tumor detection for each image patch or tile (i.e., each image tile contains tumor image or normal image). A binary, qualitative or quantitative score can be generated based on one or more patch-specific predictions (e.g., whether any specific cell types are detected and / or a quantity of cell types that are detected). A result can be output based on the score.

[0037] It will be appreciated that techniques disclosed herein can further be applied to configure a model to perform cell segmentation and / or cell classification in the context of a domain shift (using normalization approaches disclosed herein). Moreover, the disclosed techniques can be utilized to develop robust and generalized models for other applications or domains such as radiology imaging.

[0038] FIG. 2 is a block diagram illustrating an example overview of a system to generate and utilize robust and generalized deep learning models for OOD dataset across DP clinical sites in accordance with an embodiment of the present disclosure. Exemplary system 200 may include an image generation system 205 connected to one or more computer systems 215 through a network 210. The image generation system 205 may include radiology imaging devices (e.g., MRI, CT scans, PET scans, fundus scans, etc.) or digital pathology imaging system. The deep learning model having normalization layers may be trained and executed on the computer system 215. The computer system 215 may also include user input and output devices (not shown) such as a keyboard, mouse, stylus, and a display / touchscreen. The computer system 215 may receive one or more digital pathology images from the image generation system 205 and may execute the deep learning model to perform classification and / or segmentation.

[0039] The computer system 215 of the exemplary system 200 may include a processing system with one or more processors, high-speed central processing unit(s) (CPU), and one or more memories. The computer system 215 may also include a memory for storing a plurality of processing modules or logical instructions that are executed by the one or more processors coupled. The computer memory that stores data may also be maintained on a computer readable medium including magnetic disks, optical disks, organic memory, and any other volatile (e.g., random access memory (RAM)) or non-volatile (e.g., read-only memory (ROM), flash memory, etc.) mass storage system readable by the CPU. The computer readable medium may include cooperating or interconnected computer readable medium, which exist exclusively on the processing system or can be distributed among multiple interconnected processing systems that may be local or remote to the processing system. The computer system 215 may be deployed locally at each clinical site or remotely, for example, using a cloud infrastructure.

[0040] The network 210 may include, internet, an intranet, a wired LAN (local area network), a wireless LAN (WiLAN), a WAN (wide area network), a MAN (metropolitan area network), a PSTN (public switched telephone network) and other types of communicationsnetworks. The network 210 may further include communication devices such as one or more gateways, routers, or bridges. Merely by way of example, the network 210 can have one or more servers and one or more web-sites accessible by users to send and receive information usable by the computer system 215. The network 210 may be any type of network familiar to those skilled in the art that can support data communications using any of a variety of available protocols, including without limitation TCP / IP (transmission control protocol / Internet protocol), SNA (systems network architecture), IPX (internet packet exchange), AppleTalk®, and the like.

[0041] The exemplary system 200 may further include the one or more databases 220 for the processing and storing of image data (e.g., histopathology images). The one or more databases 220 may store a training dataset and a validation dataset of a given domain to train the deep learning model. The one or more databases 220 may also store unseen test data from different clinical (or client) sites and results that are generated by the deep learning model. The one or more databases 220 may be integral to a memory system on a computer or in secondary storage such as a hard disk, floppy disk, optical disk, or other non-volatile mass storage devices. The computer system 215 may include a client terminal in communication with one or more servers, or personal digital / data assistants (PDA), laptop computers, mobile computers, internet appliances, one or two-way pagers, mobile phones, or other similar desktop, mobile or hand-held electronic devices.

[0042] Further, the computer system 215 may provide a means for inputting image data from the image generation system 205 to one or more databases 220. For example, the image data may depict one or more scanned digital pathology slides from the image generation system 205. The image data may include data related to color channels (RGB) for brightfield imaging. In fluorescence imaging the image data may include data related to multiple distinct channels. Each channel may correspond to or be responsible for capturing a particular spectral range or (signal) wavelengths emitted from the fluorophores. Hence, each channel provides an image with a representation of a specific biomarker. For instance, a biological specimen, for example, a tissue section may need to be stained by means of application of a staining assay to highlight one or more different biomarkers associated with chromogenic stains for brightfield imaging or fluorophores for fluorescence imaging. Staining assays can use chromogenic stains for brightfield imaging, organic fluorophores, quantum dots, or organic fluorophores together with quantum dots for fluorescence imaging, or any other combination of stains and viewing orimaging devices. In the analysis of biological specimens, for example, cancerous tissues, different stains are specified to identify one or more types of biomarkers, for example, immune cells.

[0043] FIG. 3 illustrates an exemplary network of the image generation system 205 of FIG. 2 to generate digital pathology images. The image generation system 205 may include a fixation / embedding system 305, a tissue slicer 310, a staining system 315, and an imaging system 320. The fixation / embedding system 305 fixes and / or embeds a tissue sample (e.g., a liquid fixing agent, such as formaldehyde solution) and / or an embedding substance (e.g., a historical wax, such as paraffin wax and / or one or more resins, such as styrene or polyethylene). Each slice may be fixed by exposing the slice to a fixating agent for a predefined period of time (e.g., at least 3 hours) and by then dehydrating the slice (e.g., via exposure to an ethanol solution and / or a clearing intermediate agent). The embedding substance can infiltrate the slice when it is in liquid state (e.g., when heated).

[0044] The tissue slicer 310 then slices the fixed and / or embedded tissue sample (e.g., a sample of a tumor) to obtain a series of sections, with each section having a thickness of, for example, 4-5 microns. Such sectioning can be performed by first chilling the sample and then slicing the sample in a warm water bath. The tissue can be sliced using (for example) a vibratome or compresstome.

[0045] Because the tissue sections and the cells within them are virtually transparent, preparation of the slides typically includes staining (e.g., automatically staining) the tissue sections to render relevant structures more visible. In some instances, the staining is performed manually. In some instances, the staining is performed semi-automatically or automatically using the staining system 315.

[0046] The staining can include exposing an individual section of the tissue to one or more different stains (e.g., consecutively, or concurrently) to express different characteristics of the tissue. For example, each section may be exposed to a predefined volume of a staining agent for a predefined period of time. The staining agent can include (for example) an RNA probe, protein probe (e.g., nuclear-protein probe or cytoplasm-protein probe), an immunohistochemistry stain, a probe for a secreted substance, etc. In some instances, the staining agent is one that stains for KAPPA mRNA or LAMBDA mRNA.

[0047] One exemplary type of tissue staining is histochemical staining, which uses one or more chemical dyes (e.g., acidic dyes, basic dyes) to stain tissue structures. Histochemical staining may be used to indicate general aspects of tissue morphology and / or cell microanatomy (e.g., to distinguish cell nuclei from cytoplasm, to indicate lipid droplets, etc.). One example of a histochemical stain is hematoxylin and eosin (H&E). Other examples of histochemical stains include trichrome stains (e.g., Masson's Trichrome), Periodic Acid-Schiff (PAS), silver stains, and iron stains. The molecular weight of a histochemical staining reagent (e.g., dye) is typically about 500 kilodaltons (kD) or less, although some histochemical staining reagents (e.g., Alcian Blue, phosphomolybdic acid (PMA)) may have molecular weights of up to two or three thousand kD. One case of a high-molecular-weight histochemical staining reagent is alpha-amylase (about 55 kD), which may be used to indicate glycogen.

[0048] Another type of tissue staining is immunohistochemistry (IHC, also called "immunostaining"), which uses a primary antibody that binds specifically to the target antigen of interest (biomarker). IHC may be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or fluorophore). In indirect IHC, the primary antibody is first bound to the target antigen, and then a secondary antibody that is conjugated with a label (e.g., a chromophore or fluorophore) is bound to the primary antibody. The molecular weights of IHC reagents are much higher than those of histochemical staining reagents, as the antibodies have molecular weights of about 150 kD or more.

[0049] The sections may then be individually mounted on corresponding slides. The imaging system 320 can then scan the slides to generate digital-pathology images 325a-n. Each section may be mounted on a slide, which is then scanned to create a digital image that may be subsequently examined by digital pathology image analysis and / or interpreted by a human pathologist (e.g., using image viewer software). The imaging system 320 may digitize pathology slides (whole slide or a section) using bright-field imaging, dark-field imaging, or fluorescence imaging. The imaging system 320 can include but is not limited to microscope with digital camera, robotic microscopes, or WSI scanners such as Ventana iScan HT, Ventana DP 200, or Ventana DP 600.

[0050] In some instances, a pathologist may review and manually annotate the digital image of the slides (e.g., tumor area, necrosis, etc.). Annotation of regions of interest may be performedautomatically using a computer-vision technique. Digital-pathology images 325a-n may be used by the machine learning model (or the deep learning model) for further processing.

[0051] A digital histopathology image (e.g., 325a) typically includes an array, usually a rectangular matrix, of pixels. Each “pixel” is one picture element and is a digital quantity that represents some property of the image at a location in the array corresponding to a particular location in the image. Typically, in continuous tone black and white images the pixel values represent a gray scale value. Pixel values for a digital image typically conform to a specified range. For example, each array element may be one byte (i.e., eight bits) representing pixel values in the range of 0 to 255. In a gray scale image, a “255” may represent absolute white and zero (‘0’) an absolute black (or visa-versa). Color images may comprise of three-color planes, generally corresponding to red, green, and blue (RGB). For a particular pixel, there is one value for each of these color planes, (i.e., a value representing the red component, a value representing the green component, and a value representing the blue component). By varying the intensity of these three components, all colors in the color spectrum are typically created. A specimen stained by multiplex IHC may be illuminated sequentially with multiple light channels matched to the absorbance bands of the chromogens to capture brightfield images. In the case of multiplex immunofluorescence, fluorescence microscopy with different filters may be used to capture fluorescence or emitted light from fluorophores associated with each biomarker.

[0052] FIG. 4 illustrates an example architecture 400 of a classification model 410 that is based on normalization layers to mitigate domain shift problem in accordance with some embodiments of the present disclosure. An input image 405 may be fed to the classification model 410 to generate an output 415. The input image 405 can be a digital pathology image (e.g., WSI), a patch (or tile), or represents a particular section of WSI. The classification model 410 may comprise a deep learning model (e.g., with at least 3 layers and / or at least 3 convolutional layers) such as deep neural network (DNN), convolutional neural network (CNN), transformer-based architectures, etc. CNN-based models may further include architectures such as ResNet, DenseNet, EfficientNet, or InceptionNet (e.g., InceptionV2, V3, V4), and the like. Moreover, the classification model 410 may include one or more, two or more, three or more normalization layers.

[0053] The example architecture 400 of the classification model 410 may represent a modified ResNet architecture (e.g., ResNetl8 model). In the modified ResNet architecture,normalization layers 430a-b are introduced. For instance, a residual block 420 of the modified ResNet architecture (as shown in FIG. 4) may include convolution layers 425a-b, normalization layers 430a-b, a rectified linear unit (ReLU) 435a. The residual block 420 may refer to as a building unit of the entire network. A complete model may include multiple residual blocks. The ResNet (residual networks) architecture may utilize skip connections or residual connections in the network to learn residual mappings instead of direct mappings, which resolves the vanishing gradient problem and enables training of deep networks. For each of the residual blocks in the example architecture 400, an input may pass through a first convolutional layer 425a, which may apply filters (e.g., 3x3) to extract feature maps. In the modified ResNet architecture, the first convolutional layer 425a may be followed by a first normalization layer 430a. Afterwards, the ReLU 435a activation can be applied. Similarly, a second convolution layer 425b and a second normalization layer 430b can be applied sequentially. Further, the input to the block (e g., the residual block 420) is added to the output of the second normalization layer 430b, forming the residual connection. Finaly, after adding the residual (input) to the output of the second normalization layer 430b, the result is passed through a second ReLU activation 435b (e.g., in the case of ResNetl8 architecture). This output is then passed as input to a next residual block or to the final network layers (e.g., fully connected neural network layers).

[0054] According to present disclosure, various normalization techniques may be employed at the normalization layers 430a-b of the classification model 410. The normalization layers 430a-b may include batch normalization (BN), instance normalization (IN), layer normalization (LN), group normalization (GN), or switchable normalization (SN). In some instances, a combination of different normalization techniques may be used within the classification model 410. For example, both BN and IN may be integrated within a single network such that BN may be applied to a subset of channels and IN to the rest of the channels, achieving a balance between stable training (BN’s strength) and the capacity to normalize style variations within the data (IN’s strength). In some instances, both techniques (IN and BN) may be applied concurrently to the same feature map (also referred herein as IBN-Net(a)). In some other instances, each normalization method (e.g., IN and BN) may be employed distinctively on two separate, nonoverlapping subsets of the feature map (also referred herein as IBN-Net(b)).

[0055] Normalization techniques may enhance the generalization of the classification model 410. Normalization techniques may focus on stabilizing the training process and may enable themodel to concentrate on the important attributes of the data. The normalization techniques typically transform inputs X using a normalization formula as shown in Equation 1.

[0056] In Equation 1, p and G2represent the mean and variance, respectively. Moreover, y and P represent scaling and shift coefficients, respectively, and are learnable parameters that allow the neural network to restore the representation power of the layer. Also, c is a small constant for numerical stability. The normalization techniques may differ primarily in how p and G2are computed.

[0057] Batch normalization (BN) operates on mini -batches of data. For each feature, BN may compute the mean (po) and variance (G2B) across the mini-batch, then normalizes the feature values. BN can help in reducing internal covariate shift, making the landscape of the optimization problem smoother. BN efficacy usually depends on the size of the mini-batch and BN may introduce noise during training with small batch sizes.

[0058] Layer normalization (LN) normalizes the input across the features instead of the batch dimension. The mean (p ) and variance (G2I ) are computed across all features for each individual data point in a layer’s input. LN is particularly effective in recurrent neural networks and other architectures where the batch size is one or varies dynamically.

[0059] Group normalization (GN) may divide the channels into groups and computes the mean (pc) and variance (G2G) within each group for normalization. GN’s effectiveness is not contingent on the size of the mini-batch, making it suitable for tasks with smaller batch sizes or when batch sizes are not consistent.

[0060] Instance normalization (IN) normalizes each channel in each training sample separately. IN computes the mean (pi) and variance G2I for each feature channel of each data instance.

[0061] Switchable normalization (SN) may utilize different normalization methods (BN, IN, LN, etc.) for each normalization layer. SN may learn a set of weights (wi) for each type of normalization by combining the strengths of BN, IN, and LN in a flexible and data-driven manner. In some instances, during the training process, the deep learning model or the classification model 410 may learn to select a normalization technique for a normalization layer of the model. The model may switch across the normalization techniques by learning the weightsof each normalization technique across the nodes of each layer during the backpropagation of the training process of the deep learning model.

[0062] In some instances, a single type of normalization is performed at each of the normalization layers 430a-b of the classification model 410 (e.g., a deep learning model). For example, instance normalization may be performed at the normalization layers 430a-b of the example architecture 400. In some instances, different normalization techniques may be performed at distinct normalization layers of the classification model 410 (e.g., as in IBN-Net(a) and IBN-Net(b)).

[0063] Afterwards, once the classification model is trained, the trained model may be used to predict a classification label in clinical applications such as tumor detection for each image patch or tile (i.e., each image tile contains tumor image or normal image). A binary, qualitative or quantitative score can be generated based on one or more patch-specific predictions (e.g., whether any specific cell types are detected and / or a quantity of cell types that are detected). A result can be output based on the score.

[0064] In digital pathology (DP), the effectiveness of predictive models relies heavily on the quality and representativeness of the training data. To achieve excellent performance, models may need to be trained and tested on data that is independent and uniformly distributed. In real- life, test data may frequently exhibit domain shifts due to practical settings such as variations in patient demographics, clinical environments, scanner types, and staining methods. In digital pathology domain, the images are diverse in nature, exhibiting varying distributions within or across distinct clinical sites, due to factors such as differences with respect to staining material, concentration of chemicals, reaction times, pre-analytic conditions of tissue sample, protocol followed, imaging device (bright field, fluorescent etc.), number of stains, biological samples type (e.g., tissue samples, cytological smears etc.) and region (e.g., liver, spleen etc.). These differences can introduce significant heterogeneity in the data across institutions or client sites. Digital pathology images, often captured at much higher resolutions (gigapixels), are complex and may need preprocessing techniques (e.g., tiling or patches, target regions, masking etc.) to standardize the data for the machine learning models. Different slide scanners (e.g., Leica, Hamamatsu, etc.) and staining protocols can further introduce variability in appearance, making it difficult to obtain accurate results using a machine learning model.

[0065] Therefore, according to some aspects of the present disclosure, normalization techniques may be utilized in the architecture of the deep learning model (or the classification model 410) to achieve generalization on OOD dataset. The disclosed technique can achieve robust model performance across a spectrum of clinical environments and conditions, such as different sites and Stainers. Moreover, the normalization layers 430a-b can be integrated seamlessly into existing model architectures without necessitating additional modules, avoiding the use or need for target data, and eliminating the need for retraining the model. Furthermore, above mentioned benefits can be achieved without compromising the model’s performance on in-distribution data or introducing artifacts in the outputs.

[0066] FIG. 5 shows an example flowchart of a system performing classification on the digital pathology images using the classification model 410 in accordance with some embodiments of the present disclosure. The blocks in flowchart are illustrated in a specific order, while the order can be modified, for example, some blocks may be performed before other, and some blocks may be performed simultaneously. The blocks can be performed by hardware or software or a combination thereof. The process at block 505 may include accessing a digital pathology image. The DP image may correspond to one or more particular imaging modalities such as multiplex digital pathology image, MPX fluorescence microscopy, or MPX brightfield microscopy. The DP image may further correspond to one or more particular stains (e.g., mIHC, H&E). For example, a slide with a slice of a tissue sample can be stained using multiple IHC markers, resulting in multiplex digital pathology image that may be a duplex, a triplex or fourplex image etc. If only one IHC marker is used thereby generating a singleplex image comprising one color / stain. The DP image can be preprocessed to break into different portions or tiles.

[0067] A label may be generated for each portion of a set of portions of the digital pathology image, at block 510. The label corresponding to each portion may predict whether the portion depicts one or more tumor cells. The label can be generated by processing the portion using a trained deep learning model that includes multiple layers. The multiple layers may include a first convolution layer, a second convolution layer, and a normalization layer that is between the first convolution layer and the second convolution layer.

[0068] A result may be generated based on the labels for the set of portions, at block 515.The result may include a diagnosis of a disease, e.g., tumor. Finally, at block 520, the result may be output.EXAMPLES;

[0069] An example implementation of the disclosed techniques to achieve OOD generalization of deep learning models is provided on three distinct datasets. In the example implementation, a modified ResNetl8 architecture having the normalization layers 430a-b was used as the classification model 410 as described in FIG. 4. The core of the classification model 410 was the modified ResNetl8 architecture appended with a binary classification head for identifying tumor presence in image tiles. For robust cross-domain generalization due to the datasets diversity from site variation, the experiments leveraged three distinct in-house hematoxylin and eosin (H&E) stained diffuse large B cell lymphoma (DLBCL) datasets, for training, validation, and testing. During preprocessing of the datasets, artifacts were removed (e.g., pen markers, crush artifacts, blur, fat, etc.) using in-house quality control (QC) models. Further, slides images or tiles were adjusted to a uniform 256x256 size and a 20X magnification (0.5pm / pixel). A pathologist non exhaustively and digitally annotated tumor areas.

[0070] FIG. 6 shows illustrative examples of sample tiles extracted from three different sites including GOYA, CAVALLI, and ISTS to demonstrate domain shift across sites. The sample tiles corresponding to each site, for example, GOYA sample tiles 605, CAVALLI sample tiles 610, and ISTS sample tiles 615, exhibit variations in appearance or distribution (e.g., in color, intensities, etc.). A balanced data distribution (e.g., in terms of tumor and normal tissue images) was established in these datasets, guaranteeing an equitable mix of normal and tumor tissue tiles. This balance facilitates avoiding any bias from imbalance labels during training and / or evaluating the model. A phase 2 CAVALLI and a phase 3 GOYA clinical trials were designated for training, and validation, respectively. CAVALLI dataset comprises 260 slides containing 60,664 normal and 58,151 tumor tiles. GOYA dataset includes 322 slides with 101,735 normal and 120,293 tumor tiles. A third dataset namely ISTS, a real-world DLBCL dataset that is independent from the CAVALLI dataset and GOYA dataset, and is taken from a different site, was used as a test dataset. The ISTS (independent site test set) dataset includes 1,351 slides with 245,898 normal and 412,073 tumor tiles.

[0071] Performance evaluation results of the classification model 410 or the modified Resnetl8 architecture based on the various normalization approaches were generated using theGOYA (validation) and ISTS (test) datasets as shown in Table I. It may be appreciated that IN exhibit excellent accuracy on the ISTS (test) dataset that is comparable to StainNorm.

[0072] Table 1 : Performance results of the classification model 410 using various normalization layers on the GOYA and ISTS dataset.

[0073] IN approach demonstrates a robust performance, marked by a macro accuracy of 76.9% on the GOYA dataset and an impressive 80.2% on the ISTS dataset. This robustness is even more pronounced when considering the false positive (FP) and false negative (FN) rates. IN maintains a relatively low number of false positives, registering 27704 FPs on ISTS. A low rate of FP is important as the analysis region in a downstream task (such as WSI classification) should contain tumor tiles for an accurate prediction.

[0074] Based on the results of the example implementation, StainNorm, although achieve slightly better macro accuracy (77.7% on GOYA and 80.9% on ISTS), does not exhibit the same level of precision in minimizing false positives with 49876 FPs on ISTS. StainNorm model (being twice in size, stain normalizing model and the machine learning model) demonstrates that its sophisticated approach does not necessarily translate to a proportional reduction in FPs.

[0075] FIG. 7 illustrates 2D embedding visualizations for the BN and IN normalization techniques in accordance with an example implementation of the present disclosure. The 2Dembeddings are generated using the t-Distributed Stochastic Neighbor Embedding (t-SNE) technique. t-SNE is a popular technique for dimensionality reduction and visualization of highdimensional data, particularly to visualize complex data, such as embeddings or feature representations, in a 2D or 3D space. The 2D embeddings for the BN and IN techniques are shown in a top image 705 and in a bottom image 710, respectively. Embeddings based on different datasets are represented by distinct colors, for example, reddish dots correspond to ISTS, greenish dots correspond to GOYA, and blueish dots correspond to CAVALLI datasets as indicated by a legend 715. A change in intensity of the dots color (brighter or lighter shades) corresponds to tumor and normal classes. The embeddings are extracted from the last layer of the modified ResNet architecture.

[0076] A comparative visualization of 2D embeddings based on the instance and batch normalization techniques reveals a stark difference in feature space organization for digital pathology applications. Instance normalization yields a more segregated and structured embedding, with clear demarcations between normal and tumor data points, indicating a potential for higher classification performance. This is particularly evident in the crescent moonlike distribution as shown in the bottom image 710, where the instance normalization manages to cluster similar samples together while keeping the different classes apart. Batch normalization produces a less distinct separation, with considerable overlap between classes. This suggests a possible compromise in the model’s ability to distinguish between pathological states (e.g., normal or tumor). The visual representation of instance normalization, therefore, suggests a more nuanced capture of the intricate patterns that facilitates the model to achieve accurate tissue classification.

[0077] The results intriguingly suggest that a straightforward IN technique provides a comparable accuracy against the more complex StainNorm. In addition, as compared to StainNorm, IN also achieves a greater reduction in false positives (a paramount concern in medical diagnostics). Therefore, IN can be used as a viable, efficient, and effective approach in digital pathology, to balancing accuracy and the need to minimize false positives. The findings advocate for a nuanced understanding of the trade-offs involved in normalization techniques, emphasizing that complexity does not always equate to superior performance, especially in the high-stakes field of medical diagnostics.

[0078] Thus, the techniques disclosed in the present disclosure can be utilized to develop deep learning models that are generalized or robust against varying data distributions or domain shifts. According to present disclosure, normalization techniques can be used to enhance the learning and generalization capabilities of deep learning (DL) networks, specifically in the realm of digital pathology. The present disclosure discloses seamless integration of the normalization techniques into DL model layers, yielding advantages such as obviating the need for access to target data, circumventing the need for model retraining, and avoiding the introduction of supplementary modules into the processing pipeline. It will be appreciated that techniques disclosed herein may be applied to models other than ResNet.

[0079] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer-program product tangibly embodied in a non-transitory machine- readable storage medium, including instructions configured to cause one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein.

[0080] The terms and expressions which have been employed are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention claimed. Thus, it should be understood that although the present invention as claimed has been specifically disclosed by embodiments and optional features, modification, and variation of the concepts herein disclosed may be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this invention as defined by the appended claims.

[0081] The description provides preferred exemplary embodiments only, and is not intended to limit the scope, applicability or configuration of the disclosure. Rather, the description of the preferred exemplary embodiments will provide those skilled in the art with an enabling description for implementing various embodiments. It is understood that various changes may bemade in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0082] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method comprising: accessing a digital pathology image; generating, for each portion of a set of portions of the digital pathology image, a label for the portion that predicts whether the portion depicts one or more tumor cells, wherein the label is generated by processing the portion using a trained deep learning model that includes multiple layers, wherein the multiple layers includes a first convolution layer, a second convolution layer, and a normalization layer that is between the first convolution layer and the second convolution layer; generating a result based on the labels for the set of portions; and outputting the result.

2. The computer-implemented method of claim 1, wherein the normalization layer is configured to perform an instance normalization.

3. The computer-implemented method of claim 1, wherein the normalization layer is configured to perform a weighted combination of an instance normalization and a layer normalization.

4. The computer-implemented method of claim 1, wherein the normalization layer is configured to perform a weighted combination of an instance normalization, a layer normalization, and a batch normalization.

5. The computer-implemented method of claim 1, wherein the multiple layers include a third convolution layer and a second normalization layer, and wherein one of the normalization layer and the second normalization layer is configured to perform an instance normalization and another of the normalization layer and the second normalization layer is configured to perform a different type of normalization.

6. The computer-implemented method of claim 5, wherein the different type of normalization includes layer normalization, batch normalization, and group normalization.

7. The computer-implemented method of claim 1, wherein the trained deep learning model includes a residual network (ResNet), a densely connected convolutional network (DenseNet), or a convolutional neural network (CNN).

8. A system comprising: one or more data processors; and a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform a set of operations including: accessing a digital pathology image; generating, for each portion of a set of portions of the digital pathology image, a label for the portion that predicts whether the portion depicts one or more tumor cells, wherein the label is generated by processing the portion using a trained deep learning model that includes multiple layers, wherein the multiple layers includes a first convolution layer, a second convolution layer, and a normalization layer that is between the first convolution layer and the second convolution layer; generating a result based on the labels for the set of portions; and outputting the result.

9. The system of claim 8, wherein the normalization layer is configured to perform an instance normalization.

10. The system of claim 8, wherein the normalization layer is configured to perform a weighted combination of an instance normalization and a layer normalization.

11. The system of claim 8, wherein the normalization layer is configured to perform a weighted combination of an instance normalization, a layer normalization, and a batch normalization.

12. The system of claim 8, wherein the multiple layers include a third convolution layer and a second normalization layer, and wherein one of the normalization layer and the second normalization layer is configured to perform an instance normalization and another of the normalization layer and the second normalization layer is configured to perform a different type of normalization.

13. The system of claim 12, wherein the different type of normalization includes layer normalization, batch normalization, and group normalization.

14. The system of claim 8, wherein the trained deep learning model includes a residual network (ResNet), a densely connected convolutional network (DenseNet), or a convolutional neural network (CNN).

15. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform a set of operations comprising: accessing a digital pathology image; generating, for each portion of a set of portions of the digital pathology image, a label for the portion that predicts whether the portion depicts one or more tumor cells, wherein the label is generated by processing the portion using a trained deep learning model that includes multiple layers, wherein the multiple layers includes a first convolution layer, a second convolution layer, and a normalization layer that is between the first convolution layer and the second convolution layer; generating a result based on the labels for the set of portions; and outputting the result.

16. The computer-program product of claim 15, wherein the normalization layer is configured to perform an instance normalization.

17. The computer-program product of claim 15, wherein the normalization layer is configured to perform a weighted combination of an instance normalization and a layer normalization.

18. The computer-program product of claim 15, wherein the normalization layer is configured to perform a weighted combination of an instance normalization, a layer normalization, and a batch normalization.

19. The computer-program product of claim 15, wherein the multiple layers include a third convolution layer and a second normalization layer, and wherein one of the normalization layer and the second normalization layer is configured to perform an instance normalization and another of the normalization layer and the second normalization layer is configured to perform a different type of normalization.

20. The computer-program product of claim 15, wherein the trained deep learning model includes a residual network (ResNet), a densely connected convolutional network (DenseNet), or a convolutional neural network (CNN).

Images