Correcting for differences in multi-scanners for digital pathology images using deep learning

Abstract

The present disclosure relates to techniques for converting digital pathology images obtained by different slide scanners into a common format for image analysis. In particular, aspects of the present disclosure relate to: obtaining a source image of a biological sample, the source image being generated from a first type of scanner; inputting a randomly generated noise vector and a latent feature vector from the source image as input data into a generator model; generating, by the generator model, a new image based on the input data; inputting the new image into a discriminator model; generating, by the discriminator model, a probability that the new image is real or fake; determining whether the new image is real or fake based on the generated probability; and outputting the new image when the image is real.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 068,585, filed August 21, 2020, which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] This disclosure relates to digital pathology, and more particularly to techniques for converting digital pathology images obtained from different slide scanners into a common format for image analysis. Background Technology

[0004] Digital pathology involves the interpretation of digitized images to accurately diagnose patients and guide treatment decisions. Whole-slide imaging (WSI) is an imaging modality used in digital pathology that scans a pre-selected region of a tissue sample or an entire slide (e.g., a histopathology or cytopathology slide) into a digital image. The digitization process comprises four consecutive parts: image acquisition (scanning), storage, editing, and image display. Image acquisition is performed by a whole-slide scanner, which typically has a light source, a slide stage, objectives, and a high-resolution camera for image capture. The whole-slide scanner captures images of tissue sections piece by piece or in line scans. Multiple images (pieces or lines, respectively) are captured and digitally combined (“stitched”) to generate a digital image of the pre-selected region or the entire slide. When paired with slide staining techniques, WSI can be categorized as brightfield, fluorescence, and multispectral. Some scanners can accommodate more than one mode, such as supporting both brightfield and fluorescence scanning. Brightfield scanning simulates standard brightfield microscopy and is a cost-effective method. Fluorescence scanning is similar to fluorescence microscopy and is used for digitizing fluorescently labeled slides (e.g., fluorescence immunohistochemistry (IHC), fluorescence in situ hybridization, etc.). Multispectral imaging captures spectral information across the entire spectrum and can be applied to both brightfield and fluorescence settings.

[0005] Many WSI systems include image viewing software that can be installed locally on the user's computer. Other vendors offer this functionality as part of a larger software suite residing on a web server, allowing users to view whole-slide images on their own devices via a network connection. For users who wish to apply image analysis algorithms to whole-slide images, some vendor-supplied image viewing software packages algorithms that can detect cells, calculate positive stains, perform region segmentation, or perform nuclear segmentation in hematoxylin and eosin (H&E) images. For users seeking more complex or specialized image analysis algorithms than those offered by their scanner vendors, third-party vendors offer numerous software solutions with a wide range of functionalities. These image analysis algorithms can often be integrated into departmental workflows, providing on-demand image analysis in conjunction with whole-slide viewing. However, most image analysis algorithms are trained on images from a specific whole-slide scanner (i.e., developed for a specific whole-slide scanner) and may therefore only work on digital images with specific characteristics from those whole-slide scanners. The characteristics of some digital images generated by different types or models of digital slide scanners may be incompatible with image analysis algorithms. Therefore, it is desirable for image analysis algorithms to be scanner-independent (to operate on images obtained by any type of scanner). Summary of the Invention

[0006] In various embodiments, a computer-implemented method is provided, comprising: obtaining a source image of a biological sample, wherein the source image is generated from a first type of scanner; inputting a randomly generated noise vector and a latent feature vector from the source image as input data into a generator model; generating a new image based on the input data using the generator model; inputting the new image into a discriminator model; generating a probability that the new image is true or false using the discriminator model, wherein true indicates that the new image has characteristics similar to those of a target image, and false indicates that the new image does not have characteristics similar to those of the target image, and wherein the characteristics of the target image are associated with a second type of scanner different from the first type of scanner; determining whether the new image is true or false based on the generated probability; and outputting the new image when the image is true.

[0007] In some embodiments, the biological sample is secured on a pathological slide, the first type of scanner is a first type of whole slide imaging scanner, and the second type of scanner is a second type of whole slide imaging scanner.

[0008] In some embodiments, the computer-implemented method further includes: inputting a new image into an image analysis model, wherein the image analysis model includes multiple model parameters learned using a training dataset comprising images obtained from a scanner of the same type as the second type of scanner; analyzing the new image using the image analysis model; generating analysis results based on the analysis of the new image using the image analysis model; and outputting the analysis results.

[0009] In some embodiments, the image analysis model is not trained on images obtained from a scanner of the same type as the first type of scanner.

[0010] In some embodiments, the computer-implemented method further includes training an image analysis model using a training dataset that includes new images.

[0011] In some embodiments, the GAN model includes using a training dataset comprising one or more pairs of images, wherein each pair of images in the one or more pairs of images comprises a first image generated by a first type of scanner and a second image generated by a second type of scanner; and wherein a discriminator model is trained based on minimizing a first loss function to maximize the probability of the training dataset and minimizing a second loss function to minimize the probability of the generated image sampled from the generator model and to maximize the probability assigned to the generated image by the discriminator model, and multiple model parameters are learned using the training dataset.

[0012] In some embodiments, the computer-implemented method further includes a user determining a diagnosis for the subject based on the analysis results.

[0013] In some embodiments, the computer-implemented method further includes administering the compound as treatment by a user based on (i) analysis results and / or (iii) a diagnosis of the subject.

[0014] In some embodiments, a system is provided that includes one or more data processors and a nontransitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of the methods disclosed herein.

[0015] In some embodiments, a computer program product is provided, tangibly embodied in a non-transitory machine-readable storage medium, and includes instructions configured to cause one or more data processors to perform some or all of the methods disclosed herein.

[0016] Some embodiments of this disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein. Some embodiments of this disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein.

[0017] The terms and expressions used are descriptive rather than restrictive, and their use is not intended to exclude any equivalents of the features shown and described or portions thereof; however, it should be recognized that various modifications are possible within the scope of the claimed invention. Therefore, it should be understood that although the claimed invention has been specifically disclosed by way of examples and optional features, those skilled in the art can employ modifications and variations of the concepts disclosed herein, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims. Attached Figure Description

[0018] The aspects and features of various embodiments will become more apparent from the examples described with reference to the accompanying drawings, in which:

[0019] Figure 1 illustrates a digital pathology workflow according to various embodiments;

[0020] Figure 2 shows a comparison of digital images obtained from two different digital pathology scanners according to various embodiments;

[0021] Figure 3 illustrates alternative digital pathology workflows according to various embodiments;

[0022] Figure 4 illustrates an instance computing environment for segmenting a target object according to various embodiments;

[0023] Figure 5 illustrates generative adversarial networks (GANs) according to various embodiments;

[0024] Figure 6 illustrates examples of paired training images from a first digital pathology slide scanner and a second digital pathology slide scanner with registration, according to various embodiments.

[0025] Figure 7 illustrates, according to various embodiments, the process for converting a source image set obtained from a first digital image scanner into a newly generated image set having characteristics similar to the target image set;

[0026] Figure 8 illustrates examples of digital pathological images stained for different biomarkers according to various embodiments; and

[0027] Figure 9 shows additional examples of digital pathological images stained for different biomarkers and their characteristics according to various embodiments. Detailed Implementation

[0028] While certain embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of protection. The devices, methods, and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions, and variations may be made to the form of the exemplary methods and systems described herein without departing from the scope of protection.

[0029] I. Overview

[0030] For example, the assessment of tissue changes caused by disease can be performed by examining thin tissue sections. Tissue samples can be sliced ​​to obtain a series of sections (e.g., 4-5 μm sections), and each tissue section can be stained with different staining agents or markers to express different characteristics of the tissue. Each section can be mounted on a slide and scanned to generate digital images for analysis using computerized digital pathology image analysis algorithms. Various types or models of digital pathology slide scanners can be used to scan and generate digital images. For example, digital images of pathology slides can be scanned and generated using the VENTANA® DP 200 slide scanner, the VENTANA iScan® HT slide scanner, the Aperio AT2 slide scanner, or other types of slide scanners. The tissue within the digital images can be manually assessed using image viewer software, or automatically analyzed using image analysis algorithms for the purpose of detecting and classifying biological objects.

[0031] In digital pathology solutions, image analysis workflows can be established to automatically detect or classify target biological objects, such as positive or negative tumor cells. Figure 1 shows an example of a digital pathology solution workflow 100. The digital pathology solution workflow 100 includes obtaining a tissue slide at box 105, scanning a pre-selected area or the entire tissue slide to obtain a digital image at box 110 using a digital image scanner (e.g., a WSI scanner), performing image analysis on the digital image using one or more image analysis algorithms at box 115, and scoring the target object based on the image analysis (e.g., quantitative or semi-quantitative scoring, such as positive, negative, intermediate, weak, etc.).

[0032] In many cases, due to potential changes in hardware (e.g., digital scanners) or staining protocols, modifications to image analysis algorithms used in the digital pathology solution workflow described with respect to Figure 1 are necessary. For example, the VENTANA® DP 200 slide scanner for high quality and high resolution has gradually replaced the previous VENTANA iScan® HT slide scanner. Figure 2 shows examples of tissue slides scanned using the VENTANA iScan® HT (A) and VENTANA® DP200 (B) full slide scanners, respectively. Evidence suggests that the images obtained from these two scanners are distinct (i.e., images obtained from one scanner possess different characteristics (e.g., improved contrast and / or resolution) compared to images obtained from the other). Several conventional image analysis algorithms developed based on images obtained using the VENTANA iScan® HT scanner may not achieve optimal performance when applied to processing images obtained from the VENTANA® DP 200 scanner. For example, experiments have shown that when the original algorithm developed for the VENTANA iScan® HT is applied directly to images acquired by the VENTANA® DP 200 scanner, the algorithm's performance may be unsatisfactory. Therefore, continuous updates or modifications to the image analysis algorithm are necessary, requiring additional resources, cost, and time. Typically, developing an image analysis algorithm for a new scanner can take more than six months, and the impact of the problem can be exacerbated when multiple image analysis algorithms are implemented with a new scanner. It should be understood that while this problem was initially discovered on proprietary scanners, many digital pathology scanners and solutions workflows in the digital pathology industry have encountered the same issue.

[0033] To overcome these and other limitations, this paper discloses a deep learning-based generative model called a Generative Adversarial Network (GAN) to transform a source image set obtained from a first digital image scanner (e.g., a VENTANA® DP 200 scanner) into a newly generated image set with properties similar to a target image set obtained from a second image scanner (e.g., a VENTANA iScan® HT). The GAN can learn to estimate two distributions (e.g., properties from the source image set and properties from the target image set), which can be used to transform instances from one distribution (e.g., the source image set) to another distribution (e.g., the target image set). Once the GAN is trained to transform the source image set obtained from the first digital image scanner into a newly generated image set with properties similar to the target image set, the newly generated image set can be analyzed using an imaging analysis algorithm trained on images from the second digital image scanner, without the need to redevelop the image analysis algorithm, and with minimal cost and time.

[0034] Figure 3 illustrates an example of a digital pathology solution workflow 300 according to aspects of this disclosure. The digital pathology solution workflow 300 includes obtaining a tissue slide at box 305, and scanning a pre-selected region or the entire tissue slide using a first digital image scanner (images from the first digital image scanner are not used to train the image analysis algorithm) to generate a source digital image set at box 310. Because the images from the first digital image scanner are not used to train the image analysis algorithm, the source digital image set is obtained at box 315 and input into a deep learning-based generative model at box 320 to transform the source digital image set into a newly generated image set at box 325, which is similar in characteristics to a target image set obtainable from a second digital image scanner (images from the second digital image scanner are used to train the image analysis algorithm). At box 330, image analysis is performed on the newly generated image set using the image analysis algorithm, and at box 335, the target object is scored based on the image analysis (e.g., quantitative or semi-quantitative scoring, such as positive, negative, moderate, weak, etc.). Therefore, image differences between the two digital image scanners can be corrected, and conventional image analysis algorithms can be applied to images obtained from scanners that have never been used to train conventional image analysis algorithms.

[0035] One illustrative embodiment of this disclosure relates to a method comprising: obtaining a source image of a biological sample, wherein the source image is generated from a first type of scanner; inputting a randomly generated noise vector and a latent feature vector from the source image as input data into a generator model; generating a new image based on the input data using the generator model; inputting the new image into a discriminator model; generating a probability that the new image is true or false using the discriminator model, wherein true indicates that the new image has characteristics similar to those of a target image, and false indicates that the new image does not have characteristics similar to those of the target image, and wherein the characteristics of the target image are associated with a second type of scanner different from the first type of scanner; determining whether the new image is true or false based on the generated probability; and outputting the new image when the image is true.

[0036] Advantageously, these techniques can make computerized digital image analysis algorithms scanner-independent by converting digital images produced by different digital scanners into images that can be analyzed using existing computerized digital image analysis algorithms, and correct for image variations in images obtained from different imaging sites. These techniques can also be used for the future algorithm development of any next-generation scanner, so that images scanned by other scanners can be converted and used as training data for the next-generation scanner. Furthermore, these techniques can be used to transfer data from different sites to correct for variations caused by pre-analysis conditions, which is one of the major challenges in image analysis algorithm development.

[0037] II. Definition

[0038] As used in this article, when an action is “based on” something, it means that the action is based at least partially on at least a part of something.

[0039] As used herein, the terms “substantially,” “about,” and “approximately” are defined as being substantially, but not necessarily entirely, as specified (and include being entirely specified), as understood by one of ordinary skill in the art. In any disclosed embodiment, the terms “substantially,” “about,” or “approximately” may be replaced with “within [a certain percentage]” for the specified meaning, where percentages include 0.1%, 1%, 5%, and 10%.

[0040] As used herein, the terms “sample,” “biological sample,” or “tissue sample” refer to any sample obtained from any organism, including viruses, that includes biomolecules such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof. Examples of other organisms include mammals (such as humans; mammals such as cats, dogs, horses, cattle, and pigs; and laboratory animals such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections and needle biopsies of tissues), cell samples (such as cytological smears, such as cervical smears or blood smears, or cell samples obtained through microdissection), or cell fractions, fragments, or organelles (such as those obtained by lysing cells and separating their components by centrifugation or other methods). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical or needle biopsy), nipple aspiration, cerumen, breast milk, vaginal secretions, saliva, swabs (such as oral swabs), or any material containing biomolecules derived from the first biological sample. In some embodiments, the term "biological sample" as used herein refers to a sample prepared from a tumor or a portion thereof obtained from a subject (such as a homogenized or liquefied sample).

[0041] III. Technologies for Digital Pathology Image Conversion

[0042] Computerized digital image analysis algorithms can be used to analyze pathological images obtained from a specific digital pathology slide scanner (a specific type of scanner, such as a specific scanner from a particular manufacturer or a specific scanner model). In this case, using an image analysis algorithm trained on images from a specific digital pathology slide scanner to analyze digital pathology images from different digital pathology slide scanners may not achieve the desired effect or accuracy. According to various aspects of this disclosure, digital pathology images obtained from different slide scanners are converted into images with similar characteristics to images from a specific digital pathology slide scanner, making it possible to use image analysis algorithms to achieve the desired effect or accuracy.

[0043] Figure 4 illustrates an example computing environment 400 according to various embodiments for converting a source digital image set obtained from a first digital image scanner into a newly generated image set having characteristics similar to a target image set obtainable from a second digital image scanner. As shown in Figure 4, in this example, the conversion of the source image set performed by the computing environment 400 includes several stages: an image acquisition stage 405, a model training stage 410, a conversion stage 415, and an analysis stage 420. The image acquisition stage 405 includes a digital image scanner 425 for obtaining a source digital image set 430 and a target digital image set 435 from a pre-selected region or a whole biological sample slide (e.g., a tissue slide). The digital image scanner 425 includes a first type of digital image scanner for obtaining the source digital image set 430 and a second type of digital image scanner for obtaining the target digital image set 435.

[0044] Model Training Phase 410 constructs and trains one or more models 440a to 440n (“n” represents any natural number) that will be used by other phases (which may be referred to individually as model 440 or collectively as model 440 in this document). Model 440 may be a machine learning (“ML”) model, such as a convolutional neural network (“CNN”), a starter neural network, a residual neural network (“ResNet”), U-Net, V-Net, a single-shot multi-box detector (“SSD”) network, a recurrent neural network (“RNN”), a rectified linear unit (“ReLU”), a long short-term memory (“LSTM”) model, a gated recurrent unit (“GRUs”) model, or any combination thereof. In various embodiments, model 440 is a generative model capable of learning any type of data distribution using unsupervised learning, such as generative adversarial networks (“GANs”), deep convolutional generative adversarial networks (“DCGANs”), variational autoencoders (VAEs), hidden Markov models (“HMMs”), Gaussian mixture models, Boltzmann machines, etc., or combinations of one or more such techniques—for example, VAE-GAN. Computational environment 400 can employ the same type of model or different types of models to transform source images into generated images. In some cases, model 440 is a GAN constructed with a loss function that attempts to classify whether the output image is real or fake, while training the generative model to minimize this loss.

[0045] In the exemplary embodiment shown in Figure 5, the model is a conditional GAN ​​(“CGAN”) 500, which is an extension of the GAN model and generates images with specific conditions or attributes. CGAN learns a structured loss, which penalizes the joint configuration of the output. Referring to Figure 5, CGAN 500 includes a generator 510 and a discriminator 515. The generator 510 is a neural network (e.g., a CNN) that takes a randomly generated noise vector 520 and a latent feature vector (or a one-dimensional vector) 525 (e.g., the conditions of the source image in the current example) as input data and feedback from the discriminator 515, and generates a new image 530 that is as close as possible to the real target image 535. In some cases, the generator 510 uses a “U-Net” architecture consisting of an encoder-decoder network, in which a skip layer is added between the downsampling and upsampling layers. The discriminator 515 is a neural network (e.g., a CNN) configured as a classifier to determine whether the generated image 530 from the generator 510 is a real image or a fake image. In some cases, the discriminator uses a "PatchGAN" architecture patch by patch. The latent feature vector 525 or conditional source is a source image or source image set 540 (e.g., images from a first digital scanner) whose encoded category (e.g., having...) n (A tissue sample containing a biomarker or stain) or a specific set of features expected from a source image 540. A randomly generated noise vector 520 can be generated from a Gaussian distribution, and the vector space can include latent or hidden variables that are important to the domain but not directly observable. The latent feature vector 525 and the random noise vector 520 can be combined as input 545 to the generator 510.

[0046] Generator 510 takes combined input 545 and generates image 530 based on latent feature vector 525 and random noise vector 520 in the problem domain (i.e., the feature domain associated with target image 535). Discriminator 515 performs conditional image classification by taking target image 535 (e.g., an image from a second digital scanner) and generated image 530 as input and predicts 550 the probability that generated image 530 is a true translation or a false translation of target image 535. The output of discriminator 515 depends on the size of generated image 530 but can be a single value or a square activation map of values. Each value is the probability that a patch in generated image 530 is true. If needed, these values ​​can be averaged to give an overall probability or classification score. The loss function of generator 510 and discriminator 515 is highly dependent on how well discriminator 515 performs its prediction of the probability that generated image 530 is a true translation or a false translation of target image 535. After sufficient training, the generator 510 will improve, and the generated image 530 will begin to look more like the target image 535. Training of GAN 500 is complete when the generated image 530 possesses characteristics similar to the target image 535, making it impossible for the discriminator to distinguish between them. Once trained, the source image set obtained from the first digital image scanner can be input into GAN 500 to transform it into a newly generated image set with characteristics similar to the target image set obtained from the second digital image scanner. The newly generated image set can then be analyzed using currently available computerized digital pathology image analysis algorithms.

[0047] Referring back to Figure 4, to train model 440 in this example, sample 445 is generated by: acquiring digital images (source digital image set 430 and target digital image set 435); dividing the images into paired image subsets 445a (at least one source image and target image pair) for training (e.g., 90%) and paired image subsets 445b for validation (e.g., 10%); preprocessing paired image subsets 445a and 445b; expanding paired image subset 445a; and, in some cases, annotating paired image subset 445a with label 450. Paired image subset 445a is acquired by one or more imaging modalities (e.g., a WSI scanner). In some cases, paired image subset 445a is acquired by a data storage structure such as a database, an image system (e.g., a digital image scanner 425), etc., associated with one or more imaging modalities. Each image depicts a biological sample, such as tissue.

[0048] Segmentation can be performed randomly (e.g., 90 / 10%, 80% / 20%, or 70 / 30%), or it can be performed using more sophisticated validation techniques (such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, nested cross-validation, etc.) to minimize sampling bias and overfitting. Preprocessing may include cropping images so that each image contains only a single target object. In some cases, preprocessing may further include normalization or standardization to place all features on the same scale (e.g., the same size scale, or the same color scale or color saturation scale). In some cases, the image size is adjusted using a predetermined minimum size (width or height) of pixels (e.g., 2500 pixels) or a predetermined maximum size (width or height) of pixels (e.g., 3000 pixels) while preserving the original aspect ratio.

[0049] For example, multiple patch images from a first scanner and a second scanner can be prepared as one or more paired image subsets for training data. Preparation of paired images may include acquiring slides of biological samples, such as IHC slides expressing one or more biomarkers, such as CD34-aSMA, FAP / PanCK, perforin / CD3, Ki67 / CD8, FoxP3, PD1, etc., or any combination thereof. Each slide is scanned using the first and second scanners to obtain a full slide image. The full slide image can then be cropped to a predetermined size (e.g., 128x128) to form multiple patch images. Pairs of patch images from the first and second scanners are selected and registered (aligning two or more images of the same object or scene). Registration may include designating one image from one scanner as a reference image, also known as a fixed image, and applying a geometric transformation or local displacement to another image from the other scanner such that the other image is aligned with the reference image. This process produces one or more paired image subsets for training data.

[0050] Figure 6 illustrates an example of paired training images 600 from a first and a second digital slide scanner with registration, according to some aspects of this disclosure. Referring to Figure 6, the images in the left column 610 are source images from the first digital slide scanner (e.g., VENTANA® DP 200), and the images in the right column 620 are target images from the second digital slide scanner (e.g., VENTANA iScan® HT). The first row 330 is a first training image pair, the second row 340 is a second training image pair, and the third row 350 is a third training image pair. The source and target images may each have a patch size of 128x128 pixels or other sizes. The 128x128 pixel patches can be input into a GAN or CGAN to train a deep learning network.

[0051] Referring back to Figure 4, augmentation can be used to artificially expand the size of the paired image subset 445a by creating modified versions of the images in the dataset. Image data augmentation can be performed by creating image transformation versions in the dataset that belong to the same category as the original images. Transformations include a range of operations from the realm of image manipulation, such as shifting, flipping, scaling, etc. In some cases, these operations include random erasing, shifting, brightening, rotating, Gaussian blurring, and / or elastic transformations to ensure that model 440 can be performed in environments outside those available from the paired image subset 445a.

[0052] The training process of model 440 includes selecting hyperparameters of model 440 and performing iterative operations to input images from paired image subsets 445a into model 440 to find a set of model parameters (e.g., weights and / or biases) that minimize one or more loss functions or error functions of model 440 (e.g., a first loss function for training the discriminator to maximize the probability of the images in the training data, and a second loss function for training the discriminator to minimize the probability of generated images sampled from the generator and training the generator to maximize the probability of the discriminator assigning itself to generated images). Hyperparameters are settings that can be tuned or optimized to control the behavior of model 440. Most models explicitly define hyperparameters that control different aspects of the model, such as memory or execution cost. However, additional hyperparameters can be defined to adapt the model to a specific scenario. For example, hyperparameters may include the number of hidden units in the model, the learning rate of the model, the kernel width of the model, or the number of convolutional kernels. Each iteration of training may involve finding a set of model parameters (configured with a defined set of hyperparameters) for model 440 such that the value of the loss or error function using that set of model parameters is less than the value of the loss or error function using a different set of model parameters in previous iterations. A loss or error function can be constructed to measure the difference between the output inferred using model 440 and the ground truth target image using label 450.

[0053] Once the model parameter set is identified, model 440 is trained and validated using paired image subsets 445b (test or validation datasets). This validation process involves iterative operations that use validation techniques (such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, nested cross-validation, etc.) to input images from paired image subsets 445b into model 440 to tune hyperparameters and ultimately find the optimal set of hyperparameters. Once the optimal set of hyperparameters is obtained, a reserved test set of images from image subset 445b is input into model 445 to obtain the output (in this example, generated images with similar characteristics to the target image), and the output is evaluated relative to the ground truth target image using relevant techniques (such as the Bland-Altman method and Spearman rank correlation coefficient) and performance metrics (such as error, accuracy, precision, recall, receiver operating characteristic curve (ROC), etc.).

[0054] It should be understood that other training / validation mechanisms are also expected and can be implemented within the computational environment 400. For example, model 440 can be trained and hyperparameters can be tuned on images from paired image subsets 445a, and images from paired image subsets 445b can be used solely for testing and evaluating the performance of model 440. Furthermore, although the training mechanisms described herein focus on training a new model 440, these mechanisms can also be used to fine-tune an existing model 440 trained on other datasets. For example, in some cases, model 440 may have been pre-trained using images of other objects or biological structures, or images of slices from other subjects or studies (e.g., human or rodent experiments). In those cases, model 440 can be used for transfer learning and retrained / validated using images 430 / 435.

[0055] The model training phase 410 outputs a training model comprising one or more trained conversion models 460 and optionally one or more image analysis models 465. In some cases, a first model 460a is trained to process a source image 430 of a biological sample. The source image 430 is generated from a first type of scanner, such as a whole-slide imaging scanner. The source image 430 is obtained by a conversion controller 470 within the conversion phase 415. The conversion controller 470 includes program instructions for using one or more trained conversion models 460 to convert the source image 430 into a new image 475 having characteristics of a target image. The characteristics of the target image are associated with a second type of scanner, different from the first type of scanner. The transformation includes: (i) inputting a randomly generated noise vector and a latent feature vector from the source image 430 as input data into a generator model (part of the transformation model 460); (ii) generating a new image 475 through the generator model; (iii) inputting the new image 475 into a discriminator model (another part of the model 460); and generating a probability (e.g., a number between 1 and 0) for the new image 475 to be true or false through the discriminator model, where true indicates that the image has characteristics similar to those of the target image, and false indicates that the image does not have characteristics similar to those of the target image.

[0056] In some cases, during the analysis phase 420, a new image 475 is transferred to an analysis controller 480. The analysis controller 480 includes program instructions for analyzing the biological sample within the new image 475 using one or more image analysis models 465; and analysis results 485 based on the analysis output. In some cases, the one or more image analysis models 465 are one or more imaging analysis algorithms (e.g., conventional image analysis algorithms) trained on images obtained from scanners of the same type as the second type of scanner and associated with characteristics of the target image, and / or trained on images obtained from scanners of a different type but having characteristics substantially similar to those of the target image. Therefore, the techniques described herein can use a pre-existing imaging analysis algorithm 480 to process the transformed source image 430 (i.e., the new image 475) without developing a new image analysis algorithm. Analyzing the biological sample within the new image 475 may include extracting measurements based on regions within the new image 475, one or more cells within the new image 475, and / or objects in the new image 475 other than cells. Region-based measurements include the most basic assessments, such as quantifying regions (two-dimensional) of a particular stain (e.g., chemical or IHC staining), regions of fat vacuoles, or other events present on a slide. Cell-based measurements aim to identify and enumerate objects, such as cells. This identification of individual cells enables subsequent subcellular compartment assessment. Finally, algorithms can be used to assess events or objects present on tissue sections that may not consist of individual cells. In some cases, pre-existing imaging analysis algorithms are configured to locate cellular or subcellular structures and provide quantitative representations of cell staining, morphology, and / or architecture, which can ultimately be used to support diagnosis and prediction.

[0057] Although not explicitly shown, it should be understood that the computing environment 400 may also include a developer device associated with the developer. Communication from the developer device to components of the computing environment 400 may indicate the type of input images to be used for the model, the number and type of models to be used, the hyperparameters of each model (e.g., learning rate and number of hidden layers), the formatting of data requests, the training data to be used (e.g., and the method of obtaining access to the training data), and the validation techniques to be used and / or the configuration of the controller processing.

[0058] Figure 7 illustrates a flowchart of an exemplary process 700 for converting a source image (e.g., a source image from a set of source images to be processed) obtained from a first digital image scanner into a new image (e.g., a new image from a new set of images to be generated) having similar properties to the target image. Process 700 can be performed using one or more computing systems, models, and networks described herein with respect to Figures 1 through 5. The process begins at box 705, where a source image of a biological sample is obtained. The source image is generated from a first type of scanner (e.g., a specific scanner or a specific scanner model from a particular manufacturer). At box 710, a randomly generated noise vector and a latent feature vector from the source image are input as input data to a generator model. At box 715, the generator model generates a new image based on the input data. At box 720, the new image is input to a discriminator model. At box 725, the discriminator model generates a probability that the new image is real or fake. A "true" value indicates that the new image has characteristics similar to those of the target image (e.g., intensity, contrast, resolution, morphological boundaries / shape, etc.), while a "false" value indicates that the new image does not have characteristics similar to those of the target image. The discriminator model is looking at the probability distribution of characteristics of the new image and comparing it to the probability distribution of characteristics of the target image learned from normalized sample images in the training dataset to predict whether the new image has a higher probability of being a target image, thus classifying the new image as a true image; otherwise, the discriminator assigns the new image as a false image. The similarity between the probability distributions of characteristics of the target image and the new image can be determined using cross-correlation, Bhattacharyya distance, or other mathematical algorithms (such as the mean squared error between the target image and the new image). The characteristics of the target image are associated with a second type of scanner, different from the first type of scanner. At box 730, the new image is determined to be true or false based on the generated probabilities. At box 735, when the image is true, a new image with a true label is output. At step 740, when the image is false, a new image with a false label is output.

[0059] In some cases, the generator model and discriminator model are part of a GAN model. A GAN model includes multiple model parameters learned using a training dataset comprising one or more pairs of images. Each pair of images within the one or more pairs of images includes a first image generated by a first type of scanner and a second image generated by a second type of scanner. In some cases, the discriminator model is trained by minimizing a first loss function to maximize the probabilities of the training dataset, and by minimizing a second loss function to minimize the probability of generated images sampled from the generator model, and the generator model is trained to maximize the probabilities assigned to generated images by the discriminator model; these multiple model parameters are learned using the training dataset.

[0060] At box 745, an action is performed using the new image output at box 735. In some cases, the action involves inputting the new image into an image analysis model. The image analysis model includes multiple model parameters learned using a training dataset comprising images obtained from a scanner of the same type as the second type of scanner. The action further includes: analyzing the new image using the image analysis model; generating analysis results based on the analysis of the new image using the image analysis model; and outputting the analysis results. For example, a second digital image scanner (e.g., iScanHT) can be used to train the image analysis model to detect specific markers (CD8, Ki67, etc.). Now, a slide has been scanned using a first digital image scanner (e.g., DP200), and the obtained image has a different feature profile (e.g., a different color and / or resolution profile) compared to the image obtained from the second digital image scanner (e.g., iScanHT). Using a GAN model, the image scanned by the first digital image scanner (e.g., DP200) can be transformed into an image with similar characteristics to the image scanned by the second digital image scanner (e.g., iScanHT). Therefore, image analysis models (such as those detecting CD8, Ki67, PanCk, CD3, etc.) can take new images, converted into profiles with similar characteristics to those scanned by a second digital image scanner (e.g., iScanHT), as input, and the image analysis model does not need to be retrained. The image analysis model may not be trained on images obtained from the same type of scanner as the first scanner (e.g., DP200). In other cases, the action involves training the image analysis model using a training dataset that includes the new images. Thus, using the new images generated by GAN, different labels can be classified without changing the image analysis model, and images can be obtained from different types of scanners. Subsequently, the user can determine a diagnosis for the subject based on the analysis results. The user can administer treatment using the compound based on (i) the analysis results and / or (iii) the diagnosis of the subject.

[0061] The method disclosed herein enables digital image analysis algorithms to be scanner-independent—digital pathology images generated by any digital pathology slide scanner can be converted into images suitable for analysis by digital image analysis algorithms. By converting images scanned by other scanners into images that can be used as training data paired with images generated by new scanners, the disclosed method is applicable to the development of image analysis algorithms for future generations of scanners. Furthermore, the disclosed method can be used to transfer data from different imaging sites located in different geographical areas to correct for image variations, such as those caused by pre-analysis conditions.

[0062] Figure 8 illustrates examples of digital pathology images stained for different biomarkers according to some aspects of this disclosure. These images are digital pathology images of four slides. The first row 820 includes the source image, generated image, and target image of the first slide. The second row 821 includes the source image, generated image, and target image of the second slide. The third row 822 includes the source image, generated image, and target image of the third slide. The fourth row 823 includes the source image, generated image, and target image of the fourth slide. Each row shows a digital pathology image of a slide that has been stained to display different biomarkers.

[0063] The source images of the four slides in the first column 810 are obtained from a first digital pathology slide scanner. The source images may not be suitable for analysis using available computerized digital pathology image analysis algorithms. The target images in the third column 812 are digital pathology images of the same four slides and can be obtained from a second digital pathology slide scanner. The target images in the third column 812 (e.g., the desired images) may be suitable for analysis using available computerized digital pathology image analysis algorithms. Aspects of this disclosure enable the conversion of source digital pathology images (in the first column 810) into images with the characteristics of the target digital pathology image (in the third column 812).

[0064] According to various aspects of this disclosure, source digital pathology images can be input into a trained GAN, and the trained GAN can output newly generated images with characteristics suitable for analysis using existing computerized digital pathology image analysis algorithms. Referring to Figure 8, the images in the second column 811 are new images generated by the trained GAN using the source images in the first column 810 as input. As can be seen from Figure 8, the generated images in the second column 811 have similar characteristics to the target images in the third column 812. Therefore, the same computerized digital pathology image analysis algorithms that can be used to analyze the digital pathology images in the third column 812 can be used to analyze the generated digital pathology images in the second column 811.

[0065] Figure 9 illustrates additional examples of digital pathology images stained for different biomarkers according to some aspects of this disclosure. These images are digital pathology images of two slides. The first row 920 includes a source image, a generated image, and a target image of the first slide. The second row 921 includes a source image, a generated image, and a target image of the second slide. The source images of the two slides in the first column 910 are obtained from a first digital pathology slide scanner. The source images may not be suitable for analysis using available computerized digital pathology image analysis algorithms. The target image in the third column 912 is a digital pathology image of the same two slides and can be obtained from a second digital pathology slide scanner. The target image in the third column 912 (e.g., the desired image) may be suitable for analysis using available computerized digital pathology image analysis algorithms. Aspects of this disclosure enable the conversion of a source digital pathology image (in the first column 910) into an image with the characteristics of a target digital pathology image (in the third column 912).

[0066] According to various aspects of this disclosure, source digital pathology images can be input into a trained GAN, and the trained GAN can output a new image with characteristics suitable for analysis using existing computerized digital pathology image analysis algorithms. Referring to Figure 9, the image in the second column 911 is a new image generated by the trained GAN using the source image in the first column 910 as input. As can be seen from Figure 9, the generated image in the second column 911 has similar characteristics to the target image in the third column 912 (see shaded dots indicating similar characteristics in similar locations). Therefore, the same computerized digital pathology image analysis algorithm that can be used to analyze the digital pathology image in the third column 912 can be used to analyze the generated digital pathology image in the second column 911.

[0067] V. Example

[0068] Example 1. Using deep learning to correct differences in multiple scanners for digital pathology images

[0069] A CGAN was developed to convert images of six different biomarker expressions (DAB, multiple bright-field IHC) acquired from a newer scanner (VENTANA DP200) into new, high-quality synthetic images with image characteristics similar to those obtained using a previous-generation scanner (VENTANA iScanHT). 12,740 images or 6,370 paired images of 128x128 patch size were used as paired iScanHT / DP200 images for training, consisting of the following biomarker expressions: CD34-aSMA (DAB / red), FAP / PanCK (yellow / purple), perforin / CD3 (DAB / red), Ki67 / CD8 (yellow / purple), FoxP3 (DAB), and PD1 (DAB). Both the iScanHT and DP200 scanners were used to scan the same tissue slide; however, patch images were selected and registered to ensure that paired images were located within the same tissue section.

[0070] Visual evaluation showed that the input DP200 images, after being transformed into output iScanHT images, possessed comparable image properties to the target images in images of different biomarkers. When the original iScanHT algorithm was applied to both the target and generated iScanHT images, the evaluation of the detected tumor cell counts between the output and target images resulted in consistency correlation coefficients (CCCs) of lin for PD1, FoxP3, Ki67CD8, FAP / PanCK, CD34-aSMA, and perforin / CD3 test images of 0.86, 0.93, 0.95, 0.82, 0.80, and 0.97, respectively. This demonstrates the feasibility of compensating for differences across multiple scanners and shows the ability to apply algorithms such as the traditional iScanHT algorithm to transformed DP200 images without developing new image analysis algorithms. This image-to-image translation method has the potential to generate large datasets for the development of future algorithms for any next-generation scanner, allowing the transformation of images scanned by other scanners and their use as training data for new scanners.

[0071] VI. Other Precautions

[0072] Some embodiments of this disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein. Some embodiments of this disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein.

[0073] The terms and expressions used are descriptive rather than restrictive, and their use is not intended to exclude any equivalents of the features shown and described or portions thereof; however, it should be recognized that various modifications are possible within the scope of the claimed invention. Therefore, it should be understood that although the claimed invention has been specifically disclosed by way of examples and optional features, those skilled in the art can employ modifications and variations of the concepts disclosed herein, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims.

[0074] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of preferred exemplary embodiments will provide those skilled in the art with a feasible description for implementing various embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the spirit and scope set forth in the appended claims.

[0075] Specific details are set forth in the following description to provide a thorough understanding of the embodiments. However, it should be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as parts in block diagram form to avoid obscuring the embodiments with unnecessary details. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments.

Claims

1. A method for image analysis, comprising: Obtain a source image of a biological sample, wherein the source image is generated from a first type of scanner; Randomly generated noise vectors and latent feature vectors from the source images are input as input data into the generator model of the Generative Adversarial Network (GAN) model, wherein the GAN model includes multiple model parameters learned using a training dataset comprising one or more pairs of images, wherein each pair of images in the one or more pairs of images comprises a first image generated by the first type of scanner and a second image generated by the second type of scanner, wherein the first image and the second image are registered for the same object or scene; A new image is generated based on the input data using the generator model; The new image is then input into the discriminator model of the GAN model; The discriminator model generates a probability that the new image is true or false, where true indicates that the new image has characteristics similar to those of the target image, and false indicates that the new image does not have characteristics similar to those of the target image, wherein the characteristics of the target image are associated with a second type of scanner different from the first type of scanner. The new image is determined to be real or fake based on the generated probability. as well as When the image is true, output the new image.

2. The method according to claim 1, wherein the biological sample is placed on a pathological slide, the first type of scanner is a first type of whole slide imaging scanner, and the second type of scanner is a second type of whole slide imaging scanner.

3. The method of claim 1, further comprising: The new image is input into an image analysis model, wherein the image analysis model includes multiple model parameters learned using a training dataset comprising images obtained from a scanner of the same type as the second type of scanner; The new image is analyzed using the image analysis model. The image analysis model generates analysis results based on the analysis of the new image. as well as Output the analysis results.

4. The method of claim 3, wherein the image analysis model was not trained on images obtained from a scanner of the same type as the first type of scanner.

5. The method of claim 1, further comprising training an image analysis model using a training dataset including the new image.

6. The method of claim 1, wherein the discriminator model is trained based on minimizing a first loss function to maximize the probability of the training dataset and the discriminator model is trained based on minimizing a second loss function to minimize the probability of the generated image sampled from the generator model and the generator model is trained to maximize the probability assigned to the generated image by the discriminator model, the plurality of model parameters being learned using the training dataset.

7. The method of claim 3, further comprising: The diagnosis of the subject is determined by the user based on the analysis results.

8. The method of claim 7, further comprising administering the treatment of the compound by a user based on (i) the results of the analysis and / or (iii) the diagnosis of the subject.

9. A system for image analysis, comprising: One or more data processors; as well as A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform actions including: Obtain a source image of a biological sample, wherein the source image is generated from a first type of scanner; Randomly generated noise vectors and latent feature vectors from the source images are input as input data into the generator model of the Generative Adversarial Network (GAN) model, wherein the GAN model includes multiple model parameters learned using a training dataset comprising one or more pairs of images, wherein each pair of images in the one or more pairs of images comprises a first image generated by the first type of scanner and a second image generated by the second type of scanner, wherein the first image and the second image are registered for the same object or scene; A new image is generated based on the input data using the generator model; The new image is then input into the discriminator model of the GAN model; The discriminator model generates a probability that the new image is true or false, where true indicates that the new image has characteristics similar to those of the target image, and false indicates that the new image does not have characteristics similar to those of the target image, wherein the characteristics of the target image are associated with a second type of scanner different from the first type of scanner. The new image is determined to be real or fake based on the generated probability. as well as When the image is true, output the new image.

10. The system of claim 9, wherein the biological sample is placed on a pathological slide, the first type of scanner is a first type of whole-slide imaging scanner, and the second type of scanner is a second type of whole-slide imaging scanner.

11. The system of claim 9, wherein the action further comprises: The new image is input into an image analysis model, wherein the image analysis model includes multiple model parameters learned using a training dataset comprising images obtained from a scanner of the same type as the second type of scanner; The new image is analyzed using the image analysis model. The image analysis model generates analysis results based on the analysis of the new image. as well as Output the analysis results.

12. The system of claim 11, wherein the image analysis model was not trained on images obtained from a scanner of the same type as the first type of scanner.

13. The system of claim 12, wherein the action further comprises training an image analysis model using a training dataset including the new image.

14. The system of claim 9, wherein the discriminator model is trained based on minimizing a first loss function to maximize the probability of the training dataset and the discriminator model is trained based on minimizing a second loss function to minimize the probability of the generated image sampled from the generator model and the generator model is trained to maximize the probability assigned to the generated image by the discriminator model, the plurality of model parameters being learned using the training dataset.

15. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform actions including: Obtain a source image of a biological sample, wherein the source image is generated from a first type of scanner; Randomly generated noise vectors and latent feature vectors from the source images are input as input data into the generator model of the Generative Adversarial Network (GAN) model, wherein the GAN model includes multiple model parameters learned using a training dataset comprising one or more pairs of images, wherein each pair of images in the one or more pairs of images comprises a first image generated by the first type of scanner and a second image generated by the second type of scanner, wherein the first image and the second image are registered for the same object or scene; A new image is generated based on the input data using the generator model; The new image is then input into the discriminator model of the GAN model; The discriminator model generates a probability that the new image is true or false, where true indicates that the new image has characteristics similar to those of the target image, and false indicates that the new image does not have characteristics similar to those of the target image, wherein the characteristics of the target image are associated with a second type of scanner different from the first type of scanner. The new image is determined to be real or fake based on the generated probability. as well as When the image is true, output the new image.

16. The computer program product of claim 15, wherein the biological sample is placed on a pathological slide, the first type of scanner is a first type of whole slide imaging scanner, and the second type of scanner is a second type of whole slide imaging scanner.

17. The computer program product of claim 15, wherein the action further comprises: The new image is input into an image analysis model, wherein the image analysis model includes multiple model parameters learned using a training dataset comprising images obtained from a scanner of the same type as the second type of scanner; The new image is analyzed using the image analysis model. The image analysis model generates analysis results based on the analysis of the new image. as well as Output the analysis results.

18. The computer program product of claim 17, wherein the image analysis model was not trained for images obtained from a scanner of the same type as the first type of scanner.

19. The computer program product of claim 15, wherein the discriminator model is trained based on minimizing a first loss function to maximize the probability of the training dataset and the discriminator model is trained based on minimizing a second loss function to minimize the probability of the generated image sampled from the generator model and the generator model is trained to maximize the probability assigned to the generated image by the discriminator model, the plurality of model parameters being learned using the training dataset.

Images