SYSTEM AND METHOD FOR PERFORMING DYE DECONVOLUTION - Patent application

Abstract

A stain deconvolution method and system using a deep-image-prior based neural network structure and associated training protocol is disclosed. The exemplary stain deconvolution network uses an autoencoder network to generate stain concentration maps of multiple stains associated with a color target image without the need for training on a large reference image dataset. The stain deconvolution network is trained using a loss function that promotes correct image generation and separation between the stain concentration maps generated by the autoencoder network. The deep image prior based stain deconvolution network can be configured to encode a fitted physical model that includes a set of parameters that model the background illumination and the nonlinear dependence of absorption on concentration and wavelength. Thus, the present exemplary method of stain deconvolution, which can be performed in the absence of prior training data, is generalizable to accommodate multiple stains and previously uncharacterized stain types, and can be extended to perform stain normalization.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 330,419, entitled "SYSTEMS AND METHODS FOR PERFORMING STAIN DECONVOLUTION," filed April 13, 2023, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] The present disclosure relates to deconvolution of stain components from color images, such as digital histopathology images.

[0003] Histopathology is a diagnostic field based on visual interpretation of cell biology captured in images using stains that specifically bind target antigens. The advent of digitized images in pathology has propelled this traditional field into what is now described as digital pathology, where pathological information from stained slides is captured, stored, and managed to create large-scale data sets for disease diagnosis, biological research, and drug discovery. Digital images are amenable to computational pathology, both for basic measurement and counting tasks, and for advanced machine learning tasks. Here, images can be assessed by machine learning for features that go beyond traditional histopathology assessments, such as directly linking images to clinical data (e.g., prognosis, mutations).

[0004] Therefore, performing stain deconvolution from multi-stained images is an essential step in most tissue image analysis algorithms. Stain deconvolution is the process of converting a stained tissue section image from the usual RGB color space into a set of stain channels. Each stain channel is a grayscale image, which represents the intensity of a particular stain expression across the original image. Stain deconvolution methods typically attempt to find an ideal stain matrix, i.e., a matrix that when multiplied by the RGB color channels produces the desired stain channels. A stain matrix consists of stain vectors (also known as "color vectors" or "stain color vectors"), where each stain vector represents a model color of a particular stain from the original image and, for each color channel, provides the correspondence between the particular stain and the absorbance (optical density).

[0005] One common problem in the analysis of tissue samples is unwanted color variation due to differences in the color response of slide scanners, the raw materials and manufacturing techniques of staining vendors, and staining protocols between different pathology laboratories. This makes image interpretation difficult by software trained on specific staining appearances, further amplifying existing inter- and intra-expert differences in diagnosis and labeling between pathologists. This variation can be corrected by performing stain deconvolution and then recombining the stain channels using a reference template; this is known as stain normalization. Summary of the Invention

[0006] A stain deconvolution method and system using a deep-image-prior based neural network structure and associated training protocol is disclosed. The exemplary stain deconvolution network uses an autoencoder network to generate stain concentration maps of multiple stains associated with a color target image without the need for training on a large reference image dataset. The stain deconvolution network is trained using a loss function that promotes correct image generation and separation between the stain concentration maps generated by the autoencoder network. The deep image prior based stain deconvolution network can be configured to encode a fitted physical model that includes a set of parameters that model the background illumination and the nonlinear dependence of absorption on concentration and wavelength. Thus, the present exemplary method of stain deconvolution, which can be performed in the absence of prior training data, is generalizable to accommodate multiple stains and previously uncharacterized stain types.

[0007] Thus, in a first aspect, there is provided a method of performing dye deconvolution on a colour target image, the method comprising: A dye deconvolution network is provided, the dye deconvolution network comprising: a plurality of convolutional autoencoder neural networks, each convolutional autoencoder neural network configured to process a respective input data set to generate a respective output data set, each input data set including a respective two-dimensional array having a dimension equal to a pixel dimension of the color target image; an absorbance calculation module configured to generate a color absorbance image using the absorbance model, the color absorbance image being generated by processing a plurality of stain vectors and a plurality of stain density maps, each stain vector and each stain density map being associated with a respective stain; providing an absorbance calculation module operatively coupled to a plurality of convolutional autoencoder neural networks, whereby each stain concentration map is obtained from an output data set of a respective convolutional autoencoder neural network; training a stain deconvolution network according to a stain deconvolution loss function, the stain deconvolution loss function including a first loss component configured to minimize a generation loss associated with the color target image and a second loss component configured to facilitate separation between the stain concentration maps, such that after training, each of the stain concentration maps represents a deconvolved stain concentration map of a stain in the color target image.

[0008] In some embodiments of the method, the absorbance calculation module is further configured to generate the color absorbance image by calculating a sum of the products of the stain vector, the stain concentration map, and the stain spectral correction coefficient over each stain, each stain having an associated stain spectral correction coefficient, and each stain spectral correction coefficient being updated during the training according to minimization of a stain deconvolution loss function.

[0009] The stain spectrum correction coefficient can be defined according to a stain spectrum correction coefficient parameter of the stain deconvolution network, where the stain spectrum correction coefficient parameter is initialized before training.

[0010] The stain deconvolution network may include a plurality of spectral correction neural networks, each of which is configured to determine a respective stain spectral correction coefficient, and each of which is trained according to a stain deconvolution loss function. At least one of the spectral correction neural networks of the stain deconvolution network may be an encoder-decoder network. The stain spectral correction coefficients corresponding to the at least one spectral correction neural network may be determined according to a latent feature of the at least one spectral correction neural network.

[0011] In some embodiments of the method, the absorbance calculation module is further configured such that the calculation of the color absorbance image includes a color background vector, and the color background vector is updated during the training according to minimization of the stain deconvolution loss function. The color background vector can be defined according to background parameters of a stain deconvolution network, and the background parameters are initialized before training. The stain deconvolution network can include a background neural network configured to determine the color background vector, and the background neural network is trained according to the stain deconvolution loss function. The background neural network can be an encoder-decoder network. The value of the color background vector can be determined according to latent features of the background neural network.

[0012] In some implementations of the method, the stain vectors are updated during training according to minimization of a stain deconvolution loss function. The stain vectors may be defined according to stain vector parameters stored in the stain deconvolution network, and the stain vector parameters are initialized before training. The stain deconvolution network may include a plurality of stain vector neural networks, each stain vector neural network configured to determine a respective stain vector, and each stain vector neural network is trained according to the stain deconvolution loss function. At least one stain vector neural network of the stain deconvolution network may be an encoder-decoder network. Values ​​of the stain vectors corresponding to the at least one stain vector neural network may be determined according to latent features of the at least one stain vector neural network.

[0013] In some implementations of the method, during at least an initial portion of the training, the staining deconvolution loss function includes an additional loss term based on the difference between the staining vectors calculated by the staining vector neural network and a predetermined initialization value of the staining vectors. The additional loss term may be included in the staining deconvolution loss function during the initial portion of the training and is not present in the staining deconvolution loss function during a subsequent portion of the training.

[0014] In some implementations, the method further comprises performing normalization using the staining vector when processing different color target images.

[0015] In some embodiments, the method further includes using the color absorbance image to generate an output image, thereby providing a reproduced version of the color target image.

[0016] In some embodiments of the method, the first loss component is based on a difference between the color absorbance image and a target color absorbance image generated from the color target image.

[0017] In some embodiments of the method, the first loss component is based on a difference between a color target image and an output image generated based on the color absorbance image.

[0018] In some embodiments of the method, at least one input data set is randomly generated.

[0019] In some embodiments of the method, at least two of the input datasets are a common input dataset.

[0020] In some implementations of the method, at least one of the convolutional autoencoder neural networks includes a skip connection.

[0021] In some implementations of the method, during at least an initial portion of training, the computation of the loss function is augmented using at least one transformation of the input dataset.

[0022] In some implementations of the method, the color target image is a first color tile of the main color image, and the staining vector obtained after training is a final staining vector, and the method further includes using the final staining vector when performing staining deconvolution of another color tile of the main color image. The final staining vector can be used to initialize the staining vector when performing staining deconvolution of other color tiles of the main color image.

[0023] In some embodiments, the method further comprises performing quantification of the staining using the staining density map.

[0024] In another aspect, a system for performing dye deconvolution on a color target image is provided, the system comprising: A control and processing circuit including at least one processor and a memory, the memory comprising: generating a dye deconvolution network, the dye deconvolution network comprising: a plurality of convolutional autoencoder neural networks, each convolutional autoencoder neural network configured to process a respective input data set to generate a respective output data set, each input data set including a respective two-dimensional array having a dimension equal to a pixel dimension of the color target image; an absorbance calculation module configured to generate a color absorbance image using an absorbance model, the color absorbance image being generated by processing a plurality of stain vectors and a plurality of stain density maps, each stain vector and each stain density map being associated with a respective stain; Including, the absorbance calculation module is operatively coupled to a plurality of convolutional autoencoder neural networks, whereby each stain concentration map is obtained from an output data set of a respective convolutional autoencoder neural network; Generating, Training a dye deconvolution network according to a dye deconvolution loss function, the dye deconvolution loss function being: a first loss component configured to minimize a generation loss associated with the color target image; a second loss component configured to facilitate separation between the stain density maps; Training, including Including, Thereby, after training, each stain density map represents a deconvolved stain density map of the stain in the color target image, a control and processing circuit including instructions executable by the at least one processor to perform operations; Includes.

[0025] A further understanding of the functional and advantageous aspects disclosed herein can be realized by reference to the following detailed description and drawings.

[0026] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0027] [Figure 1] 1 illustrates a schematic diagram of a conventional matrix factorization based dye color elimination method; [Diagram 2] 1 shows a schematic of an example of a conventional deep learning approach to stain normalization. [Figure 3A] 1 illustrates an exemplary embodiment of a stain deconvolution network based on the use of multiple deep image priors to determine a stain density map. [Figure 3B] FIG. 3B shows an exemplary implementation of an autoencoder network suitable for use with the exemplary dye deconvolution network of FIG. 3A. [Figure 4] We present an example of a deep image prior-based staining deconvolution network in which staining vectors are refined during training of the autoencoder neural network. [Diagram 5] 1 illustrates generally different exemplary methods for refining staining vectors during training of a deep image prior-based staining deconvolution network. [Figure 6] 13 illustrates an exemplary embodiment in which a stain deconvolution network includes an autoencoder neural network that is used to generate a stain concentration map associated with two stains. [Figure 7] 1 illustrates an exemplary method for performing stain normalization using a deep image prior-based stain deconvolution network. [Figure 8] 1 illustrates an exemplary system for performing stain deconvolution. [Figure 9]13 shows unsupervised clustering results of physical parameters estimated by dye deconvolution networks consistent with scanner type for an exemplary dataset. [Figure 10] We show how dye spectral correction factors can reduce systematic errors in color deconvolution compared to conventional approaches. [Figure 11] We present stain deconvolution results on three types of digital pathology images, as well as examples of how the spectral correction factors improve stain deconvolution performance. [Figure 12] We present a comparison of the proposed approach with state-of-the-art stain deconvolution methods for color deconvolution of H&E images, where the proposed approach achieved the highest point biserial correlation. [Figure 13] We present a comparison of the proposed approach with state-of-the-art stain deconvolution methods for color deconvolution of H&E images, where the proposed approach achieved the best structural similarity index measure. [Figure 14] We present an example of dye transformation and normalization using a dye deconvolution network on a dataset with images captured by four different types of scanners. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Various embodiments and aspects of the present disclosure are described with reference to the details discussed below. The following description and drawings are illustrative of the present disclosure and should not be construed as limiting the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the present disclosure. However, in certain instances, well-known or conventional details are not set forth in order to provide a concise description of the embodiments of the present disclosure.

[0029] As used herein, the terms "comprises" and "comprising" are to be interpreted as inclusive and open-ended, and not exclusive. Specifically, when used in this specification and claims, the terms "comprises" and "comprising" and variations thereof mean that the specified features, steps, or components are included. These terms are not to be interpreted as excluding the presence of other features, steps, or components.

[0030] As used herein, the term "exemplary" means "serving as an example, instance, or illustration," and should not be construed as preferred or advantageous over other configurations disclosed herein.

[0031] As used herein, the terms "about" and "approximately" are meant to encompass variations that may exist at the upper and lower limits of a range of values, such as variations in properties, parameters, and dimensions. Unless otherwise specified, the terms "about" and "approximately" mean up to plus or minus 25 percent.

[0032] Unless otherwise stated, any specified range or group should also be understood as a shorthand for referring individually to every element of a range or group, and each of the possible subranges or subgroups contained therein, as well as any subranges or subgroups therein. Unless otherwise stated, the present disclosure relates to and expressly incorporates every specific element and combination of subranges or subgroups.

[0033] As used herein, the term "on the order of" when used in conjunction with an amount or parameter refers to a range ranging from about one-tenth to ten times the recited amount or parameter.

[0034] Despite the great interest in dye deconvolution methods, their generalizability to specific image analysis problems is limited. Traditional dye deconvolution methods rely on matrix decomposition algorithms based on the Beer-Lambert law, which relates the attenuation of unscattered light to the properties of the dye used for staining.

[0035] A conventional approach to stain deconvolution is shown in Figure 1. In conventional approaches, the color fading problem is treated as a matrix factorization problem. After preprocessing steps such as resizing and background illumination correction, the Beer-Lambert transform is applied to convert the observed image intensity values ​​in each RGB channel as optical densities, and then the stain concentration maps and stain color vectors for each stain are recovered using matrix factorization. To achieve better performance than baseline non-negative matrix factorization (NMF) or singular value decomposition (SVD), conventional physics-based methods rely on manually created priors (e.g., criteria for sparsity, structural shape, histogram / spectral matching), but these priors are tedious, heuristic, and stain-specific. The color absorption properties of the dyes (stain vectors) are predefined from empirical observations or estimated using template matching from manually selected regions of interest (ROIs).

[0036] While most stain deconvolution methods remain relatively robust for hematoxylin and eosin (H&E) stained images, their application to specialized stains is severely affected. Indeed, stain deconvolution for other stains relies on problem-specific designs that require extensive domain expertise and experience to develop for one set of stains and rarely transfer to other sets of stains.

[0037] Specifically, in stain deconvolution based on stains other than H&E, the Beer-Lambert law assumption usually fails in practice, often leading to significant deconvolution errors. Furthermore, with the rapid development of digital pathology, specialized stains are becoming increasingly popular to reveal detailed information and structures that improve disease diagnosis and treatment beyond what traditional H&E stains can provide.

[0038] With the recent success in deep learning, the focus of stain deconvolution methods has shifted to neural network-based approaches, highlighting stain deconvolution of special stains. Figure 2 shows an example of a traditional deep learning approach to stain normalization. Deep learning approaches typically do not explicitly calculate concentration maps and stain vectors, but instead directly perform image-to-image transformation (converting an image with one stain type / source to another). To achieve this, a large dataset of images from the two sources is required. Similar to traditional physics-based approaches, the images are first preprocessed and then fed into two neural networks, leveraging the powerful learning ability of neural networks to learn color transformation and imaging structure simultaneously. Apart from the image generation branch (GAN, VAE, etc.) for image reconstruction or image-to-image transformation, deep learning-based approaches typically incorporate an auxiliary branch for performance improvement by solving another related task (e.g., tissue type clustering). Compared to traditional methods, deep learning-based methods generally perform better because color separation / style transfer does not rely on manually created priors and manually selected ROIs.

[0039] However, deep learning-based techniques have much lower interpretability and the normalization procedure requires a template image from the target domain. Existing deep learning-based approaches are claimed to be unsupervised since they do not require separate staining layer annotations, but are semi-supervised / weakly supervised since they rely on large curated image datasets from different domains.

[0040] Moreover, although the powerful capabilities of deep learning allow stain deconvolution methods to circumvent the limitations of the Beer-Lambert law, their accessibility and generalizability are usually problematic. Even though existing deep learning-based approaches are called “unsupervised” because they do not require ground truth of isolated stain layers for training, they still require large manually selected datasets, careful management of data, and a high level of expertise in network design and training. The data-driven nature of existing deep learning-based stain deconvolution methods, on the one hand, makes them very powerful on the trained data, but on the other hand, reduces their performance on unseen data. The lack of generally applicable stain deconvolution algorithms is an obstacle to the development of digital pathology analysis.

[0041] The inventors have recognized that the limitations of conventional deep learning methods for stain deconvolution can be circumvented by using a deep learning framework that does not require prior training based on a large image dataset. Indeed, by using a deep image prior-based deep learning framework, accurate stain deconvolution can be performed using only the image data itself to train the network, instead of relying on a large training dataset.

[0042] Deep image prior (DIP) neural networks were first proposed for image inverse problems such as denoising and restoration, based on the concept that the structure of the neural network, especially the convolutional filters, are capable of capturing the imaging statistics of natural images without being trained on a specific task. The deep image prior neural network is trained using randomly initialized network parameters to generate an output image based on a loss function involving a target image, with random noise as the input to the neural network. During the training process, the convolutional filters of the network capture the internal patch regressions (also known as image priors), which can be used as the basic building blocks to reconstruct the target image, thus removing extraneous image information such as noise or artifacts, facilitating the restoration of a denoised or corrected version of the target image.

[0043] Deep image prior-based neural networks have also been implemented to include two deep image prior networks (e.g., a so-called "double DIP" network) based on the observation that when multiple deep image prior neural networks are combined to reconstruct an image, each deep image prior neural network tends to split the image into a meaningful decomposition, e.g., background and foreground, blurred and deblurred images, or overlapping images with different levels of transparency.

[0044] The inventors have recognized that a multi-deep image prior neural network can be used to perform stain deconvolution using a set of autoencoder networks to generate stain density maps of multiple stains associated with a color target image, without requiring training on a large reference image dataset. Instead, the inventors have reasoned that a multi-deep image prior network can be trained using a loss function that uses (i) a loss component that promotes correct image generation based on a color target image, and (ii) a loss component that promotes the separation between the stain density maps generated by the autoencoder networks.

[0045] Unlike previous stain deconvolution methods, the present stain deconvolution method, which uses a deep image prior neural network structure and associated training protocol, can be used to perform stain deconvolution on images with any number and type of stains, as described in detail below. The present deep image prior-based stain deconvolution network can also be configured to encode an adaptive physical model that includes a set of parameters that describe the characteristics of the background illumination light, the polychromatic dye, and the sensor, thereby improving interpretability. Because test-time stain deconvolution is performed without the need for prior training data, the present method can be generalized to accommodate new and previously uncharacterized stain types without the risk of overfitting or the need for ground truth labels. Furthermore, because the present exemplary deep image prior-based method does not require training on an image dataset, but instead only needs to be trained on the color target image itself, the present method can be applied to entry-level graphics processing cards.

[0046] FIG. 3A shows an exemplary embodiment of a stain deconvolution network based on the use of multiple deep image priors. The exemplary stain deconvolution network does not require prior training and is therefore stain type independent. A neural network portion 310 including multiple autoencoder neural networks is used to generate stain density maps for at least two stains, including stain density maps 321 and 322 corresponding to a first stain and a second stain, and one or more optional additional stain density maps 323 corresponding to one or more optional additional stains.

[0047] As shown, the stain density maps 321-323 are processed in an absorbance calculation module 340 with corresponding stain vectors 330 to generate color absorbance and / or color intensity images, as shown at 350. While the RGB color space is often used in the examples provided in this disclosure, it will be appreciated that other color spaces, such as, but not limited to, HSV and Lab, may alternatively be used.

[0048] The deconvolution is performed on the color target image, which is a two-dimensional image with dimensions L x W and three color channels, and can therefore be represented by a matrix with dimensions 3 x L x W. Thus, each stain concentration map has dimensions L x W, and each stain vector is a three-dimensional vector, one dimension for each color channel.

[0049] In an exemplary method of the invention, the absorbance module generates a color absorbance image based on the calculated stain density maps 321-323 using the Beer-Lambert law. For example, for a color target image corresponding to a sample with n mixed stains, the absorbance (optical density, OD) is calculated as k and staining concentration map S k where k=1...n represents the number of dyes, as follows:

number

[0050] As explained in more detail below, this formula is only one example of an absorbance calculation, and in other embodiments below, alternative methods of calculating absorbance are disclosed that include additional coefficients and / or terms.

[0051] The optical density (OD) is related to the intensity image by the following formula:

number

[0052] In a conventional deep learning-based workflow, a neural network is trained based on a training image data set to modify the activations and weights of the neural network. Then, a target image is processed by the trained network. However, in stark contrast, the stain deconvolution network shown in FIG. 3A does not need to be trained based on a training data set, and the stain deconvolution network can determine the stain density maps 321-323 of a target color image without the target color image being provided as an input to the network. Instead, as described below, the stain density maps 321-323 can be determined based on an appropriate loss function involving the color target image using the concept of a deep image prior.

[0053] 3A, the neural network portion 310 includes, for each stain, a respective autoencoder network 311-313, which are used to generate a respective stain concentration map 321-323. The autoencoder networks 311-313 do not need to be trained prior to performing stain deconvolution on the color target image.

[0054] A non-limiting example of a suitable autoencoder network 311 is shown in FIG. 3B. In this exemplary implementation, skip connections are included between the first layer of the encoder and the fifth layer of the decoder, and between the second layer of the encoder and the fourth layer of the decoder. It will be understood that the use of skip connections and the number of skip connections are optional and may vary between implementations of this embodiment. Furthermore, it will be understood that other hyperparameters, including, for example, the number of channels, the number of layers, and the type of activation, are also variable according to different implementations.

[0055] 3A, each autoencoder network may be provided as input with a random noise dataset 360 (e.g., random uniform noise in the range [-0.5, 0.5]), or other datasets including the color target image itself. Although the figure shows each autoencoder network being fed with the same input dataset, this is just one implementation and the autoencoder networks may be fed with different input datasets in other exemplary implementations.

[0056] It may seem counterintuitive that a randomly initialized autoencoder neural network can generate accurate stain density maps based on random input data that is in no way correlated with the color target image, but the autoencoder neural networks 311-313 of the exemplary stain deconvolution network are capable of performing this function based on the color target image and using a loss function that includes a term that enforces separation between the stain density maps.

[0057] The loss function for training the neural network portion 310 may include at least the following two loss components: The first loss component is a generation loss component used to minimize the generation error. This generation loss component may be configured to, for example, minimize the difference between a color absorbance image (hereinafter referred to as a "generated color absorbance image") generated based on a stain density map according to the Beer-Lambert law and an absorbance image calculated from a color target image, or to, for example, minimize the difference between a final image generated based on a stain density map (e.g., using the formula shown above) and the color target image. The generation loss may be an L1 type loss. For example, in some exemplary implementations, the generation loss may be calculated as an L1 loss of absorbance for each color channel, and may optionally include a loss term based on the sum of the absorbance for the three color channels.

[0058] The second loss term provides an exclusion loss between the stain density maps to facilitate separation between different stain layers, since different stains should enhance different regions of interest.

[0059] For example, for a two-color dye separation problem (i.e., the color target image includes contributions from two dyes), the net loss function may include four production losses (an absorption production loss term for each of the three color channels, one absorption sum production loss term (across all color channels)) and one exclusion loss term that facilitates separation between the two dye concentration maps. This exemplary loss function can be generalized to N>3 colors by incorporating the aforementioned production loss terms and N(N-1) / 2 exclusion loss terms.

[0060] In some exemplary embodiments, the staining vectors 330 are refined during the training of the autoencoder neural network. For example, FIG. 4 illustrates an exemplary embodiment in which the staining vectors 330 are shown to reside with the neural network portion 310 and their values ​​are updated during the training of an epoch of the autoencoder neural network. The staining vectors may be initialized, for example, from randomly initialized values, from user-input staining vectors, or from staining vectors that may be automatically calculated from user-selected regions of interest in the input image as prior knowledge. In the latter case, the network automatically adjusts the user-input staining vectors to achieve better separation results.

[0061] In an exemplary implementation, the staining vector 330 may be incorporated into the staining deconvolution network as a parameter that is updated during training. For example, as shown in FIG. 4, the staining vector 330 may be provided without a neural network structure as a set of parameters (e.g., six parameters in the two staining example) that are stored and refined during training. The values ​​of the staining vector parameters are initialized by initial values ​​(e.g., estimated or random values), as shown at 370. This configuration is also shown diagrammatically in case 1 of FIG. 5.

[0062] However, in other exemplary implementations, each stain vector can be generated by a respective stain vector neural network structure present in the neural network portion 310 and trained together with an autoencoder network that calculates the stain concentration map. Two exemplary implementations of such an embodiment are shown in Cases 2 and 3 of FIG. 5. Case 2 shows an exemplary case where a given stain vector is calculated according to a feedforward neural network 332, and Case 3 shows an exemplary case where a given stain vector is calculated according to an encoder-decoder network 334. When the encoder-decoder network 334 is used to generate the stain vectors, the value of the given stain vector can be generated, for example, by a subset of pixels of the decoded image (e.g., including a pixel in the center of the decoded image), or can be generated, for example, based on latent features generated by the network.

[0063] For example embodiments in which the staining vectors 330 are refined during training of the autoencoder neural network, the loss function may be adapted to include a loss component for the staining vectors. For example, a color-fixed loss term may be included in the loss function to adjust the staining deconvolution network during at least a portion of the training.

[0064] In one exemplary implementation, to constrain the stain vectors during the initial part of training (e.g., during an initial number of epochs less than 2000), an L1 color-fixed loss is calculated between the estimated stain color vectors and the initial values ​​of the stain vectors, and is removed during the remaining part of training. This initial color-fixed loss helps make the network training more robust during the initial training by focusing on learning internal structural patterns before learning the stain vectors. In the exemplary case of a color target image associated with two stains, two color-fixed loss terms per stain may be included, which can be generalized to one color-fixed loss term per stain.

[0065] In one exemplary implementation, a color-fixed loss is trained on the network in the first 25% epochs and removed in the last 75% epochs, allowing the network to learn the staining vectors of the stains.

[0066] In some exemplary implementations, augmentations may be provided during training of the neural network portion. Non-limiting examples of augmentations include 90 degree rotation and mirror operations performed on the input image data (e.g., random noise) to help the network learn a pose-invariant deep image in advance. To help the network focus on deconvolving the stains, augmentations may be included during the early part of training (e.g., during the first 75% of training epochs) and disabled during the later part of training (e.g., during the last 25% of epochs).

[0067] The inventors have also recognized that the exemplary staining deconvolution networks shown in FIGS. 3A and 4 can be further adapted to overcome limitations of conventional staining deconvolution methods.

[0068] In fact, as we will explain in detail in the examples below, the assumption of the Beer-Lambert law, although widely used in almost all matrix decomposition-based approaches, is always violated in practice. Although the Beer-Lambert law assumes monochromatic light (of a single wavelength), the background illumination light, dye absorption spectra, the response of the image sensor, and the color-matching observer function are all wavelength-dependent. Furthermore, although the Beer-Lambert law describes color absorbance, some dyes are light scatterers. Furthermore, existing methods of dye deconvolution are problematic because the perceived color depends on spectral dependencies resulting from light conditions, room temperature, staining time, batch effects, and other factors that vary between images acquired from the same scanner.

[0069] Therefore, the assumption that the stain vectors are independent of stain concentration usually does not hold due to the complex spectral responses of imaging cameras and light sources, and stain deconvolution methods based on stain vectors without considering the dependence on stain concentration therefore yield less accurate concentration results.

[0070] We realized that to address the systematic errors caused by the mismatch between the nonlinear concentration dependences of stain vectors, our multi-deep image pre-stained deconvolution network could be adapted to use a modified form of the Beer-Lambert law. Indeed, we found that the issues associated with this nonlinearity could be addressed by introducing a stain spectrum correction factor that approximates the Beer-Lambert law (an integrated version of the Beer-Lambert law) at the individual image level, as opposed to the dataset level used by all existing methods.

[0071] In some exemplary embodiments, the Beer-Lambert law is modified to include stain spectrum correction factors that assist in resolving the aforementioned concentration dependencies.

number

[0072] The stain spectral correction coefficients approximately compensate for nonlinearities from effects such as, but not limited to, light scattering, non-monochromatic light composition, and non-monochromatic light reception of the camera. The full nonlinear relationship for each stain is very difficult to calculate without a large amount of data, but the present exemplary method approximates the nonlinearities based on scalar parameters for each stain that are assumed to be uniform across each image tile. Thus, the stain spectral correction coefficients are assumed to be fixed for an image tile, but may vary between images even from the same whole slide image, and therefore cannot be calculated by any existing method.

[0073] However, the present method using a deep image prior based network can be easily adapted to determine the stain spectrum correction coefficients during training. For example, the stain spectrum correction coefficients M k can be stored in the network as parameters that are refined during training of the autoencoder neural network that generates the stain concentration map. Alternatively, a feed-forward neural network trained according to a loss function, or an encoder-decoder network, may be used to generate the stain spectrum correction coefficients (similar to the different embodiment shown for the stain vectors in FIG. 5).

[0074] Although exemplary embodiments of the invention use a single parameter stain spectral correction coefficient for each stain, it will be appreciated that other exemplary embodiments may use multiple stain spectral correction parameters for each stain, including coefficients that define a non-linear function of concentration. For example, such parameters may be determined by modeling the dependence of absorbance as a function of concentration using a polynomial functional form.

[0075] Background illumination can affect the perceived color of images captured by a scanner and can vary significantly between different scanners. Existing methods rely on additional preprocessing steps based on Gaussian denoising filters or sampling from background pixels to recover the original image. To achieve optimal performance with such existing methods, preprocessing steps such as background illumination correction and noise removal need to be performed before dye deconvolution. These procedures require histogram matching of manually selected background patches with hand-crafted noise cancellation filters, which need to be designed for each domain (dataset / scanner) and can lead to bias.

[0076] In stark contrast, the preceding exemplary embodiments of the present disclosure may be adapted to allow for automatic estimation and correction of background illumination and noise without the need for human intervention. The presence and effect of background illumination may be addressed by including a background correction vector (a three-element vector with one element for each color channel) that accounts for changes in the optical density values ​​of each channel.

number

[0077] The background correction vectors may be stored in the network as parameters that are refined during training of the autoencoder neural network that generates the stain concentration map. Alternatively, a feed-forward neural network trained according to a loss function, or an encoder-decoder network, may be used to generate the background correction vectors (again, similar to the different implementations shown for the stain vectors in FIG. 5).

[0078] 6 illustrates an exemplary embodiment in which the stain deconvolution network includes autoencoder neural networks 311 and 312 trained to generate stain concentration maps 321 and 322 associated with two stains based on color target image data and an appropriate loss function. As shown, the neural network portion 310 also includes autoencoder neural networks 313 and 314 trained to determine a stain vector 330. The network portion 310 also includes two optional additional autoencoders 315 and 316, either of which may be optionally included and trained to generate stain spectrum correction coefficients 380 and / or background correction vectors 390 according to a loss function. Although the figure illustrates a single autoencoder neural network 315 for generating stain spectrum correction coefficients 380, separate autoencoder neural networks may be provided to calculate each stain spectrum correction coefficient. The network can be expanded by adding the following additional modules for each additional stain: additional concentration map autoencoder, additional stain vector autoencoder (or parameter set or feed-forward network), additional stain spectrum correction coefficient autoencoder (or parameter set, or feed-forward network). Furthermore, the stain deconvolution network can be expanded by including additional modules that capture additional components, including but not limited to, for example, ink marks, color temperature, and white balance.

[0079] In some exemplary embodiments, stain normalization may be performed using a deep image prior-based stain deconvolution network. Stain normalization refers to the use of a standard (or predetermined) set of stain vectors in combination with a given set of stain density maps to acquire images, in order to reduce the variability between images acquired using different scanners, thereby reducing the difficulty of downstream analysis.

[0080] FIG. 7 shows a schematic diagram of a method for generating a normalized reconstructed source image 430 from a source image 405 via a Beer-Lambert transform 435 (or a modified version thereof) using a target stain vector 420 obtained by processing a color target image 400 with a deep image prior-based stain deconvolution network 410. As shown in the lower part of the figure, the stain deconvolution network 410 is trained based on the target color image 400 according to the present exemplary embodiment, which includes the use of multiple deep image prior modules (autoencoders). The training results in the calculation of the target stain vector 420, as well as the target stain concentration map 440, and the stain deconvolution network may be further configured to determine additional target color parameters 450 associated with a physical model, such as target stain spectral correction coefficients and / or target background vectors.

[0081] As shown in the upper part of FIG. 7, the stain deconvolution network 410 is also trained based on the source color image 405 according to the present exemplary embodiment, including the use of multiple deep image prior modules (autoencoders). The training results in the calculation of the source stain vector 425, as well as the target stain density map 445, and, if the stain deconvolution network is so configured, additional source color parameters 455, such as source stain spectral correction coefficients and / or source background vectors. To generate a normalized reconstructed source image 430, the source stain density map 445 and optional additional source parameters 455 are processed using the target stain vector 420 instead of the source stain vector 425, using a physical model (e.g., an appropriate form of the Beer-Lambert law).

[0082] 7 also illustrates an exemplary case where the target stain vector 420 has already been determined and there is no need to perform training of the stain deconvolution network at the bottom of the figure; these unnecessary operations are indicated by dashed lines. In such a case, the target stain vector 420 may be used to replace the source stain vector 425 when generating the normalized reconstructed source image 430.

[0083] In some exemplary implementations, the normalization methods described above, or variations thereof, may be used for stain normalization across slide images. When images are scanned and / or processed in different laboratories, the appearance of the images often varies widely. Pathologists become accustomed to a certain appearance and prefer images that appear with a similar range of colors and intensities. Also, many image analysis pipelines and AI models rely on consistent colors to function. In some exemplary implementations, stain vectors obtained after training based on one image tile can be used to initialize stain vectors used when performing stain deconvolution of another tile of the image.

[0084] In some exemplary embodiments, quantification of staining may be performed on images generated according to the exemplary stain deconvolution methods. For example, a set of stain density maps determined by a stain deconvolution network generated according to (or based on) embodiments disclosed herein may be used to quantify the relative proportions of a given cell type. This may be done, for example, by identifying and quantifying cells within individual stain density maps and generating a ratio measure, for example, based on cell counts. Alternatively, quantification may be performed based on pixels to distinguish stained and unstained tissue, applicable to nuclear, cytoplasmic, or membrane staining.

[0085] In one non-limiting example, the present exemplary method can be used to quantify the abundance of p53 protein in cancer tissue. Tissue may be stained with an anti-p53 antibody, a secondary antibody containing peroxidase is used to visualize the DAB chromogen, and counterstained with hematoxylin. The secondary antibody is linked to a peroxidase enzyme, which causes the DAB chromogen to produce a visible brown precipitate at the site of the antigen (i.e., p53). Application of a staining deconvolution network according to the present exemplary method provides staining density maps of DAB (p53) and hematoxylin (cell nuclei). In one example of a quantification method, in a region of interest, the presence of p53 can be quantified by the number of p53 positive cells stained with DAB divided by the total number of cells stained with hematoxylin and / or DAB. In another exemplary embodiment, quantification can be performed based on pixels in the staining density map as opposed to identified cells.

[0086] It will be appreciated that when performing quantification of staining, one or more density maps may be thresholded to remove background pixels (e.g., low density pixels) in order to remove pixels that are not expected to belong to cells. The pathologist / researcher may determine, for example by visual inspection, an appropriate threshold value for defining positive pixels.

[0087] It will be understood that there are many antigens for different applications, and P53 is just one example of an antigen. Other examples of antigens include, for example, antigens that bind to ER and PR receptors in breast cancer, or antigens that highlight different types of inflammatory cells. The antigen determines the distribution of the stain, and two common distribution types are generated from nuclear and membrane stains. P53, ER, and PR are all examples of nuclear stains. Meanwhile, chromatin determines the color. The choice of chromatin varies by application and laboratory, but DAB stain / hematoxylin counterstain is a common stain combination. As mentioned above, the present stain deconvolution embodiment, which does not require the existence of a training dataset, is suitable for processing both known and novel color combinations.

[0088] As described in the Examples section below, we have found that our deep image prior-based method can outperform existing deep learning-based approaches for stain deconvolution for a variety of types of stains. Furthermore, we have found that our deep image prior-based method is unique in that it can deconvolve any type of stain without the need for task-specific empirical optimization.

[0089] Without intending to be limited by theory, it is believed that the strong generalizability of the present deep image prior based approach arises from the encoding of the physical model with the image prior based deep neural network, which complements and completes each other. Specifically, the deep neural network leverages the adjustment of the physical model to achieve optimization based only on the target image itself. This is unprecedented in deep learning based stain deconvolution. In particular, in the exemplary embodiment in which the stain spectrum correction coefficients are calculated via the deep learning neural network during the training of the entire network, it is possible to broaden the scope of application of the underlying physical theory and significantly reduce the deconvolution error.

[0090] The demonstrated strong performance of the exemplary embodiments of the present invention demonstrates that it is possible to train a competent machine learning model at test time based on the target image itself, which not only avoids the labor-intensive and time-consuming data collection and curation process but also prevents overfitting. The synergy between the physical model and the deep image prior-based deep neural network also addresses another challenge of neural networks: interpretability. The output of some exemplary dye deconvolution networks of the present disclosure includes a set of physical parameters that explicitly describe the conditions and properties involved in the entire process of light emission from the dye to the scanner camera. The results shown in the following examples suggest that the exemplary dye deconvolution network of the present invention using the modified Beer-Lambert physical model, with the associated physical parameters, allows the dye deconvolution network to perform well with less commonly used stains, confirming the effectiveness of the modifications to the physical model.

[0091] It is further noted that in addition to stain deconvolution, the present deep image prior-based method also automates and enables image denoising. Digital pathology images contain a large amount of noise and artifacts. Most noise appears meaningless to human perception, but is evident to the "eye" of a computational classifier. The implementation of a deep image prior-based network structure and training protocol facilitates this denoising capability. The autoencoder used and trained to generate stain concentration maps utilizes a limited amount of convolution filters to capture the regression of interior patches of the color target image. This network structure and training protocol allows for the learning of components of regular structure, but not of irregular noise and small artifacts. Furthermore, in embodiments that address and model the nonlinear dependence of stain vectors on concentration, as described above, and in embodiments that model the characteristics of background light, these effects can be directly incorporated into the end-to-end training of color deconvolution, enabling a simpler and more convenient solution to the required pre-processing steps.

[0092] In fact, stain deconvolution methods implemented using the framework of deep image priors are entirely dependent on the information provided by the test images. It should be noted that their performance may degrade if the quality of the test images is poor, especially if the proposed generalized assumptions (individual stains are uniform in the target image, while the properties of the stains are different) are violated. For example, if the image has large ink marks with high density, some implementations may tend to calculate biased stain vectors and density maps as a compromise to fit both the stain structure and the ink mark. Similarly, if the colors of the two stains are too similar, some implementations may have suboptimal performance in assigning the structure to the correct stain. However, it should be noted that these aspects are not a result of the network structure design itself, but instead are common challenges for all stain deconvolution methods. In fact, the aforementioned exemplary embodiment may be adapted to address these challenges by stacking additional deep image prior modules (autoencoders), updating the physics model (e.g., by adding terms and / or parameters to the Beer-Lambert law) to take into account large artifacts, and by estimating all intermediate physical parameters for better quality control.

[0093] Therefore, it is evident that the present deep image prior based method has the beneficial potential to provide a new generation of stain deconvolution methods that are generalizable to a wide variety of stains without requiring specialized knowledge or prior data collection.

[0094] The present deep image prior-based stain deconvolution method may find applications in a wide variety of settings, including but not limited to clinical pathology, pathology research, and pharmaceutical research. Indeed, in pharmaceutical laboratories, immunohistochemistry is widely used to explore different biomarkers and develop companion diagnostic tests. Such methods include staining slides with several different dyes that label different antigens or markers. A very typical example is the use of hematoxylin to stain cell nuclei, and DAB staining bound to the antigen of interest. To provide more accurate quantitative results, a method is needed to separate these stains. In some research applications, there may be multiple different stain colors used to identify various structures in the tissue. Although several stain separation methods exist, they generally need to be adapted for each new analysis. The present deep image prior-based stain deconvolution method may be advantageous for such applications because it can automatically perform stain deconvolution for multiple stains, potentially more than two stains, without the need for training on prior images and without ground truth labels, thereby removing the need for skilled operators to spend significant time tuning image processing algorithms.

[0095]

[0023] Referring now to Figure 8, an exemplary system for performing stain deconvolution is shown. Control and processing hardware 500 is used to process images received from a scanning device, such as a slide scanner 600 (e.g., a light source, a slide support, and an imaging camera) to perform stain deconvolution. As shown in Figure 8, in one embodiment, the control and processing hardware 500 may include a processor 510, a memory 520, a system bus 505, one or more input / output devices 530, and a number of optional additional devices, such as a communication interface 560, a display 540, external storage 550, and a data acquisition interface 570.

[0096] An exemplary method of the present invention for performing deep image prior based stain convolution may be implemented via processor 510 and / or memory 520. As shown in Figure 8, the process of training a stain deconvolution network based on a color target image and an appropriate loss function may be implemented by control and processing hardware 500 via executable instructions represented as a stain deconvolution module 580. Similarly, a stain normalization method may be implemented by control and processing hardware 500 via executable instructions represented as a stain normalization module 590.

[0097] The functionality described herein may be implemented partially through hardware logic in the processor 510 and partially using instructions stored in memory 520. Some embodiments may be implemented using the processor 510 without additional instructions stored in memory 520. Some embodiments are implemented using instructions stored in memory 520 for execution by one or more general-purpose microprocessors. In some exemplary embodiments, customized processors such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) may be used. Thus, the present disclosure is not limited to any particular configuration of hardware and / or software.

[0098] 8, it should be understood that the illustrated exemplary system is not intended to be limiting to the components that may be used in a given implementation. For example, the system may include one or more additional processors. Additionally, one or more components of the control and processing hardware 500 may be provided as external components interfaced to a processing device.

[0099] While some embodiments may be implemented as fully functional computers and computer systems, various embodiments may be distributed as computing products in a variety of forms and may be applied without regard to the particular type of machine or computer-readable medium used to actually effect the distribution.

[0100] At least some aspects disclosed herein may be embodied, at least in part, in software, i.e., the techniques may be executed in a computer system or other data processing system in response to a processor thereof, such as a microprocessor, executing sequences of instructions contained in a memory, such as a ROM, volatile RAM, non-volatile memory, cache or remote storage device.

[0101] A computer readable storage medium can be used to store software and data that, when executed by a data processing system, causes the system to perform various methods. The executable software and data can be stored in a variety of locations including, for example, ROM, volatile RAM, non-volatile memory, and / or cache. Portions of this software and / or data can be stored in any of these storage devices. As used herein, the phrases "computer readable material" and "computer readable storage medium" refer to all computer readable media except for the transitory, propagating signals themselves. EXAMPLES

[0102] The following examples are presented to enable those skilled in the art to understand and practice embodiments of the present disclosure, and should not be construed as limitations on the scope of the disclosure, but merely as illustrative and representative thereof.

[0103] Example 1: Implementation of a deep image prior-based staining deconvolution network In some exemplary implementations, each autoencoder neural network of the dyed deconvolution network can be implemented according to the design of U-Net, where a U-shaped architecture consisting of a downsampling encoder and an upsampling decoder is used to generate the target image. As mentioned above, in some exemplary implementations, the input image is random noise, and the size of the convolution filter is 5×5. The encoder consists of five convolution layers, each of which is followed by Lanczos downsampling, batch normalization, and LeakyReLU activation. The decoder consists of five convolution layers, each of which is followed by bilinear upsampling, batch normalization, and LeakyReLU activation, except for the last layer, which is activated by sigmoid activation. The scale factor for upsampling and downsampling is 2, and the input of each layer is padded accordingly. Furthermore, the fourth and fifth layers of the encoder are skip-connected to the second and first layers of the decoder, respectively.

[0104] The model was implemented on a consumer-level graphics processing unit (GPU), with most experiments run on a GTX 1660 Super GPU. We believe prediction speeds of a few seconds can be achieved on a datacenter-level GPU using optimized code.

[0105] Using the algorithm of this example, stain deconvolution was performed on the MIDOG dataset (https: / / imi.thi.de / midog / the-challgenge / ) using 200 high-power field images (approximately 7000 x 5000 pixels each) generated by imaging 50 breast cancer tissue samples using four different whole slide image scanners (scanner 1: Hamamatsu XR nanozoomer 2.0, scanner 2: Hamamatsu S360 (0.5NA), scanner 3: Aperio ScanScope CS2, scanner 4: Leica GT450). Locations were randomly selected at x=2000:2512, y=2000:2512 for every high-power field image to create 200 512 x 512 cropped image tiles to evaluate the method of this example. The physical parameters estimated for each image are visualized in two-dimensional space using t-SNE. The reduced representation of the physical parameters formed four clusters, clearly corresponding to the type of scanner used to image the samples.

[0106] The present exemplary embodiment of the deep image prior-based stain deconvolution method explicitly encodes a list of variables in its network structure to describe the physical parameters involved in the light transmission from the stained tissue sample to the scanner camera. Compared to conventional stain deconvolution methods that output stain concentration maps and stain vectors using conventional deep learning-based approaches, the present method outputs more variables that more clearly characterize nonlinearities and background illumination, thereby improving interpretability.

[0107] Example 2: Theoretical basis of dye spectrum correction factors Based on the Beer-Buguerre-Lambert equation, the linear relationship between absorbance A and concentration c is defined as follows:

number

[0108] In practice, if the stained image is captured with a non-monochromatic device, such as an RGB color camera, Equation 1 may lead to systematic errors due to the non-stained, and therefore non-linear, formation of the optical signal. To address the non-linearity, the red, green, and blue spectral bands are used.

number

[0109]

number

number

number

[0110] In the formula, λ1 and λ2 are the lower and upper limits of the visible wavelength range,

number

number

[0111] If the device and environment are assumed to be fixed relative to the target image when the image is captured, the absorbance A can be written as a function of concentration c as follows:

[0112]

number

[0113] f λ Although the parameters of f cannot be measured, they are consistent for pixels with the same intensity level. Therefore, we use the proposed staining deconvolution neural network to estimate the f λ By estimating the value of f λ It is possible to approximate f as a function of c using polynomials. λ By modeling It is possible to reduce systematic errors due to nonlinearity. In digital pathology image analysis, there are few density levels associated with each staining, but the results of this experiment show that for each type of staining, λ We show that even when is approximated as a constant floating-point number (i.e., a constant), hereafter referred to as the staining spectrum correction factor, the deconvolution results are significantly improved.

[0114] As shown in Figure 10, estimating the nonlinear color signal using a linear model leads to significant errors (difference between the blue and red curves). By introducing a spectral correction factor, the absorbance model can be adapted to approximate the underlying nonlinear signal formation. In the rightmost graph of Figure 10, an example is shown of how a float (constant parameter) as a spectral correction factor can change the slope of the linear model to approximate the concentration level of interest (orange circle), e.g., the actual absorbance value of the target cell nucleus of the stain.

[0115] Example 3: Example of dye deconvolution using an embodiment of a deep image prior-based dye deconvolution network In this example, deconvolution results of images stained with three different combinations of dyes are shown using an exemplary implementation of the proposed algorithm from Example 1. Specifically, the model was tested on hematoxylin and eosin (H&E) stained breast tissue images, and 3,3'-diaminobenzidine (DAB) stained bone marrow tissue images as well as SMA CD34 MF3 antibody stained breast tissue images, with the results shown in rows 1, 2 and 3 of FIG.

[0116] This implementation of the proposed staining deconvolution network successfully deconvolved all three staining combinations into a concentration map, a color vector, and a background correction vector, as shown in Figure 11. A generated image is created by combining all the deconvolution components, which is compared to the target image to calculate a loss function during training. A background illumination corrected version of the target image is obtained by combining all the components except the background correction vector. It is also possible to perform quantification of staining by calculating the ratio of positively stained pixels to total pixels.

[0117] The performance of the staining convolutional network embodiment was also compared on DAB stained (DAB is a light scatterer) bone marrow tissue images with and without the spectral correction factor (rows 4 and 5 in FIG. 11 , respectively). The staining deconvolution with spectral correction correctly estimated the color of DAB dye to be brown (row 4), while the linear model (without the spectral correction factor) yields a clearly erroneous pink color vector for DAB (row 5). This example shows that incorporating the spectral correction factor improves the performance of the staining deconvolution network for light scatterers.

[0118] The stain deconvolution performance was also compared with two state-of-the-art conventional stain deconvolution algorithms, Masenko's algorithm and Bahadane's algorithm. First, tissue samples were stained with DAPI fluorescent imaging to obtain a gold standard (concentration map) of the hematoxylin-sensitive tissue layer. Then, the samples were washed and re-stained with hematoxylin and eosin (H&E). The DAPI images were registered to the corresponding H&E images and Otsu's thresholding was performed to obtain the ground truth. All three methods were applied to separate the stains and the resulting hematoxylin concentration maps were used for quantitative comparison. The results are shown in Figures 12 and 13. Two metrics were used for the comparison: point biserial correlation to measure the general correlation (between the concentration map and the ground truth) of high and low hematoxylin concentrations, and structural similarity. As can be seen from Figs. 12 and 13, our method is at least at the same level as the comparison methods in terms of point biserial correlation, and is significantly better at preserving local structure compared to the other two methods.

[0119] Example 4: Example of stain normalization using an embodiment of a deep image prior-based stain deconvolution network In this section, we provide an example of stain normalization using an embodiment of the proposed stain deconvolution network. Stain normalization was performed using the same embodiment as in Example 1 and on the same dataset: the MDOG dataset (https: / / imi.thi.de / midog / the-challgenge / ).

[0120] Figure 14 shows example results of stain normalization using the proposed network. Four image patches with different contents from four different scanners (scanner 1: Hamamatsu XR nanozoomer 2.0, scanner 2: Hamamatsu S360 (0.5NA), scanner 3: Aperio ScanScope CS2, scanner 4: Leica GT450) were selected for stain normalization. The density maps and stain vectors of each image were calculated using the stain normalization network and mixed-and-matched to generate all 16 possible combinations of style and content. For example, the first row shows the simulated appearance of content 1 scanned by the four scanners, while the second column shows the simulated appearance of all contents scanned by scanner 2. It is also possible to normalize image patches scanned by different scanners using a predefined "standard" color vector to make them appear similar in style and thus improve the performance of downstream analyses.

[0121] While the specific embodiments described above are shown by way of example, it should be understood that these embodiments are susceptible to various modifications and alternative forms. It is to be further understood that the claims are not intended to limit the invention to the particular forms disclosed, but rather are intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the disclosure.

Claims

1. A method for performing staining deconvolution on a color target image, To provide a stained inverse convolution network, the stained inverse convolution network is A plurality of convolutional autoencoder neural networks, each convolutional autoencoder neural network configured to process its respective input dataset to generate its respective output dataset, each input dataset comprising its respective two-dimensional array having dimensions equal to the pixel dimensions of the color target image, An absorbance calculation module configured to generate a color absorbance image using an absorbance model, wherein the color absorbance image is generated by processing a plurality of staining vectors and a plurality of staining density maps, and each staining vector and each staining density map is associated with its respective stain; Includes, The absorbance calculation module is operably coupled to the plurality of convolutional autoencoder neural networks, thereby obtaining each staining concentration map from the output dataset of each convolutional autoencoder neural network. The provision of the above, Training the stained deconvolution network according to a stained deconvolution loss function, wherein the stained deconvolution loss function is A first loss component configured to minimize the generation loss associated with the aforementioned color target image, A second loss component configured to facilitate separation between the aforementioned staining concentration maps, The training includes, Includes, As a result, after the training, each of the stain density maps represents the inverse convolution stain density map of the stain in the color target image. method.

2. The absorbance calculation module is further configured such that the color absorbance image is generated by calculating the sum of the products of the staining vector, the staining density map, and the spectral correction coefficient for each stain. Each stain has an associated staining spectral correction factor. Each staining spectral correction coefficient is updated during training in accordance with the minimization of the staining inverse convolution loss function. The method according to claim 1.

3. The method according to claim 2, wherein the spectral correction coefficient is defined according to the staining spectral correction coefficient parameter of the inverse convolutional network, and the staining spectral correction coefficient parameter is initialized before the training.

4. The method according to claim 2, wherein the stained deconvolutional network comprises a plurality of spectral correction neural networks, each spectral correction neural network configured to determine its own stained spectral correction coefficient, and each spectral correction neural network is trained according to the stained deconvolutional loss function.

5. The method according to claim 4, wherein at least one spectral correction neural network of the stained deconvolutional network is an encoder-decoder network.

6. The method according to claim 5, wherein the staining spectral correction coefficient corresponding to the at least one spectral correction neural network is determined according to the latent features of the at least one spectral correction neural network.

7. The absorbance calculation module is further configured such that the calculation of the color absorbance image includes a color background vector. The aforementioned color background vector is updated during training in accordance with the minimization of the staining inconvolution loss function. The method according to any one of claims 1 to 6.

8. The method according to claim 7, wherein the color background vector is defined according to the background parameters of the stained inverse convolution network, and the background parameters are initialized before the training.

9. The method according to claim 7, wherein the stained deconvolutional network includes a background neural network, the background neural network is configured to determine the color background vector, and the background neural network is trained according to the stained deconvolutional loss function.

10. The method according to claim 9, wherein the background neural network is an encoder-decoder network.

11. The method according to claim 10, wherein the value of the color background vector is determined according to the latent features of the background neural network.

12. The method according to any one of claims 1 to 6, wherein the staining vector is updated during training in accordance with the minimization of the staining inconvolution loss function.

13. The method according to claim 12, wherein the staining vector is defined according to staining vector parameters stored in the staining deconvolutional network, and the staining vector parameters are initialized before the training.

14. The method according to claim 12, wherein the stained deconvolutional network comprises a plurality of stained vector neural networks, each stained vector neural network configured to determine its respective stained vector, and each stained vector neural network is trained according to the stained deconvolutional loss function.

15. The method according to claim 14, wherein at least one stained vector neural network of the stained deconvolutional network is an encoder-decoder network.

16. The method according to claim 15, wherein the value of the staining vector corresponding to at least one staining vector neural network is determined according to the latent features of the at least one staining vector neural network.

17. The method according to claim 14, wherein during at least the initial portion of the training, the staining deconvolution loss function includes an additional loss term based on the difference between the staining vector calculated by the staining vector neural network and a predetermined initial value of the staining vector.

18. The method according to claim 17, wherein the additional loss term is included in the stained deconvolution loss function during the initial portion of the training, but is not present in the stained deconvolution loss function during the subsequent portion of the training.

19. The method according to claim 12, further comprising performing normalization using the staining vector when processing different color target images.

20. The method according to any one of claims 1 to 6, further comprising generating an output image using the color absorbance image, thereby providing a regenerated version of the color target image.

21. The method according to any one of claims 1 to 6, wherein the first loss component is based on the difference between the color absorbance image and the target color absorbance image generated from the color target image.

22. The method according to any one of claims 1 to 6, wherein the first loss component is based on the difference between the color target image and the output image generated based on the color absorbance image.

23. The method according to any one of claims 1 to 6, wherein at least one input dataset is randomly generated.

24. The method according to any one of claims 1 to 6, wherein at least two of the input datasets are common input datasets.

25. The method according to any one of claims 1 to 6, wherein at least one of the convolutional autoencoder neural networks includes skip connections.

26. The method according to any one of claims 1 to 6, wherein during at least the initial portion of the training, the calculation of the loss function is extended using at least one transformation of the input dataset.

27. The method according to any one of claims 1 to 6, wherein the color target image is a first color tile of a main color image, the staining vector obtained after training is a final staining vector, and the method further comprises using the final staining vector when performing a deconvolution of staining another color tile of the main color image.

28. The method according to claim 27, wherein the final staining vector is used to initialize the staining vector when performing a deconvolution of staining other tiles of the main color image.

29. The method according to any one of claims 1 to 6, further comprising performing quantification of staining using the staining concentration map.

30. A system for performing staining deconvolution on a color target image, the system comprising a control and processing circuit including at least one processor and memory, the memory being The method involves generating a stained inverse convolution network, wherein the stained inverse convolution network is A plurality of convolutional autoencoder neural networks, each convolutional autoencoder neural network configured to process its respective input dataset to generate its respective output dataset, each input dataset comprising its respective two-dimensional array having dimensions equal to the pixel dimensions of the color target image, An absorbance calculation module configured to generate a color absorbance image using an absorbance model, wherein the color absorbance image is generated by processing a plurality of staining vectors and a plurality of staining density maps, and each staining vector and each staining density map is associated with its respective stain; Includes, The absorbance calculation module is operably coupled to the plurality of convolutional autoencoder neural networks, thereby obtaining each staining concentration map from the output dataset of each convolutional autoencoder neural network. The above-mentioned generation, Training the stained deconvolution network according to a stained deconvolution loss function, wherein the stained deconvolution loss function is A first loss component configured to minimize the generation loss associated with the aforementioned color target image, A second loss component configured to facilitate separation between the aforementioned staining concentration maps, The training includes, Includes, As a result, after the training, each of the stain density maps represents the inverse convolution stain density map of the stain in the color target image. The instructions include instructions that can be executed by at least one processor to perform an operation, system.

Images