Systems and methods for performing stain deconvolution

Abstract

A stain deconvolution method and system employing a depth image priori based neural network structure and associated training schemes are disclosed. An example stain deconvolution network employs an autoencoder network to generate a stain concentration map of a plurality of stains associated with a color target image without training based on a large reference image dataset. The stain deconvolution network is trained using a loss function that facilitates correct image generation and separation between the stain concentration maps generated by the autoencoder network. The depth image priori-based stain deconvolution network may be configured to encode an adapted physical model that includes a set of parameters that model background illumination and non-linear dependency of absorption on concentration and wavelength. This example method of stain deconvolution, which may be performed without previous training data, may thus be generalized to accommodate multiple stains and previously uncharacterized stain types, and may be expanded to perform stain normalization.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 330,419, filed on April 13, 2023, entitled “SYSTEMS AND METHODS FOR PERFORMING STAIN DECONVOLUTION,” the entire contents of which are incorporated herein by reference. Background Art

[0003] The present disclosure relates to deconvolution of stain components from color images, such as digital pathology tissue images.

[0004] Histopathology is a diagnostic discipline based on the visual interpretation of cellular biological features captured in images using stains that specifically bind to their target antigens. The advent of digitized images in pathology has propelled this traditional field into what is now described as digital pathology, where pathological information from stained slides is acquired, stored, and managed to create large-scale datasets for disease diagnosis, biological research, and drug development. Digital images are suitable for computational pathology, both for basic measurements and counts as well as for advanced machine learning tasks. Images can now be evaluated through machine learning to derive features beyond traditional histopathology assessments, such as connecting images directly to clinical data (e.g., prognosis, mutations).

[0005] Therefore, performing stain deconvolution on multi-stained images is an essential step in most histological image analysis algorithms. Stain deconvolution is the process of transforming a stained tissue section image from a normal RGB color space into a series of stain channels. Each stain channel is a grayscale image that represents the intensity of expression of a specific stain across the original image. Stain deconvolution methods typically attempt to find an ideal stain matrix, i.e., a matrix that produces the desired stain channels when multiplied with the RGB color channels. The stain matrix consists of stain vectors (also called "color vectors" or "stain color vectors"), each of which represents the model color of a specific stain in the original image and provides a correspondence between the contribution of a specific stain to each color channel and the absorbance (optical density).

[0006] A common problem in tissue sample analysis is undesired color variation due to differences in the color response of slide scanners, raw materials and manufacturing techniques of stain suppliers, and staining protocols of different pathology laboratories. This makes image interpretation difficult for software trained on the appearance of a specific stain, and further increases the inter- and intra-expert diagnostic and labeling variability that already exists between pathologists. This variation can be corrected by performing stain deconvolution and then recombining the stain channels using a reference template. This is called stain normalization. Summary of the invention

[0007] A dye deconvolution method and system using a neural network structure based on a deep image prior and an associated training scheme are disclosed. An example dye deconvolution network uses an autoencoder network to generate dye concentration maps for multiple dyes associated with a color target image without the need for training based on a large reference image dataset. The dye deconvolution network is trained using a loss function that promotes correct image generation and separation between the dye concentration maps generated by the autoencoder network. The dye deconvolution network based on a deep image prior can be configured to encode an adapted physical model that includes a set of parameters that model background illumination and the nonlinear dependence of absorption on concentration and wavelength. The present example method of dye deconvolution, which can be performed without prior training data, can therefore be generalized to accommodate multiple dyes and previously uncharacterized dye types.

[0008] Therefore, in a first aspect, there is provided a method of performing colorant deconvolution on a color target image, the method comprising:

[0009] A colorant deconvolution network is provided, the colorant deconvolution network comprising:

[0010] a plurality of convolutional autoencoder neural networks, each convolutional autoencoder neural network configured to process a respective input data set to generate a respective output data set, each input data set comprising a respective two-dimensional array having a size equal to a pixel size of the color target image;

[0011] an absorbance calculation module configured to generate a color absorbance image using an absorbance model, wherein the color absorbance image is generated by processing a plurality of dye vectors and a plurality of dye concentration maps, wherein each dye vector and each dye concentration map is associated with a corresponding dye;

[0012] The absorbance calculation module is operatively coupled to the plurality of convolutional autoencoder neural networks such that each stain concentration map is obtained from an output dataset of a corresponding convolutional autoencoder neural network; and

[0013] A colorant deconvolution network is trained according to a colorant deconvolution loss function, wherein the colorant deconvolution loss function includes: a first loss component, which is configured to minimize a generation loss associated with a color target image; and a second loss component, which is configured to promote separation between colorant concentration maps; so that after training, the colorant concentration map correspondingly represents a deconvolved colorant concentration map of a colorant within the color target image.

[0014] In some implementations of the method, the absorbance calculation module is also configured to generate a color absorbance image by calculating the sum of the products of the dye vector, the dye concentration map and the dye spectral correction factor for each dye; wherein each dye has an associated dye spectral correction factor; and wherein each dye spectral correction factor is updated during training according to the minimization of the dye deconvolution loss function.

[0015] The dye spectral correction factor may be defined according to a dye spectral correction factor parameter of a dye deconvolution network, and wherein the dye spectral correction factor parameter is initialized prior to training.

[0016] The dye deconvolution network may include a plurality of spectral correction neural networks, each spectral correction neural network is configured to determine a corresponding dye spectral correction factor, and each spectral correction neural network is trained according to a dye deconvolution loss function. At least one spectral correction neural network of the dye deconvolution network may be an encoder-decoder network. The dye spectral correction factor corresponding to the at least one spectral correction neural network may be determined according to the latent features of the at least one spectral correction neural network.

[0017] In some implementations of the method, the absorbance calculation module is further configured such that the calculation of the color absorbance image includes a color background vector; wherein the color background vector is updated during training according to the minimization of the dye deconvolution loss function. The color background vector may be defined according to background parameters of the dye deconvolution network, and wherein the background parameters are initialized prior to training. The dye deconvolution network may include a background neural network configured to determine the color background vector, and wherein the background neural network is trained according to the dye deconvolution loss function. The background neural network may be an encoder-decoder network. The value of the color background vector may be determined according to a latent feature of the background neural network.

[0018] In some implementations of the method, the dye vector is updated during training according to the minimization of a dye deconvolution loss function. The dye vector may be defined according to dye vector parameters stored within the dye deconvolution network, and wherein the dye vector parameters are initialized prior to training. The dye deconvolution network may include a plurality of dye vector neural networks, each dye vector neural network configured to determine a corresponding dye vector, and wherein each dye vector neural network is trained according to the dye deconvolution loss function. At least one dye vector neural network of the dye deconvolution network may be an encoder-decoder network. The value of the dye vector corresponding to at least one dye vector neural network may be determined according to the latent features of the at least one dye vector neural network.

[0019] In some implementations of the method, during at least an initial portion of training, the colorant deconvolution loss function includes an additional loss term based on a difference between a colorant vector calculated by the colorant vector neural network and a predetermined initialization value of the colorant vector. The additional loss term may be included in the colorant deconvolution loss function during the initial portion of training and not present in the colorant deconvolution loss function during a subsequent portion of training.

[0020] In some implementations, the method further includes employing the colorant vector to perform normalization when processing different color target images.

[0021] In some implementations, the method further includes using the color absorbance image to generate an output image to provide a reconstructed version of the color target image.

[0022] In some implementations of the method, the first loss component is based on a difference between the color absorbance image and a target color absorbance image generated from the color target image.

[0023] In some implementations of the method, the first loss component is based on a difference between a color target image and an output image generated based on the color absorbance image.

[0024] In some implementations of the method, at least one input data set is randomly generated.

[0025] In some implementations of the method, at least two of the input data sets are common input data sets.

[0026] In some implementations of the method, at least one of the convolutional autoencoder neural networks includes a skip connection.

[0027] In some implementations of the method, during at least an initial portion of the training, at least one transformation of the input data set is used to enhance the computation of the loss function.

[0028] In some implementations of the method, the color target image is a first color patch of the main color image, and wherein the colorant vector obtained after training is a final colorant vector, the method further comprising using the final colorant vector when performing colorant deconvolution on another color patch of the main color image. When performing colorant deconvolution on another color patch of the main color image, the final colorant vector may be used to initialize the colorant vector.

[0029] In some implementations, the method further includes performing dye quantization using the dye concentration map.

[0030] In another aspect, a system for performing colorant deconvolution on a color target image is provided, the system comprising:

[0031] Control and processing circuitry comprising at least one processor and a memory, the memory comprising instructions executable by the at least one processor to perform operations comprising:

[0032] A dye deconvolution network is generated, wherein the dye deconvolution network comprises:

[0033] a plurality of convolutional autoencoder neural networks, each convolutional autoencoder neural network configured to process a respective input data set to generate a respective output data set, each input data set comprising a respective two-dimensional array having a size equal to a pixel size of the color target image;

[0034] an absorbance calculation module configured to generate a color absorbance image using an absorbance model, wherein the color absorbance image is generated by processing a plurality of dye vectors and a plurality of dye concentration maps, wherein each dye vector and each dye concentration map is associated with a corresponding dye;

[0035] The absorbance calculation module is operatively coupled to the plurality of convolutional autoencoder neural networks such that each stain concentration map is obtained from an output dataset of a corresponding convolutional autoencoder neural network; and

[0036] The dye deconvolution network is trained according to a dye deconvolution loss function, which includes:

[0037] a first loss component configured to minimize a generation loss associated with a color target image; and

[0038] a second loss component configured to facilitate separation between dye concentration maps;

[0039] Such that after training, the dye concentration map accordingly represents the deconvolved dye concentration map of the dye within the color target image.

[0040] A further understanding of the functionality and advantages of the present disclosure may be gained by referring to the following detailed description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Embodiments of the present invention will now be described by way of example and with reference to the accompanying drawings, in which:

[0042] Figure 1 A conventional matrix decomposition based dye color deconvolution method is schematically shown.

[0043] Figure 2 An example of a conventional deep learning method for dye normalization is schematically shown.

[0044] Figure 3A An example implementation of a dye deconvolution network based on the use of multiple depth image priors for determining a dye concentration map is shown.

[0045] Figure 3B Shown with Figure 3A An example implementation of a suitable autoencoder network for use with an example colorant deconvolution network.

[0046] Figure 4 An example of a deep image prior based colorant deconvolution network that refines colorant vectors during training of an autoencoder neural network is shown.

[0047] Figure 5 Different example methods for refining colorant vectors during training of a colorant deconvolution network based on deep image priors are schematically illustrated.

[0048] Figure 6 An example implementation of a dye deconvolution network comprising an autoencoder neural network employed to generate a dye concentration map associated with two dyes is shown.

[0049] Figure 7 An example method is shown in which a deep image prior based color deconvolution network is adopted to perform color normalization.

[0050] Figure 8 An example system for performing stain deconvolution is shown.

[0051] Fig. 9 Unsupervised clustering results of the physical parameters estimated by the stain deconvolution network matched to the scanner types in the example dataset are shown.

[0052] Fig.10 It is shown how dye spectral correction factors can reduce systematic errors in color deconvolution compared to traditional methods.

[0053] Fig.11 Shown are the results of stain deconvolution in three types of digital pathology images and examples of how the spectral correction factor improves the stain deconvolution performance.

[0054] Fig.12 A comparison of the proposed method with the state-of-the-art stain deconvolution methods for color deconvolution of H&E images is shown, where the proposed method achieves the highest point-wise biserial correlation.

[0055] Fig.13 A comparison of the proposed method with the state-of-the-art stain deconvolution methods for color deconvolution of H&E images is shown, where the proposed method achieves the best structural similarity index metric.

[0056] Fig.14 An example of using a dye deconvolution network to perform dye transformation and normalization on a dataset of images captured by 4 different types of scanners is shown. DETAILED DESCRIPTION

[0057] Various embodiments and aspects of the present disclosure will be described with reference to the details discussed below. The following description and accompanying drawings illustrate the present disclosure and should not be construed as limiting the present disclosure. Numerous specific details are described to provide a thorough understanding of the various embodiments of the present disclosure. However, in some cases, in order to provide a brief discussion of the embodiments of the present disclosure, well-known details or conventional details are not described.

[0058] As used herein, the terms "comprises" and "comprising" should be interpreted as inclusive and open-ended rather than exclusive. Specifically, when used in the specification and claims, the terms "comprises" and "comprising" and variations thereof mean including the specified features, steps, or components. These terms should not be interpreted to exclude the presence of other features, steps, or components.

[0059] As used herein, the term "exemplary" means "serving as an example, instance, or illustration," and should not be construed as preferred or advantageous over other configurations disclosed herein.

[0060] As used herein, the terms "about" and "approximately" are intended to cover variations that may exist within the upper and lower limits of a range of values, such as variations in properties, parameters, and dimensions. Unless otherwise indicated, the terms "about" and "approximately" mean ±25% or less.

[0061] It should be understood that, unless otherwise indicated, any specified range or group is used as a shorthand way to refer individually to each member of the range or group, and each possible subrange or subgroup encompassed therein, and the same applies to any subrange or subgroup therein. Unless otherwise indicated, the present disclosure is directed to and expressly combines each specific member and combination of subranges or subgroups.

[0062] As used herein, the term "approximately" when used in connection with a quantity or parameter refers to a range spanning from about one-tenth to ten times the quantity or parameter.

[0063] Despite the strong interest in dye deconvolution methods, their general applicability to specific image analysis problems is often limited. Traditional dye deconvolution methods rely on matrix decomposition algorithms based on the Beer-Lambert Law, which relates the attenuation of non-scattered light to the properties of the dye used for staining.

[0064] Figure 1 Conventional methods for dye deconvolution are shown in . Conventional methods convert the color deconvolution problem into a matrix decomposition problem. After preprocessing steps such as resizing and background illumination correction, a Beer-Lambert transform is applied to convert the image intensity values ​​observed in each RGB channel into optical density, and then matrix decomposition is used to recover the dye concentration map and dye color vector for each dye. In order to achieve better performance than the baseline non-negative matrix factorization (NMF) or singular value decomposition (SVD), traditional physics-based methods rely on manually created priors (e.g., criteria for sparsity, structural shape, histogram / spectral matching), but these priors are laborious, heuristic, and dye-specific. The color absorption properties of the dye (dye vector) are pre-defined by empirical observations or estimated using template matching based on manually selected regions of interest (ROIs).

[0065] While most stain deconvolution methods remain relatively robust for hematoxylin and eosin (H&E) stained images, their applicability to specialized stains is severely compromised. In fact, stain deconvolution for other stains relies on problem-specific designs, which require a lot of domain expertise and experience for one set of stains and rarely translate to other sets of stains.

[0066] Specifically, the assumption of the Beer-Lambert law often fails in practice in stain deconvolution based on stains other than H&E, which often leads to severe deconvolution errors. In addition, with the rapid development of digital pathology, special stains are becoming increasingly popular due to their ability to reveal detailed information and structures beyond what traditional H&E stains can provide, which improves disease diagnosis and treatment.

[0067] The recent success of deep learning has shifted the focus of dye deconvolution methods to neural network-based methods and shed light on dye deconvolution of special dyes. Figure 2 An example of a conventional deep learning method for stain normalization is shown. Instead of explicitly computing concentration maps and stain vectors, deep learning methods typically perform image-to-image translation directly (converting images of one stain type / source to another type / source). To achieve this, large image datasets from two sources are required. Similar to conventional physics-based methods, the images are first preprocessed and then fed into two neural networks, leveraging the powerful learning capabilities of neural networks to simultaneously learn color transformations and imaging structures. In addition to image generation branches (GANs, VAEs, etc.) for image reconstruction or image-to-image translation, deep learning-based methods typically incorporate auxiliary branches to gain performance gains from solving another related task (e.g., tissue type clustering). Deep learning-based methods generally have better performance than traditional methods because their color separation / style transfer does not rely on manually created priors and manually selected ROIs.

[0068] However, deep learning based techniques suffer from poor interpretability and the normalization procedure requires a template image from the target domain. Although existing deep learning based methods claim to be unsupervised as they do not require separate color layer annotations, they rely on curated large image datasets from different domains and are thus semi-supervised / weakly supervised.

[0069] In addition, while the power of deep learning enables dye deconvolution methods to circumvent the limitations of the Beer-Lambert law, their accessibility and generalizability are often problematic. Although existing deep learning-based methods are called "unsupervised" because they do not require the true values ​​of individual dye layers for training, they require large manually selected datasets, data curation, and a high level of expertise in network design and training. The data-driven nature of existing deep learning-based dye deconvolution methods makes them very powerful on the data used for their training, but on the other hand reduces their performance on unseen data. The lack of a universally applicable dye deconvolution algorithm is an obstacle to the development of digital pathology analysis.

[0070] The inventors have realized that the limitations of conventional deep learning methods for dye deconvolution can be circumvented by adopting a deep learning framework that does not require pre-training on large image datasets. In fact, by adopting a deep learning framework based on deep image priors, accurate dye deconvolution can be performed using only the image data itself to train the network, rather than relying on a large training dataset.

[0071] Deep Image Prior (DIP) neural networks were originally proposed for inverse image problems such as denoising and inpainting, based on the concept that the structure of neural networks (especially convolutional filters) can capture the imaging statistics of natural images without being trained for a specific task. Deep Image Prior neural networks are trained with randomly initialized network parameters to take random noise as input to the neural network and generate output images based on a loss function involving the target image. During the training process, the network's convolutional filters capture internal patch recurrences (also called image priors) and use them as basic building blocks for reconstructing the target image, so that external image information (such as noise or artifacts) can be removed, thereby helping to recover a denoised or corrected version of the target image.

[0072] Deep image prior based neural networks including two deep image prior networks (e.g., so-called “double DIP” networks) have also been implemented, based on the observation that when multiple deep image prior based neural networks are combined to reconstruct an image, each deep image prior based neural network tends to separate the image into important decomposed parts, such as background and foreground, hazy and dehazed images, or overlapping images with different degrees of transparency.

[0073] The inventors have realized that a multi-deep image prior neural network can be used to perform stain deconvolution, wherein a set of autoencoder networks are used to generate stain concentration maps for multiple stains associated with a color target image without the need for training based on a large reference image dataset. Instead, the inventors have reasoned that a multi-deep image prior network can be trained using a loss function that employs: (i) a loss component that encourages correct image generation (based on the color target image), and (ii) a loss component that encourages separation between stain concentration maps generated by the autoencoder networks.

[0074] As will be described in detail below, unlike previous dye deconvolution methods, the dye deconvolution method of the present invention, which uses a deep image prior neural network structure and an associated training scheme, can be used to perform dye deconvolution on images with any number and any type of dyes. The dye deconvolution network based on the deep image prior of the present invention can also be configured to encode an adapted physical model, thereby providing enhanced interpretability, and the physical model includes a set of parameters describing the properties of background illumination, multi-color dyes, and sensors. Since test-time dye deconvolution can be performed without the need for previous training data, the method of the present invention can be generalized to adapt to new and previously uncharacterized dye types without the risk of overfitting or the need for true value labels. In addition, since the deep image prior-based method of this example does not need to be trained on an image dataset, but only needs to be trained based on the color target image itself, the method can be applied to entry-level graphics processing cards.

[0075] Figure 3A An example implementation of a stain deconvolution network based on the use of multiple depth image priors is shown. The example stain deconvolution network does not require pre-training and is therefore independent of the stain type. A neural network portion 310 containing multiple autoencoder neural networks is employed to generate stain concentration maps for at least two stains, wherein the figure shows stain concentration maps 321 and 322 corresponding to a first stain and a second stain, and one or more optional additional stain concentration maps 323 corresponding to one or more optional additional stains.

[0076] As shown, the dye concentration maps 321-323 are processed together with the corresponding dye vector 330 in the absorbance calculation module 340 to generate a color absorbance image and / or a color intensity image as shown at 350. Although the RGB color space is often used in the examples provided in this disclosure, it should be understood that other color spaces, such as but not limited to HSV and LAB, may also be used in the alternative.

[0077] The deconvolution is performed on the color target image, which is a two-dimensional image with dimensions LxW and three color channels, so this can be represented by a matrix of dimensions 3xLxW. Therefore, each stain concentration map has dimensions LxW, and each stain vector is a three-dimensional vector with one dimension for each color channel.

[0078] In the present example method, the absorbance module uses the Beer-Lambert law to generate a color absorbance image based on the calculated dye concentration graphs 321-323. For example, in the case of a color target image corresponding to a sample with n mixed dyes, the absorbance (optical density, OD) is as follows according to the dye vector C k and stain concentration map Sk Generate, where k = 1...n represents the number of dyes:

[0079]

[0080] where OD is a matrix of size 3xLxW representing the color optical density of the color target image (color and pixel index not shown in the equation), and where each dye concentration map S k is a matrix of size LxW that represents the pixel-by-pixel concentration of a given dye (the pixel index is not shown in the equation), and where each dye vector C k is a 3-vector relating the spectral optical density of a given stain to the concentration of the stain (where each dimension of the stain vector corresponds to a different color channel).

[0081] As described in more detail below, this equation is only one example of an absorbance calculation, and other embodiments below disclose alternative absorbance calculation methods that include additional coefficients and / or terms.

[0082] The absorbance (OD) is related to the intensity image by the following equation:

[0083]

[0084] where I is the measured transmitted intensity through the sample, and I 0 is the incident intensity on the sample. This equation can be inverted to generate the final color image from the color absorbance image.

[0085] In a conventional deep learning-based workflow, a neural network is trained on a training image dataset to fix the activations and weights of the neural network. The trained network is then used to process the target image. However, in stark contrast, Figure 3A The dye deconvolution network shown does not need to be trained based on a training data set, and the dye deconvolution network can determine the dye concentration map 321-323 of the target color image without providing the target color image as an input to the network. Instead, as described below, the concept of deep image priors can be used to determine the dye concentration map 321-323 based on a suitable loss function involving the color target image.

[0086] like Figure 3A As shown, the neural network portion 310 includes a corresponding autoencoder network 311-313 for each dye, which is used to generate a corresponding dye concentration map 321-323. The autoencoder network 311-313 does not need to be trained before performing dye deconvolution on the color target image.

[0087] Figure 3B A non-limiting example of a suitable autoencoder network 311 is shown in FIG. In this example implementation, skip connections are included between the first layer of the encoder and the fifth layer of the decoder and between the second layer of the encoder and the fourth layer of the decoder. It should be understood that the use of skip connections and the number of skip connections are optional and may vary between implementations of embodiments of the present invention. In addition, it should be understood that other hyperparameters may also vary according to different implementations, including, for example, the number of channels, the number of layers, and the activation type.

[0088] Reference again Figure 3A , each autoencoder network may be provided as input with a random noise dataset 360 (e.g., random uniform noise in the range [-0.5, 0.5]) or any other dataset, including the color target image itself. Although the figure shows that the same input dataset is fed to each autoencoder network, this is only one implementation, and in other example implementations, different input datasets may be fed to the autoencoder networks.

[0089] Although it may seem counterintuitive that a randomly initialized autoencoder neural network can generate accurate dye concentration maps based on random input data that has no connection to the color target image, the autoencoder neural networks 311-313 of the example dye deconvolution network are able to achieve this functionality based on the use of a loss function that is based on the color target image and includes a term that enforces separation between the dye concentration maps.

[0090] The loss function for training the neural network portion 310 may include at least the following two loss components. The first loss component is a generation loss component that is used to minimize the generation error. The generation loss component may be configured, for example, to minimize the difference between a color absorbance image generated based on a dye concentration map according to the Beer-Lambert law (hereinafter referred to as a "generated color absorbance image") and an absorbance image calculated from a color target image, or, for example, to minimize the difference between a final image generated based on a dye concentration map (e.g., using the equation shown above) and a color target image. The generation loss may be an L1-type loss. For example, in some example implementations, the generation loss may be calculated as an L1 loss for the absorbance of each color channel, and optionally includes a loss term based on the sum of the absorbances of the three color channels.

[0091] The second loss term provides a repulsion loss between the stain concentration maps to facilitate distinguishing different stain layers, since different stains should highlight different regions of interest.

[0092] For example, in the case of a two-color dye separation problem (i.e., a color target image includes contributions from two dyes), the net loss function may include four generation losses (three per-color channel absorption generation loss terms, one absorption and (for all color channels) generation loss term), and one repulsion loss term that promotes separation between the two dye concentration maps. This example loss function can be generalized to N>3 colors by combining the aforementioned generation loss terms and N(N-1) / 2 repulsion loss terms.

[0093] In some example embodiments, the dye vector 330 is refined during the training of the autoencoder neural network. For example, Figure 4 An example embodiment is shown in which the stain vectors 330 are shown to reside in the neural network portion 310, thereby indicating that their values ​​are updated during training rounds of the autoencoder neural network. The stain vectors can be initialized, for example, according to random initialization values, according to a stain vector input by a user, or according to a stain vector that can be automatically calculated according to a user-selected region of interest in the input image (as a priori knowledge). In the latter case, the network will automatically adjust the user-input stain vector to achieve better separation results.

[0094] In one example implementation, the dye vector 330 may be incorporated into the dye deconvolution network as a parameter that is updated during training. Figure 4 As shown, the colorant vector 330 can be provided as a set of parameters (e.g., six parameters in the example case of two colorants) that are stored and refined during training without a neural network structure. The values ​​of the colorant vector parameters are initialized by initial values ​​(e.g., estimated values ​​or random values), as shown at 370. Figure 5 Case 1 also schematically shows this configuration.

[0095] However, in other example implementations, each dye vector may be generated by a corresponding dye vector neural network structure that resides within the neural network portion 310 and is trained together with the autoencoder network that calculates the dye concentration map. Figure 5 Two example implementations of this embodiment are shown in Cases 2 and 3 of . Case 2 shows an example case where a given colorant vector is calculated according to the feedforward neural network 332, while Case 3 shows an example case where a given colorant vector is calculated according to the encoder-decoder network 334. In the case where the encoder-decoder network 334 is used to generate the colorant vector, the value of the given colorant vector can be generated by, for example, a subset of pixels of the decoded image (including, for example, the center pixel of the decoded image), or, for example, based on latent features generated by the network.

[0096] In the case of an example embodiment where the colorant vector 330 is refined during training of the autoencoder neural network, the loss function can be adjusted to include a loss component related to the colorant vector. For example, a color fixation loss term can be included in the loss function to adjust the colorant deconvolution network during at least a portion of the training.

[0097] In one example implementation, an L1 color fixation loss is calculated between the estimated colorant color vector and the initial value of the colorant vector to constrain the colorant vector during an initial portion of training (e.g., during an initial number of rounds, where the initial number of rounds is less than 2000), and removed during the remainder of training. This initial color fixation loss helps make network training more robust during the initial training period by focusing on learning internal structural patterns before learning the colorant vectors. In the example case of a color target image involving two colorants, two color fixation loss terms may be included, one for each colorant, which may be generalized to one color fixation loss term for each colorant.

[0098] In one example implementation, a color fixation loss conditions the network in the first 25% of the rounds and is removed in the last 75% of the rounds to allow the network to learn the coloring vectors for the coloring.

[0099] In some example implementations, augmentation may be provided during training of a portion of a neural network. Non-limiting examples of augmentation include 90 degree rotation and mirroring operations performed on input image data (e.g., random noise) to help the network learn pose-invariant deep image priors. Augmentation may be included during an initial portion of training (e.g., during the first 75% of training rounds) and disabled during a later portion of training (e.g., during the last 25% of rounds) to assist the network in focusing on deconvolution of colorants.

[0100] The inventors also realized that Figure 3A and Figure 4 The example dye deconvolution network shown can be further tuned to overcome the limitations of conventional dye deconvolution methods.

[0101] In fact, as described in detail in the following examples, although widely used in almost all matrix decomposition-based methods, the assumptions of the Beer-Lambert law are always violated in practice. The Beer-Lambert law assumes monochromatic light (with a single wavelength), but the background illumination, the dye absorption spectrum, the image sensor response, and the observer functions of color matching are all related to wavelength. In addition, the Beer-Lambert law describes color absorbance, and some dyes are scatterers of light. In addition, existing dye deconvolution methods are flawed because the perceived color depends on the spectral dependence caused by other factors that vary between images taken by the same scanner, such as lighting conditions, room temperature, dyeing time, batch effects, and the like.

[0102] Due to the complex spectral response of imaging cameras and light sources, the assumption that the dye vector is independent of the dye concentration does not generally hold. Therefore, dye deconvolution methods based on the dye vector without considering the dependence on the dye concentration produce less accurate concentration results.

[0103] The inventors have realised that a multi-depth image prior dye deconvolution network can be adapted to employ a modified form of the Beer-Lambert law to account for systematic errors caused by differences between the nonlinear concentration dependencies of the dye vectors. In fact, it has been found that the problems associated with this nonlinearity can be addressed by introducing a dye spectral correction factor (an integral form of the Beer-Lambert law) that approximates the Beer-Lambert law at the level of a single image (in contrast to the dataset level used in all prior methods).

[0104] In some example implementations, the Beer-Lambert law is modified to include a stain spectral correction factor that helps accommodate the aforementioned concentration dependence:

[0105]

[0106] Among them, M k is a per-colorant scalar value.

[0107] The dye spectral correction factors approximately compensate for nonlinearities caused by effects such as (but not limited to) light scattering, non-monochromatic light composition, and non-monochromatic light reception by the camera. Although it would be extremely challenging to calculate the complete nonlinear relationship for each dye without a large amount of data, the present example method approximates the nonlinearity based on per-dye scalar parameters that are assumed to be consistent across each image tile. Therefore, the dye spectral correction factors are assumed to remain fixed for one image tile, but may vary even between images from the same whole-slide image, and therefore cannot be calculated using any existing methods.

[0108] However, the method of the present invention using a network based on deep image priors can be easily adapted to determine the dye spectral correction factor during training. k can be stored as parameters in the network that are refined during the training of the autoencoder neural network that generates the stain concentration map. Alternatively, a feed-forward neural network or encoder-decoder network trained according to a loss function can be used to generate the stain spectral correction factors (similar to the different implementations shown for the stain vectors, in Figure 5 ).

[0109] Although the present exemplary embodiment employs a single parameter dye spectrum correction factor for each dye, it will be appreciated that other exemplary embodiments may employ multiple dye spectrum correction parameters for each dye including coefficients defining a nonlinear function of concentration. For example, such parameters may be determined by modeling the dependence of absorption on concentration as a function using a polynomial function form.

[0110] Background illumination may affect the perceived color of images captured by scanners and may vary significantly between different scanners. Existing methods rely on additional preprocessing steps based on Gaussian denoising filters or sampling from background pixels to recover the original image. To achieve optimal performance under such existing methods, preprocessing steps such as background illumination correction and denoising should be performed before color deconvolution. These procedures require histogram matching of manually selected background patches and manually created denoising filters, which need to be designed for each domain (dataset / scanner) and can introduce bias.

[0111] In stark contrast, the aforementioned example embodiments of the present disclosure may be adapted to permit automatic estimation and correction of background illumination and noise without manual intervention. The presence and effects of background illumination may be addressed by including a background correction vector (a 3-element vector, one element per color channel) that takes into account the variation in optical density values ​​in each channel:

[0112]

[0113] The method of the present invention employing a network based on a deep image prior can be easily adapted to determine the background vector during training.

[0114] The background correction vector may be stored as a parameter in the network that is refined during the training of the autoencoder neural network that generates the stain concentration map. Alternatively, a feedforward neural network or encoder-decoder network trained according to a loss function may be employed to generate the background correction vector (again similar to the different implementations shown for the stain vector, in Figure 5 ).

[0115] Figure 6 The dye deconvolution network includes an example embodiment of an autoencoder neural network 311 and 312, which is trained based on color target image data and a suitable loss function to generate dye concentration maps 321 and 322 associated with two dyes. As shown, the neural network portion 310 also includes an autoencoder neural network 313 and 314 trained to determine the dye vector 330. The network portion 310 also includes two optional additional autoencoders 315 and 316, either of which can be optionally included and trained according to the loss function to generate a dye spectrum correction factor 380 and / or a background correction vector 390. Although the figure shows a single autoencoder neural network 315 for generating a dye spectrum correction factor 380, a separate autoencoder neural network can be provided to calculate each dye spectrum correction factor. The network can be expanded by adding the following additional modules for each additional dye: an additional concentration map autoencoder, an additional dye vector autoencoder (or a parameter set or a feedforward network), and an additional dye spectrum correction factor autoencoder (or a parameter set or a feedforward network). Furthermore, the colorant deconvolution network may be augmented, for example, by including additional modules that capture additional components including, but not limited to, ink, color temperature, and white balance.

[0116] In some example embodiments, a deep image prior based dye deconvolution network may be used to perform dye normalization. Dye normalization refers to using a set of standard (or otherwise predetermined) dye vectors in conjunction with a given set of dye concentration maps to obtain images in order to reduce the differences between images obtained using different scanners, thereby reducing the difficulty of downstream analysis.

[0117] Figure 7 Schematically illustrates a method of using a target dye vector 420 obtained by processing a color target image 400 with a dye deconvolution network 410 based on a deep image prior to generate a normalized reconstructed source image 430 from a source image 405 via a Beer-Lambert transform 435 (or a modified form thereof). As shown in the lower portion of the figure, according to the present example embodiment involving the use of multiple deep image prior modules (autoencoders), the dye deconvolution network 410 is trained based on the target color image 400. The training results in the calculation of the target dye vector 420 and the target dye concentration map 440, and the dye deconvolution network can also be configured to determine additional target color parameters 450 associated with the physical model, such as a target dye spectral correction factor and / or a target background vector.

[0118] like Figure 7As shown in the upper portion of , according to this example embodiment involving the use of multiple deep image prior modules (autoencoders), the colorant deconvolution network 410 is also trained based on the source color image 405. The training produces the calculation of the source colorant vector 425 and the target colorant concentration map 445, and if the colorant deconvolution network is so configured, it will also produce additional source color parameters 455, such as source colorant spectral correction factors and / or source background vectors. To generate a normalized reconstructed source image 430, a physical model (e.g., a suitable form of the Beer-Lambert law) is used to process the source colorant concentration map 445 and optional additional source parameters 455 using the target colorant vector 420 instead of the source colorant vector 425.

[0119] Figure 7 Also shown is an example case where the target color vector 420 has been determined so that it is not necessary to perform training of the color deconvolution network in the lower portion of the figure, where these unnecessary operations are shown in dashed lines. In this case, the target color vector 420 can be used instead of the source color vector 425 when generating the normalized reconstructed source image 430.

[0120] In some example implementations, the aforementioned normalization method or a variation thereof may be used to perform stain normalization on whole slide images. When images are scanned and / or processed in different laboratories, the appearance of the images often varies greatly. Pathologists are accustomed to a specific appearance and prefer images to be displayed in similar color and intensity ranges. In addition, many image analysis pipelines and AI models rely on consistent colors to work. In some example implementations, a stain vector obtained after training on one image tile may be used to initialize the stain vector used when performing stain deconvolution on another tile of the image.

[0121] In some example embodiments, stain quantification may be performed on images generated according to the present example stain deconvolution method. For example, a set of stain concentration maps determined by a stain deconvolution network generated according to (or based on) the embodiments disclosed herein may be used to quantify the relative fraction of a given cell type. This may be performed, for example, by identifying and quantifying cells within each stain concentration map, and, for example, by generating a ratiometric measure based on cell counts. Alternatively, quantification may be performed based on pixels to identify stained and unstained tissues, which may be applicable to nuclear, cytoplasmic, or membrane stains.

[0122] In a non-limiting example, the present example method can be used to quantify the abundance of p53 protein in cancerous tissue. Tissue can be stained with primary anti-p53 antibodies, and DAB chromogen can be visualized with secondary antibodies and peroxidase, while counterstaining is performed with hematoxylin. The secondary antibody is connected to peroxidase, which causes the DAB chromogen to produce a visible brown precipitate at the antigen (i.e., p53) site. A stain concentration map of DAB (p53) and hematoxylin (nucleus) is provided by applying a stain deconvolution network according to the present example method. In an example quantification method, in a region of interest, the presence of p53 can be quantified by dividing the number of p53-positive cells stained with DAB by the total number of cells stained with hematoxylin and / or DAB. In another example implementation, quantification can be performed based on pixels (rather than identified cells) within the stain concentration map.

[0123] It should be understood that when performing stain quantification, one or more concentration maps may be thresholded to remove background pixels (e.g., low concentration pixels) so as to remove pixels that are not expected to belong to cells. A pathologist / researcher may determine an appropriate threshold to define a positive pixel, for example, by visual inspection.

[0124] It should be understood that there are many antigens for different applications, and P53 is only one example of an antigen. Other examples of antigens include, for example, antigens that bind to ER and PR receptors in breast cancer, or antigens that highlight different types of inflammatory cells. The distribution of the antigen determines the stain, and two common distribution types are produced by nuclear and membrane stains. P53, ER and PR are all examples of nuclear stains. On the other hand, chromatin determines the color. The choice of chromatin varies according to the application and laboratory, but DAB stain / hematoxylin counterstaining is a common staining combination. As described above, the stain deconvolution embodiment of the present invention, which does not require the presence of a training data set, is suitable for processing known and novel color combinations.

[0125] As described in the Examples section below, the inventors have found that the deep image prior based methods of the present invention can outperform existing deep learning based methods for stain deconvolution of various types of stains. In addition, the inventors have found that the deep image prior based methods of the present invention are unique because they can deconvolute any type of stain without the need for task-specific empirical optimization.

[0126] Without intending to be limited by theory, we believe that the powerful universality of the method based on deep image priors of the present invention comes from the encoding of physical models by deep neural networks based on image priors, which makes them complement and improve each other. Specifically, the deep neural network uses the adjustment of the physical model so that optimization can be achieved based only on the target image itself, which is unprecedented in deep learning-based dye deconvolution. In particular, in the example embodiment of calculating the dye spectral correction factor via the deep learning neural network during the entire network training, the application scope of the basic physical theory can be broadened and the deconvolution error can be significantly reduced.

[0127] The demonstrated strong performance of this example implementation demonstrates that a capable machine learning model can be trained based on the target image itself at test time, which not only avoids labor-intensive and time-consuming data collection and management processes, but also prevents overfitting. The synergy between the physical model and the deep neural network based on deep image priors also solves another problem of neural networks - interpretability. The output of some example dye deconvolution networks disclosed in the present invention includes a set of physical parameters that explicitly describe the conditions and properties involved in the entire light emission process from the dye to the scanner camera. The results shown in the following examples indicate that the present example dye deconvolution network using a modified Beer-Lambert physical model and associated physical parameters enables the dye deconvolution network to perform well on less commonly used dyes, thereby confirming the effectiveness of the modification of the physical model.

[0128] It should also be noted that in addition to dye deconvolution, the method based on deep image priors of the present invention can also automate and realize image denoising. Digital pathology images contain a lot of noise and artifacts. Although most noise seems insignificant to human perception, it is obvious in the "eyes" of the computational classifier. The implementation of the network structure and training scheme based on deep image priors facilitates this denoising capability. The automatic encoder adopted and trained to generate the dye concentration map uses a limited number of convolution filters to capture the internal patch reproduction of the color target image. This network structure and training scheme make it possible to learn the building blocks of regular structures instead of learning irregular noise and small artifacts. In addition, as described above, in the implementation method of solving and modeling the nonlinear dependence of the dye vector on the concentration, and modeling the properties of the background light, these effects can be directly incorporated into the end-to-end training of color deconvolution, thereby allowing a more direct and convenient solution to the essential preprocessing steps.

[0129] In fact, the dye deconvolution method implemented using the deep image prior framework is completely dependent on the information provided by the test image. It should be noted that when the test image quality is low, especially when the general assumption proposed (each dye is uniform in the target image, and the nature of the dye is different) is violated, its performance may decline. For example, when there is a large block of ink with high concentration in the image, some implementations may tend to calculate biased dye vectors and concentration maps to adapt to both dyeing structure and ink as a compromise. Similarly, when the colors of two dyes are too similar, some implementations may not be able to achieve optimal performance when the structure is assigned to the correct dye. However, it should be noted that these aspects are not the result of the design of the network structure itself, but a general challenge to all dye deconvolution methods. In fact, the aforementioned example implementation can be suitable for solving these challenges in the following ways: stacking additional deep image prior modules (auto encoders) and updating the physical model (e.g., adding items and / or parameters to Beer-Lambert's law) to consider large artifacts, and estimating all intermediate physical parameters to achieve better quality control.

[0130] It is therefore apparent that the deep image prior based approach of the present invention can be beneficial in providing a new generation of dye deconvolution methods that can generalize to a wide variety of dyes while requiring neither expert knowledge nor prior data collection.

[0131] The stain deconvolution method based on deep image prior of the present invention can be applied to a variety of environments, including but not limited to clinical pathology, pathological research and drug research. In fact, immunohistochemistry is widely used in drug research laboratories to explore different biomarkers and develop companion diagnostic tests. Such methods involve staining slides with several different dyes that mark different antigens or markers. A very typical example is the use of hematoxylin for nuclear staining and DAB stains that bind to the antigen of interest. Methods for separating these stains are needed to provide more accurate quantitative results. In some research applications, there may be a variety of different stain colors for identifying different structures in tissues. Although there are some stain separation methods, they usually need to adapt to each new determination. The stain deconvolution method based on deep image prior of the present invention may be beneficial to such applications because stain deconvolution can be automatically performed for a variety of stains that can exceed two stains without the need to train the prior image and without the need for true value labels, thereby eliminating the need for skilled operators to spend a lot of time adjusting image processing algorithms.

[0132] Reference now Figure 8, shows an example system for performing stain deconvolution. Control and processing hardware 500 is employed to process images received from a scanning device such as a slide scanner 600 (e.g., a light source, a slide holder, and an imaging camera) to perform stain deconvolution. Figure 8 As shown, in one embodiment, the control and processing hardware 500 may include a processor 510, a memory 520, a system bus 505, one or more input / output devices 530, and a plurality of optional additional devices, such as a communication interface 560, a display 540, an external storage 550, and a data acquisition interface 570.

[0133] The present example method of performing color convolution based on a depth image prior may be implemented via the processor 510 and / or the memory 520. Figure 8 As shown, the process of training a color deconvolution network based on a color target image and a suitable loss function can be implemented by the control and processing hardware 500 via executable instructions represented as a color deconvolution module 580. Similarly, the color normalization method can be implemented by the control and processing hardware 500 via executable instructions represented as a color normalization module 590.

[0134] The functionality described herein may be implemented in part via hardware logic in the processor 510 and in part using instructions stored in the memory 520. Some embodiments may be implemented using the processor 510 without additional instructions stored in the memory 520. Some embodiments are implemented using instructions stored in the memory 520 for execution by one or more general purpose microprocessors. In some example embodiments, a custom processor may be employed, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Therefore, the present disclosure is not limited to a specific configuration of hardware and / or software.

[0135] Reference again Figure 8 It should be understood that the example systems shown in the figures are not intended to be limited to the components that may be employed in a given implementation. For example, the system may include one or more additional processors. In addition, one or more components of the control and processing hardware 500 may be provided as external components that interface with a processing device.

[0136] While some embodiments may be implemented on fully functional computers and computer systems, various embodiments may be distributed as computing products in various forms and may be used regardless of the specific type of machine or computer readable medium used to implement the distribution.

[0137] At least some aspects disclosed herein may be embodied at least in part in software. That is, the technology may be implemented in a computer system or other data processing system in response to its processor (such as a microprocessor) executing sequences of instructions contained in a memory (such as ROM, volatile RAM, non-volatile memory, cache, or remote storage device).

[0138] Computer readable storage media can be used to store software and data, which, when executed by a data processing system, causes the system to perform various methods. Executable software and data can be stored in various places, including, for example, ROM, volatile RAM, non-volatile memory and / or cache. Portions of the software and / or data can be stored in any of these storage devices. As used herein, the phrases "computer readable material" and "computer readable storage medium" refer to all computer readable media other than the transient propagation signal itself.

[0139] Example

[0140] The following examples are presented to enable those skilled in the art to understand and practice the embodiments of the present disclosure. They should not be considered as limiting the scope of the present disclosure, but are merely considered to illustrate and represent the present disclosure.

[0141] Example 1: Example implementation of a colorant deconvolution network based on deep image priors

[0142] In some example implementations, each autoencoder neural network of the colorant deconvolution network can be implemented according to the design of U-Net, in which a U-shaped architecture consisting of a downsampling encoder and an upsampling decoder is adopted to generate the target image. As described above, in some example implementations, the input image is random noise and the convolution filter size is 5x5. The encoder consists of 5 convolutional layers, each followed by Lanczos downsampling, batch normalization, and LeakyReLU activation. The decoder consists of 5 convolutional layers, each followed by bilinear upsampling, batch normalization, and LeakyReLU activation, with the exception of the final layer activated by Sigmoid activation. The scale factor of upsampling and downsampling is 2, and the input of each layer is padded accordingly. In addition, the 4th and 5th layers of the encoder are jump-connected to the 2nd and 1st layers of the decoder, respectively.

[0143] The model was implemented on consumer-grade graphics processing units (GPUs), with most experiments performed on GTX 1660 Super GPUs. It is believed that by optimizing the code, prediction speeds of a few seconds can be achieved on datacenter-class GPUs.

[0144] This example algorithm was adopted to perform stain deconvolution on the MIDOG dataset (https: / / imi.thi.de / midog / the-challgenge / ), which has 200 high-power field images (each image is approximately 7000x5000 pixels) generated by imaging 50 breast cancer tissue samples using 4 different whole-slide image scanners (Scanner 1: Hamamatsu XR nanozoomer 2.0; Scanner 2: Hamamatsu S360 (0.5NA); Scanner 3: Aperio ScanScope CS2; Scanner 4: Leica GT450). For each high-power field image, a site was randomly selected at the position of x=2000:2512, y=2000:2512 to create 200 512x512 cropped image tiles to evaluate this example method. The physical parameters estimated for each image were visualized in two-dimensional space using t-SNE. The simplified representation of the physical parameters forms four clusters, which clearly match the type of scanner used to image the samples.

[0145] This example implementation of a deep image prior based stain deconvolution method explicitly encodes a series of variables in its network structure to describe the physical parameters involved in the light transfer from the stained tissue sample to the scanner camera. When compared to traditional stain deconvolution methods that output stain concentration maps and stain vectors through conventional deep learning based methods, this example method outputs more variables that more clearly characterize nonlinearity and background illumination, thereby providing improved interpretability.

[0146] Example 2: Theoretical basis for the spectral correction factor of the dye

[0147] Based on the Bouguer-Lambert-Beer equation, the linear relationship between absorbance A and concentration c is defined as follows:

[0148]

[0149] Where λ is the wavelength, I is the intensity, and I 0 is the maximum intensity, δ is the molar optical density representing the probability of light absorption for the disease, and c i is the concentration of the ith dye.

[0150] In practice, when a non-monochromatic device (e.g., an RGB color camera) captures a dye image, Equation 1 may produce systematic errors due to the non-color and therefore non-linear formation of the light signal. To account for the non-linearity, the red, green, and blue spectral bands A R , A G , A B The absorbance value of can be modeled as follows:

[0151]

[0152] where λ 1 and λ 2 are the lower and upper visible wavelength regions, τ is the spectral transmittance, s is the spectral sensitivity function of the imaging device, and I 0 (λ) is the spectral intensity distribution of the illumination. According to Equation 2, the concentration c is no longer linearly related to the absorbance A. According to the above model, the relationship between absorbance and concentration cannot be calculated accurately because there are too many nonlinearities, and it is almost impossible to accurately measure the parameters in the integration, such as the spectral sensitivity s of the imaging device to all visible light wavelengths.

[0153] If it is assumed that the device and environment are fixed for the target image when the image is captured, the absorbance A can be written as a function of the concentration c as follows:

[0154] A=f λ (c) (3)

[0155] Although it is not possible to measure f λ , but it is consistent for pixels with the same concentration level. Therefore, it can be approximated by estimating the value at certain concentration levels using the proposed dye deconvolution neural network. λ f λ By using the polynomial f λ Modeling it as a function of c reduces the system from nonlinear

[0156] In digital pathology image analysis, only a few concentration levels are relevant for each stain, and the results of this experiment show that even for each type of stain, it is possible to λ Approximating φ to a constant floating point number (ie, a constant), hereinafter referred to as the stain spectral correction factor, can also greatly improve the deconvolution results.

[0157] like Fig.10 As shown, estimating the nonlinear color signal using a linear model results in significant errors (difference between the blue and red curves). By introducing a spectral correction factor, the optical absorbance model can be adapted to approximate the underlying nonlinear signal formation. Fig.10 The rightmost plot shows an example of how a floating point number (constant parameter) used as a spectral correction factor can change the slope of a linear model to approximate the true absorbance value for concentration levels of interest (orange circles), such as target nuclei for a stain.

[0158] Example 3: Colorant Deconvolution Example Using an Implementation of a Colorant Deconvolution Network Based on Deep Image Priors

[0159] In this example, the deconvolution results for images stained with three different dye combinations are shown using an example implementation of the algorithm proposed in Example 1. Specifically, the model was tested with breast tissue images stained with hematoxylin and eosin (H&E), bone marrow tissue images stained with hematoxylin and 3,3'-diaminobenzidine (DAB), and breast tissue images stained with SMACD34 MF3 antibody, and the results are shown in Fig.11 In the 1st, 2nd and 3rd rows.

[0160] like Fig.11 As shown, the current implementation of the proposed stain deconvolution network successfully deconvolves all three types of stain combinations into a concentration map, a color vector, and a background correction vector. All deconvolved components are combined to produce a generated image, which is compared to the target image to calculate the loss function during training. Combining all components except the background correction vector results in a background illumination corrected version of the target image. Stain quantization can also be performed by calculating the ratio of positively stained pixels to all pixels.

[0161] We also analyzed the bone marrow tissue images with and without the spectral correction factor for DAB staining (DAB is a scatterer of light). Fig.11 The performance of the Dye Deconvolution Network implementations is compared in the 4th and 5th rows of Figure 1. Dye Deconvolution with spectral correction correctly estimates the color of the DAB dye as brown (4th row), while the linear model (without spectral correction factor) yields a pink color vector that is clearly incorrect for DAB (5th row). This example shows that incorporating a spectral correction factor improves the performance of the Dye Deconvolution Network for light scatterers.

[0162] The stain deconvolution performance was also compared with two state-of-the-art conventional stain deconvolution algorithms: Macenko's algorithm and Vahadane's algorithm. Tissue samples were first stained by DAPI fluorescence imaging to obtain a gold standard (concentration map) of hematoxylin-sensitive tissue layers. The samples were then washed and restained with hematoxylin and eosin (H&E). The DAPI images were registered to the corresponding H&E images, and Otsu thresholding was performed to obtain the true values. All three methods were applied to separate the stains, and the resulting hematoxylin concentration maps were used for quantitative comparison. Fig.12 and Fig.13 The results are shown in . Two metrics were used for comparison: point biserial correlation, which measures the generalized correlation between high and low hematoxylin concentrations (between the concentration plot and the true value), and structural similarity. Fig.12 and Fig.13It can be seen that the method of the present invention is at the same level as the comparison method at least in terms of point biserial correlation, and is significantly better than the other two methods in preserving local structure.

[0163] Example 4: Colorant Normalization Example Using an Implementation of a Colorant Deconvolution Network Based on Deep Image Priors

[0164] In this section, an example of color normalization using the proposed implementation of the color deconvolution network is provided. Color normalization is performed using the same implementation as in Example 1 and on the same dataset: MIDOG dataset (https: / / imi.thi.de / midog / the-challgenge / ).

[0165] Fig.14 Example results of stain normalization using the proposed network are shown. Four image patches with different contents from four different scanners (Scanner 1: Hamamatsu XR nanozoomer 2.0; Scanner 2: Hamamatsu S360 (0.5NA); Scanner 3: Aperio ScanScope CS2; Scanner 4: LeicaGT450) were selected for stain normalization. The concentration map and stain vector for each image were calculated using the stain normalization network, and mixed and matched to generate all 16 possible styles and content combinations. For example, the first row shows the simulated appearance of content 1 scanned by 4 scanners, and the second column shows the simulated appearance of all contents scanned by scanner 2. Image patches scanned by different scanners can also be normalized with predefined 'standard' color vectors to make them look similar in style and thus improve the performance of downstream analysis.

[0166] The specific embodiments described above have been shown by way of example, and it should be understood that these embodiments may allow for various modifications and alternative forms. It should be further understood that the claims are not intended to be limited to the specific forms disclosed, but are intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the present disclosure.

Claims

1. A method for performing colorant deconvolution on a color target image, the method comprising: A colorant deconvolution network is provided, the colorant deconvolution network comprising: a plurality of convolutional autoencoder neural networks, each convolutional autoencoder neural network configured to process a respective input data set to generate a respective output data set, each input data set comprising a respective two-dimensional array having a size equal to a pixel size of the color target image; an absorbance calculation module configured to generate a color absorbance image using an absorbance model, wherein the color absorbance image is generated by processing a plurality of dye vectors and a plurality of dye concentration maps, wherein each dye vector and each dye concentration map is associated with a corresponding dye; The absorbance calculation module is operatively coupled to the plurality of convolutional autoencoder neural networks such that each stain concentration map is obtained from the output data set of a corresponding convolutional autoencoder neural network; and The dye deconvolution network is trained according to a dye deconvolution loss function, wherein the dye deconvolution loss function comprises: a first loss component configured to minimize a generation loss associated with the color target image; and a second loss component configured to facilitate separation between the dye concentration maps; Such that after the training, the dye concentration map correspondingly represents the deconvolved dye concentration map of the dye within the color target image.

2. The method according to claim 1, wherein the absorbance calculation module is further configured to generate the color absorbance image by calculating the sum of the products of the dye vector, the dye concentration map and the dye spectrum correction factor for each dye; wherein each stain has an associated stain spectral correction factor; and Each dye spectral correction factor is updated during the training according to minimization of the dye deconvolution loss function.

3. The method of claim 2, wherein the dye spectral correction factor is defined according to a dye spectral correction factor parameter of the dye deconvolution network, and wherein the dye spectral correction factor parameter is initialized prior to the training.

4. The method of claim 2, wherein the dye deconvolution network comprises a plurality of spectral correction neural networks, each spectral correction neural network being configured to determine a corresponding dye spectral correction factor, and wherein each spectral correction neural network is trained according to the dye deconvolution loss function.

5. The method of claim 4, wherein at least one spectral correction neural network of the dye deconvolution network is an encoder-decoder network.

6. The method of claim 5, wherein the dye spectral correction factor corresponding to the at least one spectral correction neural network is determined based on a latent feature of the at least one spectral correction neural network.

7. The method according to any one of claims 1 to 6, wherein the absorbance calculation module is further configured such that the calculation of the color absorbance image includes a color background vector; The color background vector is updated during the training according to minimization of the colorant deconvolution loss function.

8. The method of claim 7, wherein the color background vector is defined according to background parameters of the colorant deconvolution network, and wherein the background parameters are initialized prior to the training.

9. The method of claim 7, wherein the colorant deconvolution network comprises a background neural network configured to determine the color background vector, and wherein the background neural network is trained according to the colorant deconvolution loss function.

10. The method of claim 9, wherein the background neural network is an encoder-decoder network.

11. The method of claim 10, wherein the value of the color background vector is determined according to a latent feature of the background neural network.

12. The method according to any one of claims 1 to 11, wherein the colorant vector is updated during the training according to minimization of the colorant deconvolution loss function.

13. The method of claim 12, wherein the colorant vector is defined according to colorant vector parameters stored within the colorant deconvolution network, and wherein the colorant vector parameters are initialized prior to the training.

14. The method of claim 12, wherein the colorant deconvolution network comprises a plurality of colorant vector neural networks, each colorant vector neural network being configured to determine a corresponding colorant vector, and wherein each colorant vector neural network is trained according to the colorant deconvolution loss function.

15. The method of claim 14, wherein at least one colorant vector neural network of the colorant deconvolution network is an encoder-decoder network.

16. The method of claim 15, wherein the value of the colorant vector corresponding to at least one colorant vector neural network is determined based on a latent feature of the at least one colorant vector neural network.

17. A method according to any one of claims 14 to 16, wherein during at least an initial portion of the training, the colorant deconvolution loss function includes an additional loss term based on the difference between the colorant vector calculated by the colorant vector neural network and a predetermined initialization value of the colorant vector.

18. The method of claim 17, wherein the additional loss term is included in the colorant deconvolution loss function during the initial portion of the training and is not present in the colorant deconvolution loss function during a subsequent portion of the training.

19. The method of any one of claims 12 to 18, further comprising employing the colorant vector to perform normalization when processing different color target images.

20. A method according to any one of claims 1 to 19, further comprising using the colour absorbance image to generate an output image to provide a reproduced version of the colour target image.

21. The method of any one of claims 1 to 19, wherein the first loss component is based on a difference between the color absorbance image and a target color absorbance image generated from the color target image.

22. The method of any one of claims 1 to 19, wherein the first loss component is based on a difference between the color target image and an output image generated based on the color absorbance image.

23. The method of any one of claims 1 to 22, wherein at least one input data set is randomly generated.

24. The method according to any one of claims 1 to 23, wherein at least two of the input data sets are common input data sets.

25. A method according to any one of claims 1 to 24, wherein at least one of the convolutional autoencoder neural networks comprises a skip connection.

26. A method according to any one of claims 1 to 25, wherein during at least an initial part of the training, at least one transformation of the input data set is used to enhance the calculation of the loss function.

27. A method according to any one of claims 1 to 26, wherein the color target image is a first color block of a primary color image, and wherein the colorant vector obtained after training is a final colorant vector, the method further comprising using the final colorant vector when performing colorant deconvolution on another color block of the primary color image.

28. The method of claim 27, wherein the final colorant vector is employed to initialize a colorant vector when performing colorant deconvolution on the another color patch of the primary color image.

29. The method of any one of claims 1 to 28, further comprising performing dye quantification using the dye concentration map.

30. A system for performing colorant deconvolution on a color target image, the system comprising: Control and processing circuitry comprising at least one processor and a memory, the memory comprising instructions executable by the at least one processor to perform operations comprising: A dye deconvolution network is generated, wherein the dye deconvolution network comprises: a plurality of convolutional autoencoder neural networks, each convolutional autoencoder neural network configured to process a respective input data set to generate a respective output data set, each input data set comprising a respective two-dimensional array having a size equal to a pixel size of the color target image; an absorbance calculation module configured to generate a color absorbance image using an absorbance model, wherein the color absorbance image is generated by processing a plurality of dye vectors and a plurality of dye concentration maps, wherein each dye vector and each dye concentration map is associated with a corresponding dye; The absorbance calculation module is operatively coupled to the plurality of convolutional autoencoder neural networks such that each stain concentration map is obtained from the output data set of a corresponding convolutional autoencoder neural network; and The dye deconvolution network is trained according to a dye deconvolution loss function, wherein the dye deconvolution loss function comprises: a first loss component configured to minimize a generation loss associated with the color target image; and a second loss component configured to facilitate separation between the dye concentration maps; Such that after the training, the dye concentration map correspondingly represents the deconvolved dye concentration map of the dye within the color target image.

Images