Systems and methods for performing colorant deconvolution

Abstract

A method and system for staining deconvolution employing a neural network architecture based on deep image priors and an associated training scheme are disclosed. An example staining deconvolution network uses an autoencoder network to generate staining concentration maps of various stains associated with a colored target image without training on a large reference image dataset. The staining deconvolution network is trained using a loss function that facilitates separation between correct image generation and the staining concentration maps generated by the autoencoder network. The deep image prior-based staining deconvolution network can be configured to encode an adaptive physical model comprising a set of parameters modeling the nonlinear dependence of background illumination and absorption on concentration and wavelength. This example method of staining deconvolution, which can be performed without prior training data, can therefore generalize to accommodate a variety of stains and previously uncharacterized staining types, and can be extended to perform staining normalization.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 330,419, filed April 13, 2023, entitled “SYSTEMS AND METHODS FOR PERFORMING STAIN DECONVOLUTION”, the entire contents of which are incorporated herein by reference. Background Technology

[0003] This disclosure relates to the deconvolution of staining components from color images, such as digital pathological tissue images.

[0004] Histopathology is a diagnostic discipline based on the visual interpretation of cellular biological features captured in images using stains that specifically bind to their target antigens. The advent of digital imaging in pathology has propelled this traditional field into what is now described as digital pathology, where pathological information from stained slides is acquired, stored, and managed to create large-scale datasets for disease diagnosis, biological research, and drug development. Digital images are suitable for computational pathology, both for basic measurements and counting, and for advanced machine learning tasks. Images can now be evaluated using machine learning to obtain features beyond those assessed in traditional histopathology, such as directly linking images to clinical data (e.g., prognosis, mutations).

[0005] Therefore, performing staining deconvolution on multi-stained images is an essential step in most histological image analysis algorithms. Staining deconvolution is the process of transforming stained tissue slice images from the normal RGB color space into a series of staining channels. Each staining channel is a grayscale image that represents the intensity of a specific stain expressed across the entire original image. Staining deconvolution methods typically attempt to find an ideal staining matrix, i.e., a matrix that, when multiplied by the RGB color channels, produces the desired number of staining channels. The staining matrix consists of staining vectors (also called "color vectors" or "staining color vectors"), where each staining vector represents the model color of a specific stain in the original image and provides a correspondence between the contribution of a specific stain to each color channel and its absorbance (optical density).

[0006] A common problem in tissue sample analysis is undesirable color variations due to differences in the color response of slide scanners, raw materials and manufacturing techniques of staining suppliers, and staining protocols among different pathology laboratories. This makes it difficult for software trained for the appearance of specific stains to interpret images and further exacerbates existing inter- and intra-expert diagnostic and labeling discrepancies among pathologists. This variation can be corrected by performing staining deconvolution and then recombining the staining channels using a reference template. This is known as staining normalization. Summary of the Invention

[0007] A method and system for staining deconvolution employing a neural network architecture based on deep image priors and an associated training scheme are disclosed. An example staining deconvolution network uses an autoencoder network to generate staining concentration maps of various stains associated with a colored target image without requiring training on a large reference image dataset. A loss function is used to train the staining deconvolution network, which facilitates separation between correct image generation and the staining concentration maps generated by the autoencoder network. The deep image prior-based staining deconvolution network can be configured to encode an adaptive physical model comprising a set of parameters modeling the nonlinear dependence of background illumination and absorption on concentration and wavelength. This example method of staining deconvolution, which can be performed without prior training data, can therefore generalize to accommodate a wide variety of stains and previously uncharacterized staining types.

[0008] Therefore, in a first aspect, a method is provided for performing stainer deconvolution on a color target image, the method comprising:

[0009] Provide a stainer deconvolution network, the stainer deconvolution network comprising:

[0010] Multiple convolutional autoencoder neural networks, each configured to process a corresponding input dataset to generate a corresponding output dataset, each input dataset comprising a corresponding two-dimensional array having a size equal to the pixel size of the color target image;

[0011] An absorbance calculation module is configured to generate a color absorbance image using an absorbance model. The color absorbance image is generated by processing multiple dye vectors and multiple dye concentration maps, wherein each dye vector and each dye concentration map is associated with a corresponding dye.

[0012] The absorbance calculation module is operationally coupled to multiple convolutional autoencoder neural networks, such that each staining concentration map is obtained from the output dataset of the corresponding convolutional autoencoder neural network; and

[0013] A stainer deconvolution network is trained based on a stainer deconvolution loss function comprising: a first loss component configured to minimize a generation loss associated with a color target image; and a second loss component configured to facilitate separation between stainer concentration maps; such that, after training, the stainer concentration maps correspondingly represent deconvolutioned stainer concentration maps of stainers within the color target image.

[0014] In some implementations of the method, the absorbance calculation module is further configured to generate a color absorbance image by calculating the sum of the products of a staining vector, a staining concentration map, and a staining spectral correction factor for each staining agent; wherein each staining agent has an associated staining spectral correction factor; and wherein each staining spectral correction factor is updated during training based on minimizing the staining deconvolution loss function.

[0015] The staining spectral correction factor can be defined based on the staining spectral correction factor parameters of the staining deconvolution network, wherein the staining spectral correction factor parameters are initialized before training.

[0016] The staining deconvolution network may include multiple spectral correction neural networks, each configured to determine a corresponding staining spectral correction factor, and wherein each spectral correction neural network is trained according to a staining deconvolution loss function. At least one spectral correction neural network of the staining deconvolution network may be an encoder-decoder network. The staining spectral correction factor corresponding to the at least one spectral correction neural network may be determined based on latent features of the at least one spectral correction neural network.

[0017] In some implementations of the method, the absorbance calculation module is further configured such that the calculation of the color absorbance image includes a color background vector; wherein the color background vector is updated during training based on minimizing the dye deconvolution loss function. The color background vector may be defined based on background parameters of the dye deconvolution network, and wherein the background parameters are initialized prior to training. The dye deconvolution network may include a background neural network configured to determine the color background vector, and wherein the background neural network is trained based on the dye deconvolution loss function. The background neural network may be an encoder-decoder network. The value of the color background vector may be determined based on latent features of the background neural network.

[0018] In some implementations of the method, the stainer vector is updated during training by minimizing the stainer deconvolution loss function. The stainer vector may be defined based on stainer vector parameters stored within the stainer deconvolution network, wherein the stainer vector parameters are initialized prior to training. The stainer deconvolution network may include multiple stainer vector neural networks, each configured to determine a corresponding stainer vector, wherein each stainer vector neural network is trained according to the stainer deconvolution loss function. At least one stainer vector neural network of the stainer deconvolution network may be an encoder-decoder network. The value of the stainer vector corresponding to the at least one stainer vector neural network may be determined based on latent features of the at least one stainer vector neural network.

[0019] In some implementations of the method, during at least the initial portion of training, the stainer deconvolution loss function includes an additional loss term based on the difference between the stainer vector calculated by the stainer vector neural network and a predetermined initial value of the stainer vector. This additional loss term may be included in the stainer deconvolution loss function during the initial portion of training but not present in the stainer deconvolution loss function during subsequent portions of training.

[0020] In some implementations, the method further includes using a dye vector to perform normalization when processing different color target images.

[0021] In some implementations, the method further includes using a color absorbance image to generate an output image, thereby providing a regenerated version of the color target image.

[0022] In some implementations of the method, the first loss component is based on the difference between the color absorbance image and the target color absorbance image generated from the color target image.

[0023] In some implementations of the method, the first loss component is the difference between the color target image and the output image generated based on the color absorbance image.

[0024] In some implementations of the method, at least one input dataset is randomly generated.

[0025] In some implementations of the method, at least two of the input datasets are common input datasets.

[0026] In some implementations of the method, at least one of the convolutional autoencoder neural networks includes skip connections.

[0027] In some implementations of the method, at least one transformation of the input dataset is used to enhance the computation of the loss function during at least the initial portion of training.

[0028] In some implementations of the method, the color target image is a first color patch of the main color image, and the stainer vector obtained after training is the final stainer vector. The method further includes using the final stainer vector when performing stainer deconvolution on another color patch of the main color image. The stainer vector can be initialized using the final stainer vector when performing stainer deconvolution on another color patch of the main color image.

[0029] In some implementations, the method further includes using a staining concentration map to perform staining dosing.

[0030] In another aspect, a system is provided for performing stainer deconvolution on a color target image, the system comprising:

[0031] A control and processing circuit, comprising at least one processor and a memory, the memory including instructions executable by the at least one processor to perform operations including:

[0032] Generate a stainer deconvolution network, the stainer deconvolution network comprising:

[0033] Multiple convolutional autoencoder neural networks, each configured to process a corresponding input dataset to generate a corresponding output dataset, each input dataset comprising a corresponding two-dimensional array having a size equal to the pixel size of the color target image;

[0034] An absorbance calculation module is configured to generate a color absorbance image using an absorbance model. The color absorbance image is generated by processing multiple dye vectors and multiple dye concentration maps, wherein each dye vector and each dye concentration map is associated with a corresponding dye.

[0035] The absorbance calculation module is operationally coupled to multiple convolutional autoencoder neural networks, such that each staining concentration map is obtained from the output dataset of the corresponding convolutional autoencoder neural network; and

[0036] The stainer deconvolution network is trained based on a stainer deconvolution loss function, which includes:

[0037] A first loss component, configured to minimize the generation loss associated with the color target image; and

[0038] A second loss component, configured to promote separation between staining concentration maps;

[0039] This results in a staining concentration map that, after training, represents the deconvolutioned staining concentration map of the staining within the colored target image.

[0040] A further understanding of the functionality and advantages of this disclosure can be obtained by referring to the following detailed description and accompanying drawings. Attached Figure Description

[0041] Embodiments of the invention will now be described by way of example and with reference to the accompanying drawings, in which:

[0042] Figure 1 The diagram illustrates a conventional matrix factorization-based method for deconvolution of dye colors.

[0043] Figure 2 An example of a conventional deep learning method for stainer normalization is illustrated schematically.

[0044] Figure 3A An example implementation of a staining deconvolutional network based on multiple depth image priors for determining staining concentration maps is shown.

[0045] Figure 3B It shows the relationship with Figure 3A The example of a suitable autoencoder network used with example colorants in deconvolutional networks is an example implementation of such a network.

[0046] Figure 4 An example of a stainer deconvolution network based on a depth image prior is shown, which refines the stainer vector during the training of an autoencoder neural network.

[0047] Figure 5 Different example methods for refining the staining vector during the training of a staining deconvolution network based on depth image priors are illustrated.

[0048] Figure 6 An example implementation of a staining deconvolution network including an autoencoder neural network is shown, which is used to generate staining concentration maps associated with two staining agents.

[0049] Figure 7 An example method is shown where a stainer deconvolution network based on depth image priors is used to perform stainer normalization.

[0050] Figure 8 An example system for performing stainer deconvolution is shown.

[0051] Figure 9 The results show unsupervised clustering of physical parameters estimated by a stainer deconvolutional network to match scanner types in the example dataset.

[0052] Figure 10 This demonstrates how the dye spectral correction factor can reduce systematic errors in color deconvolution compared to conventional methods.

[0053] Figure 11 Examples of staining deconvolution results in three types of digital pathology images and how the spectral correction factor improves staining deconvolution performance are shown.

[0054] Figure 12 The proposed method is shown in comparison with state-of-the-art staining deconvolution methods for color deconvolution of H&E images, where the proposed method achieves the highest point bicolumn correlation.

[0055] Figure 13 The proposed method is shown in comparison with state-of-the-art staining deconvolution methods for color deconvolution of H&E images, where the proposed method achieves the best structural similarity index measure.

[0056] Figure 14 An example is shown of using a staining deconvolution network to perform staining transformation and normalization on a dataset of images captured by four different types of scanners. Detailed Implementation

[0057] Various embodiments and aspects of this disclosure will be described with reference to the details of the following discussion. The following description and figures illustrate this disclosure and should not be construed as limiting it. Numerous specific details are described to provide a thorough understanding of the various embodiments of this disclosure. However, in some cases, well-known or conventional details have not been described in order to provide a brief discussion of embodiments of this disclosure.

[0058] As used herein, the terms “comprises” and “comprising” should be interpreted as inclusive and open-ended, not exclusive. Specifically, when used in the specification and claims, the terms “comprises” and “comprising”, and variations thereof, mean to include the specified features, steps, or components. These terms should not be construed as excluding the presence of other features, steps, or components.

[0059] As used herein, the term “exemplary” means “used as an example, instance or illustration” and should not be construed as being preferred or advantageous over other configurations disclosed herein.

[0060] As used herein, the terms “about” and “approximately” are intended to cover variations that may exist within the upper and lower limits of the range of values, such as variations in properties, parameters, and dimensions. Unless otherwise specified, the terms “about” and “approximately” mean ±25% or less.

[0061] It should be understood that, unless otherwise specified, any designated scope or group is used as an abbreviation to refer individually to each member of the scope or group, and to each possible subscope or subgroup covered therein, and is equally applicable with respect to any subscope or subgroup therein. Unless otherwise specified, this disclosure relates to and explicitly combines each specific member and combination of subscopes or subgroups.

[0062] As used herein, the term “approximate” when used in conjunction with a quantity or parameter refers to a range spanning approximately one-tenth to ten times the quantity or parameter.

[0063] Despite the great interest in dye deconvolution methods, their applicability to specific image analysis problems is often limited. Traditional dye deconvolution methods rely on matrix factorization algorithms based on the Beer-Lambert Law, which correlates the attenuation of nonscattered light with the properties of the dye used for staining.

[0064] Figure 1 The image illustrates a conventional method for dye deconvolution. This conventional method transforms the color deconvolution problem into a matrix factorization problem. After preprocessing steps such as resizing and background illumination correction, a Beer-Lambert transform is applied to convert the observed image intensity values ​​in each RGB channel into optical density. Matrix factorization is then used to recover the dye concentration map and dye color vector for each dye. To achieve better performance than baseline nonnegative matrix factorization (NMF) or singular value decomposition (SVD), traditional physics-based methods rely on manually created priors (e.g., sparsity, structural shape, histogram / spectral matching criteria), but these priors are laborious, heuristic, and dye-specific. The color absorption properties of the dye (dye vector) are empirically observed and predefined, or estimated using template matching based on a manually selected region of interest (ROI).

[0065] While most staining deconvolution methods remain relatively robust for images stained with hematoxylin and eosin (H&E), their application to specific stains is severely hampered. In fact, staining deconvolution for other stains relies on a problem-specific design, requiring significant domain expertise and experience for a given set of stains, and rarely translates to other sets.

[0066] Specifically, the assumptions of the Beer-Lambert law often fail in the practice of staining deconvolution based on staining agents other than H&E, which often leads to severe deconvolution errors. Furthermore, with the rapid development of digital pathology, specialized staining agents are becoming increasingly popular due to their ability to reveal more detailed information and structures than traditional H&E staining agents can provide, thus improving disease diagnosis and treatment.

[0067] The recent success of deep learning has shifted the focus of stainer deconvolution methods to neural network-based approaches and has provided inspiration for stainer deconvolution with specific stainers. Figure 2 An example of a conventional deep learning method for staining normalization is shown. Deep learning methods typically do not explicitly compute concentration maps and staining vectors, but instead directly perform image-to-image transformations (converting an image of one staining type / source to another). To achieve this, large image datasets from both sources are required. Similar to conventional physics-based methods, the images are first preprocessed and then fed into two neural networks, leveraging the powerful learning capabilities of neural networks to simultaneously learn color transformations and imaging structures. In addition to image generation branches (GANs, VAEs, etc.) used for image reconstruction or image-to-image transformation, deep learning-based methods often incorporate auxiliary branches to gain performance gains from solving another related task (e.g., tissue type clustering). Deep learning-based methods generally exhibit better performance compared to conventional methods because their color separation / style transfer does not rely on manually created priors and manually selected ROIs.

[0068] However, deep learning-based techniques suffer from poor interpretability, and the normalization process requires template images from the target domain. Although existing deep learning-based methods claim to be unsupervised because they do not require separate staining layer annotations, they rely on large, managed image datasets from different domains and are therefore semi-supervised / weakly supervised.

[0069] Furthermore, while the power of deep learning allows staining deconvolution methods to circumvent the Beer-Lambert law, their accessibility and universality are often problematic. Although existing deep learning-based methods are termed "unsupervised" because they do not require ground truth values ​​for individual staining layers for training, they still require manually selected large datasets, data management (curation), and high levels of expertise in network design and training. The data-driven nature of existing deep learning-based staining deconvolution methods makes them very powerful on the data they are trained on, but this also reduces their performance on unseen data. The lack of universally applicable staining deconvolution algorithms is an obstacle to the development of digital pathology analysis.

[0070] The inventors realized that by employing a deep learning framework that does not require pre-training on large image datasets, the limitations of conventional deep learning methods in stainer deconvolution can be avoided. In fact, by using a deep learning framework based on deep image priors, accurate stainer deconvolution can be performed using only the image data itself to train the network, rather than relying on a large training dataset.

[0071] Deep Image Prior (DIP) neural networks were initially proposed to address inverse image problems such as denoising and inpainting. They are based on the concept that the structure of a neural network (especially convolutional filters) can capture imaging statistics of natural images without being trained for a specific task. DIP neural networks are trained with randomly initialized network parameters, using random noise as input, and generating an output image based on a loss function involving the target image. During training, the network's convolutional filters capture internal patch recurrences (also known as image priors) and use them as basic building blocks for reconstructing the target image, allowing the removal of external image information (such as noise or artifacts), thus contributing to the recovery of a denoised or corrected version of the target image.

[0072] A deep image prior-based neural network (e.g., the so-called "dual DIP" network) has also been implemented, which is based on the observation that when multiple deep image prior neural networks are combined to reconstruct an image, each deep image prior neural network tends to divide the image into important decomposition parts, such as background and foreground, foggy and defogging images, or overlapping images with different degrees of transparency.

[0073] The inventors realized that multi-depth image prior neural networks can be used to perform staining deconvolution, wherein a set of autoencoder networks is used to generate staining concentration maps of multiple stains associated with a color target image, without training on a large reference image dataset. Instead, the inventors inferred that a loss function can be used to train the multi-depth image prior network, the loss function employing: (i) a loss component that promotes correct image generation (based on the color target image), and (ii) a loss component that promotes separation between staining concentration maps generated by the autoencoder network.

[0074] As will be described in detail below, unlike previous dye deconvolution methods, the dye deconvolution method of this invention, employing a deep image prior-based neural network structure and an associated training scheme, can be used to perform dye deconvolution on images with any number and type of dyes. The dye deconvolution network of this invention, based on deep image priors, can also be configured to encode an adaptive physical model, thereby providing enhanced interpretability. This physical model includes a set of parameters describing the properties of the background illumination, multicolor dyes, and sensors. Since test-time dye deconvolution can be performed without prior training data, the method of this invention can generalize to adapt to new and previously uncharacterized dye types without the risk of overfitting or the need for ground truth labels. Furthermore, since the deep image prior-based method of this example does not require training on an image dataset but only on the colored target image itself, the method can be applied to entry-level graphics cards.

[0075] Figure 3A An example implementation of a staining deconvolution network based on multiple depth image priors is shown. The example staining deconvolution network does not require pre-training and is therefore independent of staining type. A neural network portion 310 containing multiple autoencoder neural networks is employed to generate staining concentration maps for at least two staining agents, wherein the maps show staining concentration maps 321 and 322 corresponding to a first and a second staining agent, and one or more optional additional staining concentration maps 323 corresponding to one or more optional additional staining agents.

[0076] As shown in the figure, the dye concentration maps 321-323, together with the corresponding dye vector 330, are processed in the absorbance calculation module 340 to generate a color absorbance image and / or color intensity image as shown at 350. Although the RGB color space is frequently used in the examples provided in this disclosure, it should be understood that other color spaces, such as, but not limited to, HSV and LAB, may also be used in alternatives.

[0077] Deconvolution is performed on a color target image, which is a two-dimensional image with a size of LxW and three color channels, and thus can be represented by a matrix of size 3xLxW. Therefore, each staining concentration map has a size of LxW, and each staining vector is a three-dimensional vector, with each color channel being a single dimension.

[0078] In this example method, the absorbance module employs the Beer-Lambert law to generate a color absorbance image based on the calculated staining concentrations (Figures 321-323). For example, in the case of a color target image corresponding to a sample with n mixed staining agents, the absorbance (optical density, OD) is as follows based on the staining vector C. k And staining agent concentration diagram Sk Generate, where k = 1…n represents the amount of dye:

[0079]

[0080] Where OD is a 3xLxW matrix, representing the color optical density of the color target image (color and pixel indices are not shown in the equation), and each dye concentration map S... k It is a matrix of size L x W, which represents the pixel-wise concentration of a given dye (pixel indices are not shown in the equation), and where each dye vector C k It is a 3-vector that correlates the spectral density of a given dye with the concentration of the dye (where each dimension of the dye vector corresponds to a different color channel).

[0081] As described in more detail below, this equation is merely an example of absorbance calculation, and other embodiments described below disclose alternative absorbance calculation methods that include additional coefficients and / or terms.

[0082] Absorbance (OD) is correlated with the intensity image by the following equation:

[0083]

[0084] Where I is the transmitted intensity measured on the sample, and I0 is the incident intensity on the sample. This equation can be inverted to generate a final color image from the color absorbance image.

[0085] In a typical deep learning-based workflow, the neural network is trained on a dataset of training images to fix the activations and weights of the network. The trained network then processes the target image. However, in stark contrast, Figure 3A The staining deconvolution network shown does not require training on a training dataset, and it can determine the staining concentration of the target color image (Figures 321-323) without providing the target color image as input to the network. Instead, as described below, the staining concentration (Figures 321-323) can be determined using the concept of depth image priors based on a suitable loss function involving the color target image.

[0086] like Figure 3A As shown, the neural network portion 310 includes corresponding autoencoder networks 311-313 for each staining agent, which are used to generate corresponding staining agent concentration maps 321-323. The autoencoder networks 311-313 do not require training before performing staining agent deconvolution on the color target image.

[0087] Figure 3BA non-limiting example of a suitable autoencoder network 311 is shown. In this example implementation, skip connections are included between the first layer of the encoder and the fifth layer of the decoder, and between the second layer of the encoder and the fourth layer of the decoder. It should be understood that the use and number of skip connections are optional and can vary between implementations of the embodiments of the invention. Furthermore, it should be understood that other hyperparameters may also vary depending on the different implementations, including, for example, the number of channels, the number of layers, and the activation type.

[0088] Refer again Figure 3A Each autoencoder network can be fed a random noise dataset 360 (e.g., random uniform noise in the range [-0.5, 0.5]) or any other dataset, including the colored target image itself, as input. Although this diagram illustrates feeding the same input dataset to each autoencoder network, this is merely one implementation, and in other example implementations, different input datasets can be fed to the autoencoder networks.

[0089] While it may seem counterintuitive that a randomly initialized autoencoder neural network can generate accurate staining density maps based on random input data unrelated to a color target image, the example staining deconvolution network autoencoder neural networks 311-313 can achieve this functionality using a loss function based on the color target image and including a term that forces separation between staining density maps.

[0090] The loss function used to train the neural network portion 310 may include at least two loss components. The first loss component is a generation loss component used to minimize the generation error. This generation loss component may, for example, be configured to minimize the difference between a color absorbance image generated based on a dye concentration map according to the Beer-Lambert law (hereinafter referred to as the "generated color absorbance image") and an absorbance image calculated from a color target image, or, for example, to minimize the difference between a final image generated based on a dye concentration map (e.g., using the equation shown above) and the color target image. The generation loss may be an L1-type loss. For example, in some example implementations, the generation loss may be calculated as an L1 loss for the absorbance of each color channel, and optionally include a loss term based on the sum of the absorbances of the three color channels.

[0091] The second loss term provides a repulsion loss between staining concentration maps to facilitate the differentiation of different staining layers, since different stainings should highlight different regions of interest.

[0092] For example, in the case of a two-color staining agent separation problem (i.e., a color target image includes contributions from two staining agents), the net loss function may include four generation losses (three per-color-channel absorption generation loss terms, one absorption and (for all color channels) generation loss term), and one repulsion loss term that promotes separation between the two staining agent concentration maps. By combining the aforementioned generation loss term and the N(N-1) / 2 repulsion loss term, this example loss function can be generalized to N>3 colors.

[0093] In some example implementations, the dye vector 330 is refined during the training of the autoencoder neural network. For example, Figure 4 An example implementation is shown where staining vectors 330 are shown residing in neural network portion 310, demonstrating that their values ​​are updated during training epochs of the autoencoder neural network. The staining vectors can be initialized, for example, based on randomly initialized values, based on user-inputted staining vectors, or based on staining vectors that can be automatically calculated based on user-selected regions of interest in the input image (as prior knowledge). In the latter case, the network automatically adjusts the user-inputted staining vectors to achieve better separation.

[0094] In one example implementation, the staining vector 330 can be incorporated into the staining deconvolution network as a parameter updated during training. For example, as... Figure 4 As shown, the staining vector 330 can be provided as a set of parameters (e.g., six parameters in the example case of two staining agents), which are stored and refined during training without a neural network structure. The values ​​of the staining vector parameters are initialized with initial values ​​(e.g., estimates or random values), as shown at 370. Figure 5 Case 1 also schematically illustrates this configuration.

[0095] However, in other example implementations, each staining vector may be generated by a corresponding staining vector neural network structure residing within neural network portion 310 and trained together with an autoencoder network that computes staining concentration maps. Figure 5 Cases 2 and 3 illustrate two example implementations of this scheme. Case 2 shows an example of calculating a given staining vector based on a feedforward neural network 332, while Case 3 shows an example of calculating a given staining vector based on an encoder-decoder network 334. When the encoder-decoder network 334 is used to generate the staining vector, the value of the given staining vector can be generated, for example, by a subset of pixels of the decoded image (including, for example, the center pixel of the decoded image), or, for example, based on latent features generated by the network.

[0096] In an example implementation where the staining vector 330 is refined during the training of the autoencoder neural network, the loss function can be adjusted to include a loss component related to the staining vector. For example, a color-fixed loss term can be included in the loss function to adjust the staining deconvolution network during at least a portion of the training.

[0097] In one example implementation, an L1 color fixation loss is computed between the estimated stainer color vector and an initial value for the stainer vector to constrain the stainer vector during the initial portion of training (e.g., during the initial number of epochs, where the initial number of epochs is less than 2000), and the color fixation loss is removed during the remainder of training. This initial color fixation loss helps make the network training more robust during the initial training phase by focusing on learning the internal structural patterns before learning the stainer vector. In the example case involving a colored target image with two stainers, two color fixation loss terms can be included, one for each stainer, which can be generalized to one color fixation loss term per stainer.

[0098] In one example implementation, the color-fixed loss is used to adjust the network in the first 25% of rounds and removed in the last 75% of rounds to allow the network to learn the color vector of the colorant.

[0099] In some example implementations, augmentations can be provided during the training of the neural network portion. Non-limiting examples of augmentations include performing 90-degree rotations and mirroring operations on the input image data (e.g., random noise) to help the network learn pose-invariant depth image priors. Augments can be included during the initial portion of training (e.g., during the first 75% of training epochs) and disabled during the later portion of training (e.g., during the last 25% of epochs) to assist the network in focusing on deconvolution with staining agents.

[0100] The inventors also realized that, Figure 3A and Figure 4 The example stainer deconvolution network shown can be further tuned to overcome the limitations of conventional stainer deconvolution methods.

[0101] In fact, as detailed in the following examples, despite its widespread use in almost all matrix factorization-based methods, the assumptions of the Beer-Lambert law are consistently violated in practice. The Beer-Lambert law assumes monochromatic light (with a single wavelength), but background illumination, dye absorption spectra, image sensor responses, and observer functions for color matching are all wavelength-dependent. Furthermore, the Beer-Lambert law describes color absorbance, while some dyes are light scatterers. Additionally, existing dye deconvolution methods are flawed because the perceived color depends on spectral dependence arising from lighting conditions, room temperature, dyeing time, batch effects, and other factors that vary between images taken by the same scanner.

[0102] Due to the complex spectral responses of imaging cameras and light sources, the assumption that the dye vector is independent of dye concentration is generally not valid. Therefore, dye deconvolution methods based on the dye vector without considering its dependence on dye concentration produce less accurate concentration results.

[0103] The inventors realized that multi-depth image prior-based staining deconvolution networks can be tuned to employ a modified form of the Beer-Lambert law to address systematic errors caused by differences in the nonlinear concentration dependence of staining vectors. Indeed, it has been found that the problem associated with this nonlinearity can be addressed by introducing a staining spectral correction factor (an integral form of the Beer-Lambert law) approximating the Beer-Lambert law at the single image level (in contrast to the dataset level used in all existing methods).

[0104] In some example implementations, the Beer-Lambert law is modified to include a dye spectral correction factor that helps accommodate the aforementioned concentration dependence:

[0105]

[0106] Where M k It is a scalar value for each dye.

[0107] The dye spectral correction factor roughly compensates for nonlinearities caused by effects such as (but not limited to) light scattering, non-monochromatic light composition, and the camera's non-monochromatic light reception. While calculating the full nonlinear relationship for each dye without a large dataset would be extremely challenging, this example method approximates the nonlinearity based on a dye-per-dye scalar parameter assumed to be consistent across each image patch. Therefore, the dye spectral correction factor is assumed to remain constant for an image patch, but it can vary even between images from the same whole-slide image, and thus cannot be calculated using any existing method.

[0108] However, the method of the present invention, employing a network based on deep image priors, can be readily adapted to determine the staining spectral correction factor during training. For example, the staining spectral correction factor M... k This can be stored as parameters in the network, which are refined during the training of the autoencoder neural network that generates the staining concentration map. Alternatively, a feedforward neural network or encoder-decoder network trained according to a loss function can be used to generate the staining spectral correction factor (similar to different implementations for staining vector representation, in...). Figure 5 (as shown in the image).

[0109] Although this example implementation employs a single-parameter stainer spectral correction factor for each stainer, it should be understood that other example implementations may employ multiple stainer spectral correction parameters for each stainer, including coefficients of a nonlinear function defining the concentration. For example, such parameters can be determined by modeling the dependence of absorption on concentration as a function using a polynomial function form.

[0110] Background illumination can affect the perceived color of images captured by a scanner and can vary significantly between different scanners. Existing methods rely on additional preprocessing steps based on Gaussian denoising filters or sampling from background pixels to recover the original image. To achieve optimal performance with such existing methods, preprocessing steps, such as background illumination correction and denoising, should be performed before staining deconvolution. These procedures require histogram matching between manually selected background patches and manually created denoising filters, which needs to be designed for each domain (dataset / scanner) and can introduce bias.

[0111] In stark contrast, the aforementioned example embodiments of this disclosure are adapted to allow for the automatic estimation and correction of background illumination and noise without human intervention. The presence and effect of background illumination can be addressed by including a background correction vector (a 3-element vector, one element per color channel) that takes into account variations in optical density values ​​in each channel.

[0112]

[0113] The method of the present invention, which employs a network based on depth image priors, can be easily adapted to determine the background vector during training.

[0114] The background correction vector can be stored as parameters in the network, which are refined during training of the autoencoder neural network that generates the staining concentration map. Alternatively, a feedforward neural network or encoder-decoder network trained according to a loss function can be used to generate the background correction vector (again, similar to the different implementations shown for the staining vector, in...). Figure 5 (as shown in the image).

[0115] Figure 6 An example implementation of the staining deconvolution network is shown, comprising autoencoder neural networks 311 and 312 trained based on color target image data and a suitable loss function to generate staining concentration maps 321 and 322 associated with two staining agents. As shown, neural network portion 310 also includes autoencoder neural networks 313 and 314 trained to determine a staining vector 330. Network portion 310 also includes two optional additional autoencoders 315 and 316, either of which can be optionally included and trained according to the loss function to generate a staining spectral correction factor 380 and / or a background correction vector 390. While this figure shows a single autoencoder neural network 315 for generating the staining spectral correction factor 380, separate autoencoder neural networks can be provided to compute each staining spectral correction factor. The network can be extended by adding the following additional modules for each additional staining agent: an additional concentration map autoencoder, an additional staining vector autoencoder (or parameter set or feedforward network), and an additional staining spectral correction factor autoencoder (or parameter set or feedforward network). Furthermore, the dye deconvolution network can be extended, for example, by including additional modules that capture additional components, including but not limited to ink stains, color temperature, and white balance.

[0116] In some example implementations, stainer deconvolution networks based on depth image priors can be used to perform stainer normalization. Stainer normalization refers to combining a set of standard (or otherwise predetermined) stainer vectors with a given set of stainer concentration maps to obtain an image, thereby reducing the differences between images obtained using different scanners and simplifying downstream analysis.

[0117] Figure 7 A method for generating a normalized reconstructed source image 430 from a source image 405 using a target staining vector 420 via a Beer-Lambert transform 435 (or a modified form thereof) is schematically illustrated. The target staining vector is obtained by processing a color target image 400 with a staining deconvolution network 410 based on depth image priors. As shown in the lower portion of the figure, the staining deconvolution network 410 is trained based on the target color image 400 according to this example embodiment involving the use of multiple depth image prior modules (autoencoders). The training produces the calculation of the target staining vector 420 and a target staining concentration map 440, and the staining deconvolution network can also be configured to determine additional target color parameters 450 associated with a physical model, such as a target staining spectral correction factor and / or a target background vector.

[0118] like Figure 7As shown in the upper part, according to this example implementation involving the use of multiple depth image prior modules (autoencoders), a staining deconvolution network 410 is also trained based on a source color image 405. The training generates the computation of a source staining vector 425 and a target staining concentration map 445, and, if the staining deconvolution network is configured as such, also generates additional source color parameters 455, such as a source staining spectral correction factor and / or a source background vector. To generate a normalized reconstructed source image 430, a physical model (e.g., a suitable form of the Beer-Lambert law) is employed to process the source staining concentration map 445 and optional additional source parameters 455 using the target staining vector 420 instead of the source staining vector 425.

[0119] Figure 7 The diagram also illustrates an instance where the target staining vector 420 has been determined such that training the staining deconvolution network in the lower part of the figure is unnecessary, with these unnecessary operations shown by dashed lines. In this case, the target staining vector 420 can be used instead of the source staining vector 425 when generating the normalized reconstructed source image 430.

[0120] In some example implementations, the aforementioned normalization method or its variations can be used to normalize the staining in whole-slice images. Images often look very different when scanned and / or processed in different laboratories. Pathologists are accustomed to a particular appearance and prefer images displayed with similar color and intensity ranges. Furthermore, many image analysis pipelines and AI models rely on consistent color to function. In some example implementations, the staining vector used when performing staining deconvolution on another patch of the image can be initialized based on the staining vector obtained after training on one image patch.

[0121] In some example implementations, staining dosing can be performed on images generated according to the staining deconvolution method of this example. For example, the relative fraction of a given cell type can be quantified using a set of staining concentration maps determined by a staining deconvolution network generated according to (or based on) the embodiments disclosed herein. This can be performed, for example, by identifying and quantifying cells within each staining concentration map, and, for example, by generating a ratiometric measure based on cell counts. Alternatively, quantization can be performed on a pixel-based basis to identify stained versus unstained tissue, which may be applicable to nuclear, cytoplasmic, or membrane staining agents.

[0122] In one non-limiting example, this example method can be used to quantify the abundance of p53 protein in cancerous tissue. The tissue can be stained with a primary anti-p53 antibody, and a secondary antibody with peroxidase is used to visualize the DAB chromogen, while counterstaining is performed with hematoxylin. The secondary antibody is linked to the peroxidase, which causes the DAB chromogen to produce a visible brown precipitate at the antigen (i.e., p53) site. A staining deconvolution network is applied according to this example method to provide staining concentration maps of DAB (p53) and hematoxylin (cell nuclei). In one example quantification method, the presence of p53 can be quantified in the region of interest by dividing the number of p53-positive cells stained with DAB by the total number of cells stained with hematoxylin and / or DAB. In another example implementation, quantification can be performed based on pixels within the staining concentration map (rather than identified cells).

[0123] It should be understood that during staining dosing, one or more concentration maps may be thresholded to remove background pixels (e.g., low-concentration pixels) to remove pixels that are not expected to belong to cells. Pathologists / researchers can determine an appropriate threshold to define positive pixels, for example, through visual inspection.

[0124] It should be understood that many antigens exist for different applications, and P53 is just one example of an antigen. Other examples of antigens include, for instance, antigens that bind to the ER and PR receptors in breast cancer, or antigens that highlight different types of inflammatory cells. The antigen determines the distribution of the staining agent, and two common distribution types are produced by nuclear and membrane staining agents. P53, ER, and PR are all examples of nuclear staining agents. On the other hand, chromatin determines the color. The choice of chromatin varies depending on the application and laboratory, but DAB staining / hematoxylin counterstaining is a common staining combination. As mentioned above, the deconvolution implementation of the staining agents of this invention, which does not require the existence of a training dataset, is suitable for handling both known and novel color combinations.

[0125] As described in the following embodiments section, the inventors have discovered that the depth image prior-based method of the present invention outperforms existing deep learning-based methods for stainer deconvolution of various types of stainers. Furthermore, the inventors have found that the depth image prior-based methods of the present invention are unique because they can perform deconvolution on any type of stainer without requiring task-specific empirical optimization.

[0126] Not intending to be limited by theory, we believe the powerful universality of the deep image prior-based method of this invention stems from the encoding of the physical model by the deep neural network based on the image prior, making them complementary and mutually reinforcing. Specifically, the deep neural network utilizes the adjustment of the physical model to the point that optimization can be achieved based solely on the target image itself, which is unprecedented in deep learning-based staining deconvolution. In particular, the example implementation that calculates the staining spectral correction factor via the deep learning neural network throughout the network training broadens the application of fundamental physical theory and significantly reduces deconvolution errors.

[0127] The robust performance demonstrated by this example implementation proves that a capable machine learning model can be trained on the target image itself at test time, avoiding not only labor-intensive and time-consuming data collection and management processes but also preventing overfitting. The synergy between the physical model and the deep neural network based on deep image priors also addresses another problem with neural networks—interpretability. The outputs of some example dye deconvolution networks of this disclosure include a set of physical parameters that explicitly describe the conditions and properties involved in the entire light emission process from the dye to the scanner camera. The results shown in the following examples demonstrate the effectiveness of the physical model modification by employing a modified Beer-Lambert physical model with associated physical parameters, enabling the dye deconvolution network to perform well on less commonly used dyes.

[0128] It should also be noted that, in addition to staining deconvolution, the method of this invention based on depth image priors can also automate and achieve image denoising. Digital pathology images contain a large amount of noise and artifacts. Although most noise may seem insignificant to human perception, it is quite noticeable to the "eyes" of a computational classifier. The implementation of the network structure and training scheme based on depth image priors facilitates this denoising capability. The autoencoder used and trained to generate staining concentration maps utilizes a limited number of convolutional filters to capture the internal patch reproduction of the color target image. This network structure and training scheme enables the learning of regular structural building blocks, rather than learning irregular noise and small artifacts. Furthermore, as described above, in the implementation that addresses and models the nonlinear dependence of staining vectors on concentration, and in the implementation that models the properties of background light, these effects can be directly incorporated into the end-to-end training of color deconvolution, thus allowing for a more direct and convenient solution to the essential preprocessing steps.

[0129] In fact, staining deconvolution methods implemented using a depth image prior framework rely entirely on the information provided by the test image. It should be noted that performance can degrade when the test image quality is low, especially when the general assumptions (that individual stains are uniform in the target image and that the properties of the stains differ) are violated. For example, when there are large, high-concentration ink blots in the image, some implementations may tend to compute biased stain vectors and concentration maps as a trade-off to accommodate both the staining structure and the ink blots. Similarly, when two stains are too similar in color, some implementations may not achieve optimal performance in assigning the structure to the correct stain. However, it should be noted that these aspects are not a result of the network architecture design itself, but rather a general challenge for all staining deconvolution methods. In fact, the aforementioned example implementations can be adapted to address these challenges by stacking additional depth image prior modules (autoencoders) and updating the physical model (e.g., adding terms and / or parameters to the Beer-Lambert law) to account for large artifacts, and estimating all intermediate physical parameters to achieve better quality control.

[0130] Therefore, it is evident that the depth image prior method of the present invention can facilitate the provision of a new generation of staining deconvolution methods that can generalize to a wide variety of staining agents, without requiring expert knowledge or prior data collection.

[0131] This invention's staining deconvolution method based on depth image priors can be applied to a wide variety of environments, including but not limited to clinical pathology, pathological research, and drug research. In fact, drug research laboratories widely use immunohistochemistry to explore different biomarkers and develop companion diagnostic tests. Such methods involve staining slides with several different dyes labeled with different antigens or markers. A very typical example is the use of hematoxylin for nuclear staining and DAB staining for binding to the antigen of interest. Methods for separating these stains are needed to provide more accurate quantitative results. In some research applications, there may be multiple different stain colors used to identify different structures in tissue. Although some stain separation methods exist, they often need to be adapted for each new assay. This invention's staining deconvolution method based on depth image priors may be advantageous for such applications because staining deconvolution can be performed automatically for multiple stains, potentially more than two, without training on prior images and without requiring ground truth labels, thus eliminating the need for skilled operators to spend significant time tuning image processing algorithms.

[0132] Now for reference Figure 8An example system for performing staining agent deconvolution is shown. Control and processing hardware 500 is employed to process images received from a scanning device such as a slide scanner 600 (e.g., a light source, slide holder, and imaging camera) to perform staining agent deconvolution. Figure 8 As shown, in one embodiment, the control and processing hardware 500 may include a processor 510, a memory 520, a system bus 505, one or more input / output devices 530, and a number of optional additional devices, such as a communication interface 560, a display 540, an external storage unit 550, and a data acquisition interface 570.

[0133] This example method, which performs staining convolution based on depth image priors, can be implemented via processor 510 and / or memory 520. For example... Figure 8 As shown, the process of training a stainer deconvolution network based on a color target image and a suitable loss function can be implemented by the control and processing hardware 500 via executable instructions represented as a stainer deconvolution module 580. Similarly, the stainer normalization method can be implemented by the control and processing hardware 500 via executable instructions represented as a stainer normalization module 590.

[0134] The functionality described herein can be implemented partly via hardware logic in processor 510 and partly using instructions stored in memory 520. Some embodiments can be implemented using processor 510 without additional instructions stored in memory 520. Some embodiments are implemented using instructions stored in memory 520 for execution by one or more general-purpose microprocessors. In some example embodiments, a custom processor, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA), may be employed. Therefore, this disclosure is not limited to specific hardware and / or software configurations.

[0135] Refer again Figure 8 It should be understood that the example system shown in the figures is not intended to be limited to components that may be used in a given implementation. For example, the system may include one or more additional processors. Additionally, one or more components of the control and processing hardware 500 may be provided as external components that interface with the processing device.

[0136] While some implementations can be implemented in fully functional computers and computer systems, various implementations can be distributed as computing products in various forms and can be applied regardless of the specific type of machine or computer-readable medium actually used to implement the distribution.

[0137] At least some of the aspects disclosed herein can be embodied, at least in part, in software. That is, the technology can be implemented in a computer system or other data processing system in response to its processor (such as a microprocessor) executing a sequence of instructions contained in memory (such as ROM, volatile RAM, non-volatile memory, cache, or remote storage device).

[0138] Computer-readable storage media can be used to store software and data that, when executed by a data processing system, cause the system to perform various methods. Executable software and data can be stored in various locations, including, for example, ROM, volatile RAM, non-volatile memory, and / or cache. A portion of the software and / or data can be stored in any of these storage devices. As used herein, the phrases "computer-readable material" and "computer-readable storage medium" refer to all computer-readable media except for the transiently propagating signal itself.

[0139] Example

[0140] The following embodiments are presented to enable those skilled in the art to understand and practice embodiments of this disclosure. They should not be considered as limiting the scope of this disclosure, but only as illustrative and representative of this disclosure.

[0141] Example 1: An example implementation of a stainer-based deconvolutional network based on depth image priors

[0142] In some example implementations, each autoencoder neural network of the stainer deconvolution network can be implemented according to the U-Net design, where a U-shaped architecture consisting of a downsampling encoder and an upsampling decoder is used to generate the target image. As described above, in some example implementations, the input image is random noise and the convolutional filter size is 5x5. The encoder consists of 5 convolutional layers, each followed by Lanczos downsampling, batch normalization, and LeakyReLU activation. The decoder consists of 5 convolutional layers, each followed by bilinear upsampling, batch normalization, and LeakyReLU activation, with the exception of a final layer activated by Sigmoid activation. The scaling factor for upsampling and downsampling is 2, and the input to each layer is padded accordingly. Furthermore, layers 4 and 5 of the encoder are skipped to layers 2 and 1 of the decoder, respectively.

[0143] The model was implemented on a consumer-grade graphics processing unit (GPU), with most experiments performed on a GTX 1660 super GPU. It is believed that with code optimization, prediction speeds of a few seconds can be achieved on data center-grade GPUs.

[0144] This example algorithm was used to perform staining deconvolution on the MIDOG dataset (https: / / imi.thi.de / midog / the-challgenge / ), which contains 200 high-power field-of-view images (each approximately 7000x5000 pixels) generated by imaging 50 breast cancer tissue samples using four different whole-slice image scanners (Scanner 1: Hamamatsu XR nanozoomer 2.0; Scanner 2: Hamamatsu S360 (0.5NA); Scanner 3: Aperio ScanScope CS2; Scanner 4: Leica GT450). For each high-power field-of-view image, a site was randomly selected at x = 2000:2512, y = 2000:2512 to create 200 512x512 cropped image patches to evaluate the example method. The estimated physical parameters for each image were visualized in two-dimensional space using t-SNE. The simplified representation of the physical parameters forms four clusters, which clearly match the type of scanner used to image the samples.

[0145] This example implementation of a staining deconvolution method based on depth image priors explicitly encodes a set of variables in its network structure to describe the physical parameters involved in the light transmission from the stained tissue sample to the scanner camera. Compared to conventional staining deconvolution methods that output staining concentration maps and staining vectors via typical deep learning-based approaches, this example method outputs more variables that more clearly characterize nonlinearity and background illumination, thus providing improved interpretability.

[0146] Example 2: Theoretical Basis for Dye Spectral Correction Factor

[0147] Based on the Bouguer-Lambert-Beer equation, the linear relationship between absorbance A and concentration c is defined as follows:

[0148]

[0149] Where λ is wavelength, I is intensity, I0 is maximum intensity, δ is molar optical density representing the probability of light absorption against the disease, and c i It is the concentration of the i-th staining agent.

[0150] In practice, when a non-monochrome device (such as an RGB color camera) captures a colored image, Equation 1 may introduce systematic errors due to the non-color nature of the light signal and the resulting nonlinearity. To address this nonlinearity, the red, green, and blue spectral bands A... R A G A B The absorbance value can be modeled in the following ways:

[0151]

[0152] Where λ1 and λ2 are the lower and upper limits of visible light wavelengths, τ is the spectral transmittance, s is the spectral sensitivity function of the imaging device, and I0(λ) is the spectral intensity distribution of the illumination. According to Equation 2, the concentration c is no longer linearly related to the absorbance A. Based on the above model, the relationship between absorbance and concentration cannot be calculated precisely because of excessive nonlinearity, and it is almost impossible to accurately measure the parameters in the integral, such as the spectral sensitivity s of the imaging device for all visible light wavelengths.

[0153] If we assume that the device and environment are fixed for the target image during image capture, then absorbance A can be written as a function of concentration c as follows:

[0154] A = f λ (c) (3)

[0155] Although f cannot be measured λ The parameters are fixed, but consistent for pixels with the same concentration level. Therefore, they can be approximated by using the proposed staining agent to estimate the values ​​at certain concentration levels via a convolutional neural network. λ f λ By using a polynomial to f λ Modeling it as a function of c can reduce the impact of nonlinear systems.

[0156] Error. In digital pathology image analysis, only a few concentration levels are relevant for each staining agent, while the results of this experiment show that even for each type of staining agent, f λ Approximating a constant floating-point number (i.e., a constant) (hereinafter referred to as the dye spectral correction factor) can also greatly improve the deconvolution results.

[0157] like Figure 10 As shown, estimating nonlinear color signals using a linear model leads to significant errors (the difference between the blue and red curves). By introducing a spectral correction factor, the absorbance model can be adapted to approximate the formation of the underlying nonlinear signal. Figure 10 The rightmost figure shows an example of how a floating-point number (a constant parameter) used as a spectral correction factor can change the slope of a linear model to approximate the true absorbance value for the concentration level of interest (orange circle), such as for the target nucleus of a dye.

[0158] Example 3: Implementation of a stainer-based deconvolution network using depth image priors.

[0159] This example illustrates the deconvolution results achieved on images stained with three different dye combinations using an exemplary implementation of the algorithm presented in Example 1. Specifically, the model was tested on images of breast tissue stained with hematoxylin and eosin (H&E), images of bone marrow tissue stained with hematoxylin and 3,3'-diaminobenzidine (DAB), and images of breast tissue stained with SMACD34 MF3 antibody. The results are shown in [the table / examples]. Figure 11 In the first, second, and third rows.

[0160] like Figure 11 As shown, the proposed current implementation of the staining deconvolution network successfully combines all three types of staining for deconvolution into a density map, a color vector, and a background correction vector. All deconvolution components are combined to produce a generated image, which is then compared to the target image to compute a loss function during training. Combining all components except the background correction vector yields a background-illuminated version of the target image. Staining dosing can also be performed by calculating the ratio of positively stained pixels to all pixels.

[0161] Also included were bone marrow tissue images stained with DAB (DAB is a light scatterer) with and without spectral correction factors (respectively...). Figure 11 Rows 4 and 5 compare the performance of implementations of dye deconvolution networks. Dye deconvolution with spectral correction correctly estimates the color of DAB dye as brown (row 4), while the linear model (without spectral correction) yields a pink color vector that is clearly incorrect for DAB (row 5). This example shows that incorporating a spectral correction factor improves the performance of dye deconvolution networks for light scatterers.

[0162] The staining deconvolution performance was also compared with two state-of-the-art conventional staining deconvolution algorithms: the Macenko algorithm and the Vahadane algorithm. Tissue samples were first stained using DAPI fluorescence imaging to obtain the gold standard (concentration map) for hematoxylin-sensitive tissue layers. The samples were then washed and restained with hematoxylin and eosin (H&E). The DAPI images were registered to the corresponding H&E images, and Otsu thresholding was performed to obtain the true values. All three methods were applied to separate the staining agents, and quantitative comparisons were performed using the resulting hematoxylin concentration maps. Figure 12 and Figure 13 The results are shown in the figure. Two metrics were used for comparison: dotted bicolumn correlation, which measures the generalized correlation between high and low hematoxylin concentrations (concentration plots and true values), and structural similarity. (See figure from...) Figure 12 and Figure 13As can be seen, the method of the present invention is at least on par with the comparative method in terms of point bicolumn correlation, and is significantly better than the other two methods in preserving local structure.

[0163] Example 4: Stain normalization example using a stain deconvolution network based on depth image priors.

[0164] This section provides an example of stainer normalization using the proposed stainer deconvolution network implementation. The same implementation as in Example 1 is used, and stainer normalization is performed on the same dataset: the MIDOG dataset (https: / / imi.thi.de / midog / the-challgenge / ).

[0165] Figure 14 Example results of staining normalization using the proposed network are shown. Four image patches with different contents from four different scanners (Scanner 1: Hamamatsu XR nanozoomer 2.0; Scanner 2: Hamamatsu S360 (0.5NA); Scanner 3: Aperio ScanScope CS2; Scanner 4: Leica GT450) were selected for staining normalization. A concentration map and staining vector for each image were calculated using the staining normalization network, and then blended and matched to generate all 16 possible style and content combinations. For example, the first row shows the simulated appearance of content 1 scanned by the four scanners, and the second column shows the simulated appearance of all contents scanned by Scanner 2. Image patches scanned by different scanners can also be normalized using predefined 'standard' color vectors to make them appear style-similar, thus improving the performance of downstream analysis.

[0166] The specific embodiments described above have been illustrated by way of example, and it should be understood that these embodiments may allow for various modifications and alternatives. It should be further understood that the claims are not intended to limit to the specific forms disclosed, but rather to cover all modifications, equivalents, and alternatives falling within the spirit and scope of this disclosure.

Claims

1. A method for performing stainer deconvolution on a color target image, the method comprising: Provide a stainer deconvolution network, the stainer deconvolution network comprising: Multiple convolutional autoencoder neural networks, each configured to process a corresponding input dataset to generate a corresponding output dataset, each input dataset comprising a corresponding two-dimensional array, the size of which is equal to the pixel size of the color target image; An absorbance calculation module is configured to generate a color absorbance image using an absorbance model. The color absorbance image is generated by processing multiple dye vectors and multiple dye concentration maps, wherein each dye vector and each dye concentration map is associated with a corresponding dye. The absorbance calculation module is operatively coupled to the plurality of convolutional autoencoder neural networks, such that each staining concentration map is obtained from the output dataset of the corresponding convolutional autoencoder neural network; and The stainer deconvolution network is trained based on a stainer deconvolution loss function, which includes: A first loss component, configured to minimize the generation loss associated with the color target image; and A second loss component is configured to provide a repulsion loss between staining concentration maps and to promote separation between the staining concentration maps. Such that after the training, the staining concentration map correspondingly represents the deconvolutioned staining concentration map of the staining within the colored target image.

2. The method of claim 1, wherein the absorbance calculation module is further configured to generate the color absorbance image by calculating the sum of the products of the dye vector, the dye concentration map, and the dye spectral correction factor for each dye; Each dye has an associated dye spectral correction factor; and Each stainer spectral correction factor is updated during the training period based on minimizing the stainer deconvolution loss function.

3. The method of claim 2, wherein the staining spectral correction factor is defined according to the staining spectral correction factor parameter of the staining deconvolution network, and wherein the staining spectral correction factor parameter is initialized before the training.

4. The method of claim 2, wherein the staining deconvolution network comprises a plurality of spectral correction neural networks, each spectral correction neural network being configured to determine a corresponding staining spectral correction factor, and wherein each spectral correction neural network is trained according to the staining deconvolution loss function.

5. The method of claim 4, wherein at least one spectral correction neural network of the stainer deconvolution network is an encoder-decoder network.

6. The method of claim 5, wherein the dye spectral correction factor corresponding to the at least one spectral correction neural network is determined based on the potential characteristics of the at least one spectral correction neural network.

7. The method according to any one of claims 1 to 6, wherein the absorbance calculation module is further configured such that the calculation of the color absorbance image includes a color background vector; The color background vector is updated during the training period based on minimizing the stainer deconvolution loss function.

8. The method of claim 7, wherein the color background vector is defined according to the background parameters of the stainer deconvolution network, and wherein the background parameters are initialized prior to the training.

9. The method of claim 7, wherein the staining deconvolution network comprises a background neural network configured to determine the color background vector, and wherein the background neural network is trained according to the staining deconvolution loss function.

10. The method of claim 9, wherein the background neural network is an encoder-decoder network.

11. The method of claim 10, wherein the value of the color background vector is determined based on latent features of the background neural network.

12. The method according to any one of claims 1 to 6, wherein the staining vector is updated during the training period according to the minimization of the staining deconvolution loss function.

13. The method of claim 12, wherein the staining vector is defined based on staining vector parameters stored within the staining deconvolution network, and wherein the staining vector parameters are initialized prior to the training.

14. The method of claim 12, wherein the staining deconvolution network comprises a plurality of staining vector neural networks, each staining vector neural network being configured to determine a corresponding staining vector, and wherein each staining vector neural network is trained according to the staining deconvolution loss function.

15. The method of claim 14, wherein at least one of the staining vector neural networks of the staining deconvolution network is an encoder-decoder network.

16. The method of claim 15, wherein the value of the staining vector corresponding to the at least one staining vector neural network is determined based on the latent features of the at least one staining vector neural network.

17. The method of claim 14, wherein during at least the initial portion of the training, the staining deconvolution loss function includes an additional loss term based on the difference between the staining vector calculated by the staining vector neural network and a predetermined initial value of the staining vector.

18. The method of claim 17, wherein the additional loss term is included in the stainer deconvolution loss function during the initial portion of the training, but is not present in the stainer deconvolution loss function during the subsequent portion of the training.

19. The method of claim 12, further comprising using the staining vector to perform normalization when processing different color target images.

20. The method according to any one of claims 1 to 6, further comprising using the color absorbance image to generate an output image, thereby providing a regenerated version of the color target image.

21. The method according to any one of claims 1 to 6, wherein the first loss component is based on the difference between the color absorbance image and the target color absorbance image generated from the color target image.

22. The method according to any one of claims 1 to 6, wherein the first loss component is based on the difference between the color target image and an output image generated based on the color absorbance image.

23. The method according to any one of claims 1 to 6, wherein at least one input dataset is randomly generated.

24. The method according to any one of claims 1 to 6, wherein at least two of the input datasets are common input datasets.

25. The method according to any one of claims 1 to 6, wherein at least one of the convolutional autoencoder neural networks includes skip connections.

26. The method according to any one of claims 1 to 6, wherein during at least the initial portion of the training, at least one transformation of the input dataset is used to enhance the computation of the loss function.

27. The method of any one of claims 1 to 6, wherein the color target image is a first color patch of a master color image, and wherein the staining vector obtained after training is a final staining vector, the method further comprising employing the final staining vector when performing staining deconvolution on another color patch of the master color image.

28. The method of claim 27, wherein the final staining vector is used to initialize the staining vector when staining deconvolution is performed on the other color patch of the primary color image.

29. The method according to any one of claims 1 to 6, further comprising using the staining concentration map to perform staining dosing.

30. A system for performing stainer deconvolution on a color target image, the system comprising: A control and processing circuit, comprising at least one processor and a memory, the memory including instructions executable by the at least one processor to perform operations including: Generate a stainer deconvolution network, the stainer deconvolution network comprising: Multiple convolutional autoencoder neural networks, each configured to process a corresponding input dataset to generate a corresponding output dataset, each input dataset comprising a corresponding two-dimensional array, the size of which is equal to the pixel size of the color target image; An absorbance calculation module is configured to generate a color absorbance image using an absorbance model. The color absorbance image is generated by processing multiple dye vectors and multiple dye concentration maps, wherein each dye vector and each dye concentration map is associated with a corresponding dye. The absorbance calculation module is operatively coupled to the plurality of convolutional autoencoder neural networks, such that each staining concentration map is obtained from the output dataset of the corresponding convolutional autoencoder neural network; and The stainer deconvolution network is trained based on a stainer deconvolution loss function, which includes: A first loss component, configured to minimize the generation loss associated with the color target image; and A second loss component is configured to provide a repulsion loss between staining concentration maps and to promote separation between the staining concentration maps. Such that after the training, the staining concentration map correspondingly represents the deconvolutioned staining concentration map of the staining within the colored target image.

Images