Staining unmixing of multiplexed bright-field images

Abstract

This disclosure relates to stain unmixing of digital pathology images by determining initial color vectors associated with digital pathology stains (or chromogens) from pure color digital pathology images. The determined color vectors may be fine-tuned or adjusted to help improve stain unmixing performance. Adjustments may be performed via interfaces and / or automated techniques that perform adjustments to the color vectors based on actual multiplex images and one or more composite singleplex images. These adjusted color vectors may be further utilized for stain unmixing of a given multiplex image. Additionally, this disclosure provides techniques for generating multiplex images from one or more digital pathology images based on composite pixels and associated color vectors, recommended stains to be added to the multiplex image, and / or target color vectors.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit and priority of U.S. Provisional Application No. 63 / 499,098, filed on April 28, 2023, which is hereby incorporated by reference in its entirety for all purposes.

Background Art

[0002] Digital pathology facilitates accurately diagnosing a subject and guiding treatment decisions. In digital pathology solutions, an image analysis workflow is used to automatically detect or classify biological components of interest (e.g., cells having one or more specific proteins or antigens). A typical workflow of a digital pathology solution includes obtaining a tissue slide, scanning a pre - selected area or the entire tissue slide using a digital image scanner (e.g., a whole slide image (WSI) scanner) to obtain a digital image, and performing image analysis on the digital image. The digital image is processed using one or more image analysis algorithms, which can facilitate the detection of cells labeled with one or more signals of interest and the quantification of such signals using image analysis (e.g., quantitative or semi - quantitative scoring such as positive, negative, moderate, weak, etc.).

[0003] Digital pathology can utilize singleplex or multiplex techniques. Singleplex uses a single stain for only one biomarker, along with a reference stain. Multiplex, on the other hand, involves staining for two or more biomarkers (in addition to the reference stain) in a single slide or tissue sample. Therefore, the multiplex technique supports the simultaneous detection and co-expression of multiple biomarkers at the single-cell level. To process multiplex images, an unmixing process can be performed to separate signals from different markers. More specifically, a color unmixing method can be performed to decompose the RGB image for each biomarker into its individual constituent stains / dyes. This facilitates the estimation of staining levels for each of the multiple stains for individual cells.

[0004] One exemplary unmixing technique is color deconvolution, which can be used to unmix signals in an RGB image having up to three stains in the transformed optical density space. (See Ruifrok AC, Johnston DA Quantification of histochemical staining by color deconvolution. Anal Quant Cytol Histol. 2001 Aug;23(4):291-9. PMID:11531144, which is incorporated herein by reference in its entirety for all purposes.) Another exemplary unmixing technique formulates the color unmixing problem into non-negative matrix factorization (NMF), performing color separation in a fully automated manner and requiring no reference stain color selection. (See Lee, Daniel and H. Sebastian Seung. "Algorithms for non-negative matrix factorization." Advances in neural information processing systems 13 (2000), which is incorporated herein by reference in its entirety for all purposes.) However, each of these techniques may produce suboptimal results, including blurring, missing signals, and noise. Unmixing techniques become particularly difficult as the number of stains used increases. For example, in a triplex situation (with three biomarker stains and one reference stain), the unmixing technique attempts to convert a 3-channel RGB image into a 4-channel output, which can lead to inaccurate predictions.

[0005] These suboptimal results may be due to multiple stains co-localizing to the same type of cell region (e.g., multiple stains may co-localize to the cell nucleus, or multiple stains may co-localize to the cell membrane). For example, in multiplex imaging, a frequently used biomarker is hematoxylin, which is used to stain the cell nucleus, allowing pathologists to visualize tissue structure and determine which cells are negative for all biomarkers. When cells are stained in the images of the present invention, they are typically assigned to three main regions: nucleus, cytoplasm, and membrane. Biomarkers may be designed for any of these regions. In duplex imaging, up to three staining colors may appear on a cell, and depending on the particular biomarker, two or more may be present in one of these regions. This is called co-localization. However, as a result of co-localization, pixels may appear as different colors associated with distinct stains, and / or it may become difficult to estimate expression levels.

[0006] Suboptimal results may also, or alternatively, be due to staining bleeding across areas of cells that are not specific to the biomarker. For example, if purple staining adheres to the cytoplasm or membrane, purple is often added to the nucleus even if the staining is not specific to that part of the cell. Furthermore, current unmixing techniques typically rely on experimental results to identify each color vector. However, the true color vector may depend on tissue type, lighting, specific imaging device, etc. Therefore, it is advantageous to identify new unmixing techniques that more reliably produce higher quality results. [Overview of the Initiative]

[0007] Some embodiments of this disclosure relate to staining unmixing of digital pathology images by determining initial color vectors associated with digital pathology stains and adjusting the color vectors through a graphical user interface (GUI). The computer implementation method includes determining color vectors associated with (e.g., at least three) digital pathology stains (e.g., chromophores or fluorophores) from a given digital pathology image acquired using bright-field imaging. To facilitate the determination of color vectors, each pixel of the digital pathology image can be mapped from the RGB space to a position in the optical density (OD) space. Each fluorophore used to stain a sample may be associated with a predefined color vector, but performing the unmixing process using those color vectors may yield suboptimal results. This may be due to differences in imaging systems, lighting, etc., between facilities. For example, even if a “green” configured to have only green (without red or blue components) is used, the ambient lighting or imaging system associated with a particular facility may result in an image with some amount of red and / or blue intensity in the stained area.

[0008] Therefore, fine-tuning or adjusting the color vectors can help improve stain unmixing performance. Adjustments can be performed via an interface and / or automated techniques, which include a visual / graphic representation of the determined color vectors in a color space (e.g., Hue Saturation Density (HSD) space), and actual multiplex digital pathology images, hereafter referred to as multiplex images, depicting biopsy sections stained with at least three digital pathology stains associated with the determined color vectors. The interface may also include a composite singleplex image associated with each stain in the multiplex image. A composite singleplex image can be generated by filtering the multiplex image using the determined color vectors. For example, the tool could allow the user to adjust the color vector of a “green” dye to a vector containing several elements of red and / or blue, thereby better capturing the light component of the “green” signal in a given image and facilitating the unmixing of the given image.

[0009] Additionally, the interface may provide one or more color adjustment tools that allow the user to interactively adjust or fine-tune each determined color vector. Input received through user interaction with the interface may specifically target adjustments made to the determined color vectors associated with a particular stain. In response to the detection of input, the interface may be automatically updated (e.g., to represent the changed orientation / proximity of the determined color vectors in representation space and / or to show exemplary unmixing results generated using the adjusted color vectors). This process can support precise and responsive customization of stains according to the multiplexed image provided through the interface, thereby improving stain unmixing performance.

[0010] In some cases, one or more color vectors may be determined using one or more single-stain (or pure-color) images. These images can depict the same or different biopsy sections compared to the multiplexed images. Single-stained images may be stained with only one of the digital pathology stains used to stain the multiplexed images, but may be acquired in the same environment and using the same imaging system as the multiplexed images. Each representation of the determined color vector may also include a marker superimposed at a specific location within the digital pathology image. The specific location of the marker can be used to define the corresponding stain color vector (or initial color vector, which can then be adjusted based on user input or further processing). In one embodiment, the color vectors may be determined using non-negative matrix factorization (NMF).

[0011] Visualizing the synthetic singleplex image via an interface can be helpful in verifying the accuracy of the adjusted color vectors. The synthetic singleplex image can be regenerated by updates within the interface corresponding to the adjustments of the determined color vectors. These synthetic singleplex images can be further combined to generate a synthetic multiplex image that can be compared to the actual multiplex image during training and / or as an indicator of the confidence of the adjusted color vectors. For example, a graphical user interface could present both the synthetic multiplex image and the actual multiplex image, and user input adjusting one or more color vectors could trigger a dynamic update to the synthetic multiplex image.

[0012] In some embodiments, filtering may be performed by leveraging one or more machine learning models (such as generative models) that can be trained to learn mappings from a given multiplex image to its constituent singleplex image conditioned on one or more color vectors (defined using one or more techniques disclosed herein).

[0013] Once the color vectors are defined (for example, using one or more techniques disclosed herein), the color vectors can be used to unmix new multiplex images to generate a set of synthetic singleplex images (each corresponding to a given biomarker stain or reference stain). The new multiplex images can be stained using the same stains as the multiplex images used in the automated techniques for facilitating and / or color tuning. In some cases, the finely tuned color vectors may be determined based on different multiplex images than the new multiplex images used for further stain unmixing (for example). The finely tuned color vectors can then be used to generate one or more synthetic singleplex images from the new multiplex images. In some cases, these synthetic singleplex images are generated from the same multiplex images used in the facilitating and / or automated techniques to support the fine-tuning of the color vectors.

[0014] In some embodiments, the disclosure provides a method for determining the adjustment of initial color vectors associated with a particular stain based on a given actual multiplex image that may not contain the particular stain. A computer implementation method includes, for example, determining initial color vectors associated with at least three stains from a corresponding pure-stained digital pathology image. The method may further include accessing an actual multiplex image stained with one or more stains associated with the initial color vectors but not containing at least one of these stains. For reference, at least one stain that is not present in the actual multiplex image is referred to as the “particular stain.” The initial color vectors may be fed into a filter configured to produce an output filtered from the actual multiplex image based on the particular stain. Filtering can be performed by utilizing one or more machine learning models, such as a generative model trained to learn the mapping from a given multiplex image to its constituent singleplex images. A generative model, such as a GAN, may be conditioned on color vectors such that when the model encounters a color vector that is not present in the input multiplex image, it cannot produce any meaningful output associated with that stain, resulting in a zero or null image. Conversely, if such a model is given color vectors present in a given multiplex image, it can generate a constituent singleplex image associated with those color vectors.

[0015] To evaluate the quality of the filtered output, a metric can be calculated that characterizes the degree of variation in stain intensity across all or part of the filtered output. For example, if it is predicted (or known) that there is no biomarker corresponding to a given stain (or color vector) in the actual multiplex image, then when conditioned on an accurate color vector, the metric (e.g., mean, median, mode, variance, standard deviation, and / or range) may be predicted to be relatively low compared to when a less accurate color vector is used. Once a metric is determined that quantifies the filtered output based on the extent to which stain is present in the actual multiplex image, spatial traverse techniques (e.g., gradient descent, Monte Carlo) can be used to find color vector adjustments associated with a particular stain. These adjustments can be incorporated into the color vector of a particular dye via an interface.

[0016] A new multiplex image may be received when the color vector associated with a particular stain is adjusted by minimizing a metric. This multiplex image may be stained with at least one of the stains associated with the initial color vector. It may also include a specific stain whose color adjustments are calculated based on the metric and spatial shift techniques. By leveraging the stain unmixing process of unmixing techniques such as NMF, a new synthetic singleplex image associated with a specific stain can be generated. Finally, the generated unmixed output can be displayed via an interface.

[0017] In another example, a technique may be provided for finding a recommended color vector for a given multiplex image. For example, a duplex image is stained with two specific stains along with a counterstain (e.g., hematoxylin). The objective may be to identify potential additional stains that are distinguishable among the existing stains, thereby converting the duplex to a triplex image. A computer implementation method may include determining an initial color vector associated with, for example, at least two stains from a corresponding pure-stained digital pathology image. This method may further include accessing an actual multiplex image stained with at least two stains associated with the initial color vector. The additional stains may be selected so as not to be associated with any of the stains in the multiplex image in a multidimensional color space.

[0018] Initial staining can be selected or designed, and the associated initial color vector can be determined, for example, by using the NMF technique. Using this initial color vector, the actual multiplex image can be filtered by leveraging a machine learning model configured to map a given multiplex to one of its constituent composite singleplex images based on the provided color vector. To characterize the filtered output, a metric (e.g., mean mode or median) can be calculated, which quantifies the amount of staining present in the given multiplex image. For example, if the staining of the selected color is indistinguishable, the calculated metric, such as the mean of the composite OD singleplex image, may have a higher value, thus indicating the presence or similarity of existing staining. The metric can be estimated by leveraging a spatial traverse technique that can include one or more objectives in the traverse. Corresponding adjustments to the initial color vector can be found based on a spatial traverse technique that minimizes the metric of the filtered output. Finally, a recommended color vector unrelated to any staining already present in the multiplex image may be output through the interface.

[0019] Other aspects of the present disclosure include a method for determining a performance prediction score representing the predicted degree to which at least three digital pathology stains are actually sufficiently separable to reliably support the generation of a synthetic singleplex image. The method may include determining, for example, initial color vectors associated with at least three stains from a corresponding pure-stained digital pathology image. An actual multiplex image stained using one or more stains associated with the initial color vectors but not including at least one of these stains (referred to as "specific stains") can be accessed. The initial color vectors may be fed into a filter configured to produce a filtered output from the actual multiplex image based on the specific stains. One or more machine learning models, such as generative models, may be trained to learn a mapping from a given multiplex image to its constituent singleplex images for filtering purposes. If such a model is conditioned on color vectors that are not present in the input multiplex image, it will not be able to produce any meaningful output associated with its stains, resulting in a result of 0 or a null image. Performance prediction scores may be generated for the filtered output and / or for other synthetic singleplex images that constitute the actual multiplex image. A performance prediction score can be output, and adjustments to the initial color vector can be performed via a GUI based on this performance prediction score.

[0020] In some cases, the performance prediction score may include the mean, median, or mode intensity of the corresponding filtered output. For example, if a biomarker is predicted (or known) to exist corresponding to a given stain in a given depicted sample or multiplexed image, then the performance prediction score, e.g., mean, median, mode, variance, standard deviation, and / or range, may be predicted to be relatively higher when an accurate color vector is used compared to when a less accurate color vector is used.

[0021] In another example, the performance prediction score can be estimated by grouping or clustering similar stains together based on the staining features of one or more singleplex images. For example, the staining features can include optical density values, color histograms, or any other features that can effectively capture the staining pattern. In yet another example, the performance prediction score can be calculated for a composite singleplex image by estimating the correlation between each staining pattern observed in the multiplex image.

[0022] In some embodiments, stain unmixing is performed by a constraint method that can reduce the complexity of multiplex images (e.g., stained with four stains) and thus support the more precise and / or more reliable generation of a synthetic singleplex image from the multiplex image. The computer implementation method includes determining initial color vectors associated with, for example, at least four stains from the corresponding pure-stained digital pathology image. In some cases, each pixel in the digital pathology image may be mapped to a position in a multidimensional color space. Of these four stains, a particular stain may be selected so as to be due to a prominent part of the color space (e.g., quadrants, parts defined by being greater / less than some y value and greater / less than some x value, wedge-shaped, cylindrical, etc.). The method further includes accessing an actual multiplex image stained with at least three digital pathology stains. Each pixel in the actual multiplex image may also be mapped to a point in a multidimensional space. For each pixel, a pixel-specific vector can be generated that predicts the degree of expression of each of the at least four stains in the portion of the biopsy section depicted in the pixel.

[0023] The process of generating pixel-specific vectors may further involve assigning pixels within a specific portion of a color map to a specific color vector that predicts the expression level of the biomarker corresponding to that portion. For each pixel associated with a particular portion, an optical density can be determined. Pixels outside that portion can be assigned "0" (or other predefined expression levels) for each of the other biomarkers corresponding to the multiplex image. For pixels outside the first portion, an unmixing technique can be used to predict the expression level of each of the other biomarkers, and the predicted expression level of "0" (or another predefined number) can be associated with the first biomarker. Finally, one or more composite singleplex images can be generated using the pixel-specific color vectors.

[0024] Specific stains may be selected based on information regarding which parts of the cell each of at least four digital pathology stains is configured to stain. The color space may include the International Commission on Illumination (CIE) color space. Parts of the color space may include wedge shapes. Parts of the color space may include parts of the space defined based on inequalities with respect to the x-coordinate and y-coordinate. Parts of the color space may include combinations of primitives. Performing an unmixing technique may include using non-negative matrix factorization (NMF). Color vectors may be determined based on one or more user inputs received using one or more color vector adjustment tools available within the interface.

[0025] In some cases, a computer implementation method is provided, comprising: determining a color vector representing a stain for each of at least three digital pathology stains; utilizing an interface for a user device, the interface comprising: each of the determined color vectors; an actual multiplex digital pathology image depicting a biopsy section stained with two or more of the at least three digital pathology stains; at least one composite singleplex image, each of which is generated by filtering the actual multiplex digital pathology image using a single color vector from the determined color vectors; and one or more color vector adjustment tools, each of which is configured to receive user input corresponding to an adjustment of a color vector representing a corresponding stain from the at least three digital pathology stains; detecting input received through interaction with the interface corresponding to a specific adjustment of a color vector representing a particular stain from the at least three digital pathology stains; and automatically updating the interface in response to the detection of input.

[0026] Each representation of the determined color vector may include a representation of its position in optical density space. The updated interface may further include at least one composite singleplex image. One or more color vector adjustment tools may include at least three color adjustment tools. Determining the color vectors may involve processing one or more single-stain images depicting the same or other biopsy sections stained with only one of at least three digital pathology stains. One or more single-stain images may include markers superimposed at specific positions in a multidimensional color space, and the color vectors are defined based on these specific positions. The actual multiplex digital pathology image may depict biopsy sections stained with at least four stains. The determined color vectors may reside in a two-dimensional color space, and the method further includes determining a portion of the color space that is expected to be attributable to a prominent signal corresponding to a specific stain among at least three digital pathology stains, and the automatic updating of the interface is performed using an unmixing technique that selectively focuses on at least three digital pathology stains by excluding specific stains. The determination of the color vectors may be performed using non-negative matrix factorization. The method may further include receiving a new multiplex image stained with at least one of at least three digital pathology stains, generating a new synthetic singleplex image based on the new multiplex image and adjusted color vectors, and outputting the new synthetic singleplex image. The actual multiplex digital pathology image may be filtered using color vectors and machine learning models.

[0027] In some embodiments, a computer-implemented method includes, for each of at least three digital pathology stains, determining a color vector representing the stain; accessing an actual multiplex digital pathology image depicting a biopsy section stained with at least one first stain of the at least three stains, wherein the depicted biopsy section is not stained with at least one second stain of the at least three stains; generating a filtered output by filtering the actual multiplex digital pathology image using a color vector representing one of the at least one second stains; generating a metric characterizing signal characteristics in the filtered output; using the metric and a spatial traversal technique to identify an adjustment to the color vector representing the second stain; receiving a new multiplex image stained with at least one of the at least three digital pathology stains; generating a new synthetic singleplex image based on the new multiplex image and the adjusted color vector representing the second stain; and outputting the new synthetic singleplex image.

[0028] For each of the at least three digital pathology stains, the color vector may be a vector in an optical density space. The spatial traversal technique may include a gradient descent technique. The spatial traversal technique may include a Monte Carlo technique. The metric may include an average, median, or mode intensity. The metric may characterize the level of staining over all or a portion of the filtered output. The filtered output may be generated by using a machine learning model.

[0029] In some embodiments, a computer-implemented method includes, for each of at least two digital pathology stains, determining a color vector representing the stain; accessing an actual multiplex digital pathology image depicting a biopsy section stained with the at least two digital pathology stains; generating a filtered output by filtering the actual multiplex digital pathology image using an initial color vector, identifying the initial color vector, generating a metric characterizing signal characteristics in the filtered output, and identifying a recommended color vector using the metric and a spatial traversal technique; and outputting the recommended color vector.

[0030] The spatial traversal technique may be performed to include, as one or more goals in the traversal, minimizing a signal in the filtered output. Minimizing a signal in the filtered output may include minimizing an average, median, or mode intensity of the corresponding filtered output. Determining the color vector may be performed using non-negative matrix factorization. For each of the at least two digital pathology stains, the color vector may be a vector in an optical density space. The filtered output may be generated by using a machine learning model. The spatial traversal technique may include a gradient descent technique.

[0031] In some embodiments, a computer implementation method is provided, comprising: determining a color vector representing the stain for each of at least three digital pathology stains; accessing an actual multiplex digital pathology image depicting a biopsy section stained with at least one first stain from the at least three digital pathology stains, wherein the depicted biopsy section is not stained with at least one second stain from the at least three stains; generating a filtered output by filtering the actual multiplex digital pathology using a color vector representing one second stain from the at least one second digital pathology stain; generating a performance prediction score representing the predicted degree to which the at least three digital pathology stains are actually sufficiently separable to ensure support for the generation of a synthetic singleplex image; and outputting the performance prediction score.

[0032] The performance prediction score may be generated using filtered outputs. The performance prediction score may include the mean, median, or mode intensity of the corresponding filtered outputs. The performance prediction score may include the correlation coefficient between each pair of synthetic singleplex images associated with the filtered outputs. For each of the at least three digital pathology stains, the color vector may be a vector in optical density space. The filtered outputs may be generated by using a machine learning model. The color vector may be adjusted via a graphical user interface (GUI) based on the performance prediction score.

[0033] In some embodiments, a computer implementation method comprising: determining a color vector representing a stain for each of at least four digital pathology stains, wherein the determined color vector is in a multidimensional color space; selecting a specific stain from the at least four digital pathology stains; determining a portion of the color space that is expected to be due to a prominent signal corresponding to the specific stain; accessing an actual multiplex digital pathology image depicting a biopsy section stained with at least three of the at least four digital pathology stains, wherein the actual multiplex digital pathology image includes a set of pixels; mapping each pixel in the set of pixels in the actual multiplex digital pathology image to a point in a multidimensional color space; and generating a pixel-specific color vector for each of the at least four digital pathology stains, predicting the degree of stain expression in the portion of the biopsy section depicted in the pixels, wherein generating the pixel-specific color vector is A computer implementation method is provided, comprising: determining that each of a first subset of pixels maps to a point within the aforementioned portion of a color space; determining the optical density for each pixel in the first subset of pixels, such that the pixel-specific color vector of the pixel identifies the degree of expression of a particular stain corresponding to the optical density; determining that each of a second subset of pixels maps to a point outside the aforementioned portion of a color space; performing an unmixing technique for each pixel in the second subset and each of at least four digital pathology stains to predict the degree of expression of the stain in the aforementioned portion of a biopsy section depicted in the pixel, such that some of the at least four digital pathology stains do not include a fixed stain, and the unmixing technique uses the color vector determined to represent each of the at least four digital pathology stains; and generating one or more composite singleplex images using the pixel-specific color vectors.

[0034] Specific stains may be selected based on information regarding which parts of cells each of at least four digital pathology stains is configured to stain. The color space may include the International Commission on Illumination (CIE) color space. Parts of the color space may include wedge shapes. Parts of the color space may include parts of space defined based on inequalities with respect to the x-coordinate and inequalities with respect to the y-coordinate. Parts of the color space may include combinations of primitives. Performing an unmixing technique may include using non-negative matrix factorization (NMF). The color vector may be determined based on one or more user inputs received using one or more color vector adjustment tools available within the interface. In some embodiments, a computer program product tangibly embodied in a non-temporary machine-readable storage medium includes instructions configured to cause one or more data processors to perform some or all of the methods or processes disclosed herein.

[0035] In some embodiments, a system is provided which includes one or more data processors and a non-temporary computer-readable storage medium that, when executed on the one or more data processors, contains instructions causing the one or more data processors to perform some or all of the methods disclosed herein.

[0036] In some embodiments, a system is provided which includes one or more means for performing some or all of one or more methods or processes disclosed herein.

[0037] The terms and expressions used are for illustrative purposes only, not limitation, and in using such terms and expressions there is no intention to exclude equivalents or parts of the features shown and described, however it is acknowledged that various modifications are possible within the scope of the invention as described in the claims. Accordingly, while the invention as described in the claims is specifically disclosed by embodiments and optional features, modifications and variations of the concepts disclosed herein may be used by those skilled in the art, and it should be understood that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims. [Brief explanation of the drawing]

[0038] The patent or application file shall contain at least one drawing made in color. A copy of this patent or patent application publication containing the color drawing shall be provided by the Patent Office upon request and payment of the required fee. This disclosure shall be described in conjunction with the following attached drawings.

[0039] [Figure 1] A diagram illustrating a workflow for acquiring and processing multiplexed images according to several embodiments of the present disclosure.

[0040] [Figure 2] A diagram showing an exemplary network for generating digital pathology images.

[0041] [Figure 3A] A diagram illustrating an exemplary workflow for facilitating the definition of a color vector associated with staining and for performing staining unmixing, according to one embodiment of the present disclosure.

[0042] [Figure 3B] A figure illustrating exemplary linear unmixing techniques according to several embodiments of the present disclosure.

[0043] [Figure 3C]A diagram showing interface components that facilitate fine-tuning of color vectors.

[0044] [Figure 3D] A diagram illustrating an exemplary architecture for generating one or more composite images by utilizing multiple machine learning models.

[0045] [Figure 3E] A diagram illustrating an exemplary architecture for generating one or more composite images by leveraging a single machine learning model.

[0046] [Figure 4] A flowchart illustrating an exemplary process for facilitating the definition of one or more color vectors for stain unmixing, according to some embodiments of the present disclosure.

[0047] [Figure 5] A figure illustrating a system, according to some embodiments of the present disclosure, for determining the adjustment of initial color vectors based on a given multiplex digital pathology image.

[0048] [Figure 6] A flowchart illustrating an exemplary process for generating a composite singleplex image using finely tuned color vectors.

[0049] [Figure 7] A flowchart illustrating the process for identifying recommended color vectors.

[0050] [Figure 8] A flowchart illustrating an exemplary process for generating a performance prediction score that represents the expected degree to which constituent digital pathology stains are sufficiently separable, in order to reliably support the generation of one or more synthetic singleplex images.

[0051] [Figure 9A]This figure shows an example of an ER-PR-HER2 triplex image where each pixel of the triplex image is mapped to a position in a multidimensional color map.

[0052] [Figure 9B] A diagram of staining unmixing of an exemplary triplex ER-PR-HER2 image from Figure 9A, according to some embodiments of the present disclosure.

[0053] [Figure 9C] A figure showing examples of staining unmixing results for ER-PR-HER2 triplex and one or more singleplex images using the disclosed constraint techniques.

[0054] [Figure 9D] Figure showing examples of staining and remixing results for ER-PR-HER2 triplex and one or more singleplex images using the disclosed constraint technique.

[0055] [Figure 9E] A figure showing the staining and remixing results of triplex ER-PR-HER2 according to several embodiments of the present disclosure.

[0056] [Figure 10A] An exemplary process flowchart for performing staining unmixing for multiplexed images by using the constraint techniques disclosed in some embodiments of this disclosure.

[0057] [Figure 10B] An illustrative flowchart further illustrating the components of Figure 10A.

[0058] [Figure 11A] A figure illustrating a comparison of staining unmixing of duplex images using an initial color matrix and an adjusted color matrix, according to an exemplary embodiment.

[0059] [Figure 11B]A figure illustrating a comparison of staining unmixing of different duplex and singleplex images using an initial color matrix and an adjusted color matrix, according to an exemplary embodiment.

[0060] [Figure 12A] This figure shows an example of a duplex image overlaid with markers (candidate species) for each nucleus detected by automated nuclear segmentation.

[0061] [Figure 12B] This figure shows a comparison of nuclear segmentation results for hematoxylin images obtained by unmixing duplex images using linear inverse convolution and NMF techniques.

[0062] [Figure 13] A diagram showing an exemplary graphical user interface (GUI) for generating composite pixels according to an exemplary embodiment.

[0063] [Figure 14A] An illustrative GUI showing the use of composite pixels to evaluate the color range from a blend of multiple stains.

[0064] [Figure 14B] A figure illustrating a comparison of one or more blended colors synthesized from stains from different reagent sources, according to an exemplary embodiment.

[0065] [Figure 14C] A diagram illustrating an example of blending two or more dyes to generate a range of colors, according to an exemplary embodiment.

[0066] [Figure 14D] A figure illustrating the evaluation of the color range assigned to hematoxylin using wedge constraints. [Modes for carrying out the invention]

[0067] Some embodiments of this disclosure relate to the unmixing of digital pathology images labeled with four or more markers (e.g., three or more biomarkers and a reference stain), wherein the digital pathology image has three or fewer channels (e.g., red, green, and blue channels). A color vector can be defined for each of the markers, and then an unmixing technique can be performed using the color vectors to separate the signals (corresponding to the four or more markers) in the digital pathology image. These color vectors can be defined using the optical density space (e.g., instead of using the RGB space, or in addition to it). Then, each pixel in the input multiplex image can be mapped from the RGB space to a position in the optical density space, where initial unmixing can be performed.

[0068] In some cases, color vectors may be determined by inputting a pure color image (e.g., showing a slice or sample stained with a single marker) into a linear technique such as non-negative matrix factorization (NMF). However, color vectors obtained from NMF can be error-prone when used for unmixing. For example, background noise, faded tissue, or obscured morphology can lead to scenarios where the initial color vectors fail to account for signals represented in the image captured in a real-world environment.

[0069] In some embodiments, fine-tuning of one or more color vectors may be performed using an interactive graphical user interface (GUI) and / or automated techniques. Such fine-tuning may be performed using images obtained in a specific environment (e.g., lighting), and as a result, the color vectors may be defined to account for the imaging effects specific to the real-world environment. For example, one or more color vectors may be defined and / or adjusted to account for any effects that the imaging system and / or lighting environment may have on the signal of a given marker in a digital pathological image or on its depiction.

[0070] The GUI can present actual multiplex images depicting slices stained with multiple dyes. The GUI may include one or more input components configured to adjust (i.e., fine-tune) the definition of one or more color vectors. For example, one or more input components may be configured to move or adjust the representation of a color vector in optical density space or RGB space. As another example, one or more input components may be configured to adjust the representation of one or more channels in a color space (e.g., one or more contributions from the red, blue, or green channels).

[0071] The GUI may also include one or more composite singleplex and / or composite multiplex images, each composite being generated (e.g., dynamically) based on a color vector defined in the interface. The GUI may include one or more input components configured to receive inputs that adjust the contributions of one or more channels corresponding to a given signal.

[0072] For example, with respect to a given marker, the GUI may be configured to receive definitions or adjustments of one or more color or frequency band channels. As another example, with respect to a given marker, the GUI may be configured to receive definitions or adjustments of the hue angle and / or optical density (representing intensity) in optical density space. The optical density space can be configured to be a two-dimensional space (e.g., a chromaticity cx-cy plane), where each position is a non-ambiguous identification of an RGB vector (e.g., a position in optical density can be deconvolved to identify a position in optical density space). Within this space, arbitrary scaling factors corresponding to angles can be defined such that the color space spans a predefined space. Within the optical density space, saturation may be represented by the distance from the center, and / or hue may be captured by an angle in polar coordinates.

[0073] The GUI can be configured to dynamically adjust (e.g., in real time) one or more displayed singleplex images and / or composite multiplex images based on a set of color vectors defined (via the interface) for the underlying channels. For example, if the underlying images are set to have identical color vectors when stained with different markers, the GUI can indicate that all composite singleplex images are identical and that the composite multiplex images lack signals from the corresponding actual multiplex images. The user can then use this information to fine-tune the color vectors.

[0074] Once fine-tuning is complete, the color vector can be used to generate one or more composite singleplex images based on the input multiplex image. The input multiplex image to be unmixed may be different from or the same as the multiplex image used to determine the color vector. By leveraging similar multiplex images, the extent to which color variations across imaging instances (e.g., due to differences in tissue type, lighting, staining protocol, imaging system, etc.) affect the ability to accurately detect labels in a given instance can be reduced.

[0075] In some cases, unmixing can be performed using NMF techniques that leverage finely tuned color vectors and coefficient matrices of input (same or different) multiplex images, thereby generating a synthetic singleplex image. As another example, stain unmixing can be performed non-linearly by leveraging machine learning models such as (e.g.) autoencoders or generative adversarial networks (GANs).

[0076] In one aspect of this disclosure, a GUI may be configured to generate a color vector of a composite stain by synthetically and interactively blending two or more staining colors in different ratios. The color of the composite stain may be displayed on the chromaticity plane cx-cy via the GUI. The composite stain may be generated by selecting multiple chromophores (or fluorophores) from a plurality of pre-identified chromophores (or fluorophores) for mixing via user interaction. The user input can then identify the relative contribution of each of the selected chromophores or fluorophores. The stain color may also be blended by generating a weighted average of the corresponding color vectors of the selected stains in OD space, where the weights are defined based on the relative contributions. The weighted average in OD space can then be converted back to RGB space (for example, for display purposes).

[0077] In some cases, the synthetic pixels or associated tuned color vectors obtained using the techniques disclosed above may be used to generate synthetic singleplex and / or synthetic multiplex images. For example, a machine learning model may be trained to convert an input contrast stained image (e.g., hematoxylin) or an input multiplex image into a synthetic image based on a given tuned color vector. The synthetic image may be used to verify the extent to which a defined color vector (e.g., based on user input) provides a basis for accurate unmixing and / or accurate mixing. Architectures that generate synthetic images may also help create additional training data for machine learning models in a faster and more cost-effective way than performing actual staining experiments in the laboratory. This may also allow pathologists to control the appropriate combination of different staining conditions, intensities, and biomarkers for, for example, synthetic multiplex images. These synthetic multiplex images can be tailored to specific needs and applications.

[0078] In some embodiments, techniques may be provided for determining the adjustment of initial color vectors based on a given actual digital pathology image. The actual image may be stained using one or more (but not all) stains associated with the initial color vectors (in the ongoing description, stains not used are referred to as “excluded stains”). One or more generative models (e.g., including one or more autoencoders (AEs), one or more image-to-image transformation networks, one or more generative adversarial networks (GANs), etc.) may be used to generate one or more synthetic singleplex images using one or more corresponding color vectors (e.g., at least one of the one or more color vectors is defined based on user input received via the interface described herein). Assuming that one or more stains are known to be excluded, the target output corresponding to those staining channels will lack any signal. Therefore, if the synthetic singleplex image generated in response to the excluded stains contains a signal (or, for example, a signal that subjectively or objectively exceeds a threshold), then the one or more color vectors used to generate the synthetic singleplex image can be presumed to be suboptimal.

[0079] In some cases, the synthetic singleplex image may be available (e.g., displayed) on the user device from which the input used to define one or more color vectors originated. Such availability may be provided in real time or near real time as the user adjusts one or more color vectors. In some cases, metrics (e.g., cumulative absolute intensity, variation over intensity, maximum intensity) may be calculated and used to automatically adjust one or more color vectors (e.g., using the metrics and a loss function associated with one or more machine learning models to generate the synthetic singleplex image). For example, if it is predicted (or known) that no biomarker corresponding to a given stain (or color vector) is present in the actual multiplex image, it may be predicted that when an accurate color vector is used, the mean, median, mode, variance, standard deviation, and / or range may be relatively low (or zero) compared to when a less accurate color vector is used. Once a metric is determined to quantify the synthetic singleplex output based on the extent to which stains are present in the actual multiplex image, spatial traverse techniques may be employed to find color vector adjustments associated with excluded stains. Spatial shift techniques can systematically explore the space of possible adjustments to the color vectors representing excluded stains. Examples of such techniques may include, but are not limited to, gradient descent, Monte Carlo methods, genetic algorithms, or other stochastic optimization techniques that iteratively adjust color vectors to optimize a specific criterion, such as minimizing a calculated metric. This adjustment is repeated iteratively until convergence or a stopping criterion is met. The goal is to find the optimal color vector that minimizes the metric, resulting in a synthetic singleplex image that accurately represents the excluded biomarkers.

[0080] When the color vectors associated with the excluded stains are adjusted by minimizing a metric, a new multiplex image may be received. The multiplex image may be stained with stains associated with the initial color vectors (including one or more excluded stains whose color adjustments are calculated based on spatial shift techniques and metrics). By leveraging the aforementioned stain unmixing process, one or more new synthetic singleplex images associated with the excluded stains may be generated.

[0081] In yet another example, the disclosed technique may also be used to identify a recommended color vector for a stain that can complement other stains depicted in a given multiplex image. This multiplex image may be stained with at least two stains. For example, a duplex image may be stained with two specific stains along with a counterstain (e.g., hematoxylin). The objective may be to identify potential additional stains that are effectively distinguishable among the existing stains. Thus, a high score may be assigned via the objective function if the unmixing result can accurately distinguish between different staining signals (e.g., signals from one or more existing stains and one or more potential additional stains).

[0082] The interface may be configured to receive user input that identifies the color vectors for an additional stain and, if the additional stain is used with one or more existing stains, presents one or more predicted unmixed outputs (e.g., one or more composite singleplex images). Additionally or alternatively, the color vectors for the additional stain may be initially selected automatically (e.g., using a predefined selection of color vectors, a default user selection of color vectors, or initial results from linear or nonlinear processing). For example, the interface may be configured to receive user input that identifies a particular chromophor or fluorophore, and the color vector associated with that particular chromophor or fluorophore may be initially assigned to the additional stain.

[0083] Using one or more color vectors defined according to the techniques disclosed herein, an actual multiplex image can be converted into one or more synthetic singleplex images (e.g., using the unmixing techniques disclosed herein, e.g., linear unmixing techniques, nonlinear unmixing techniques, or machine learning models). One or more metrics can be calculated to characterize the quality of one or more synthetic singleplex images. For example, in a scenario where the input image depicts a sample slice that was not stained with a given stain (but was stained with, e.g., one or more other stains), the metric can quantify the extent to which a signal associated with the given stain is present in the synthetic singleplex image. For example, the metric could be the mean, median, maximum, or range of intensity in the synthetic singleplex image. In this scenario, the ideal synthetic singleplex image would contain no signal (because we know that the given stain was not present in the initial slice), and therefore the ideal metric would be 0. The metrics and / or synthetic singleplex images may be presented on an interface so as to allow the user to fine-tune one or more color vectors.

[0084] In another scenario, a metric can be calculated that characterizes the synthetic singleplex image corresponding to the stain actually used to stain the corresponding multiplex slice. In this scenario, the signal component is predicted in the synthetic singleplex image, and therefore a metric that is not close to zero can be predicted (if the slice is known to have a biomarker corresponding to the stain).

[0085] A performance prediction score may be generated using one or more metrics, potentially using one or more target metrics. For example, a performance prediction score (or its contributing components) may be defined to positively correlate with a metric in the synthetic singleplex image that characterizes the presence of a signal (e.g., mean, median, mode, maximum) or the complexity of the signal (e.g., variability or range), if the sample depicted in the corresponding multiplex image is known to have a signal from the stain associated with the synthetic singleplex image. Furthermore, a performance prediction score (or its contributing components) may be defined to negatively correlate with a metric in the synthetic singleplex image that characterizes the presence of a signal or the complexity of the signal, if the sample depicted in the corresponding multiplex image is known not to have a signal from the stain associated with the synthetic singleplex image (e.g., because the stain was not applied to the sample). Thus, a performance prediction score may be generated to represent the extent to which the stain can be accurately detected and / or distinguished in the multiplex image.

[0086] In some cases, the performance prediction score may be estimated by further, or alternatively, performing clustering analysis based on image features associated with multiple synthetic singleplex images. For each synthetic singleplex image, one or more features may be defined or learned to characterize (e.g.) optical density values ​​in the image, RGB values ​​in the image, etc. For example, a feature may include statistics across each of one or more axes in optical density or RGB space (e.g., mean, median, range, maximum, variance mode, etc.). As another example, a feature may characterize spatial contrast of intensity (e.g., where contrast correlates with the amount and / or degree to which intensity differs across adjacent or neighboring pixels). Features may be clustered using clustering techniques (e.g., k-means, hierarchical clustering, or spatial clustering of applications with density-based noise (DBSCAN)). For example, k-means clustering may be used if the number of clusters is defined (e.g., equal to the number of stains applied to the scenario, or the number of stains plus one or more other categories such as blank signal categories). Such a clustering algorithm divides the feature space into clusters. Ideally, such clusters can be sufficiently isolated and compact from one another, and the image features associated with each given type of stain can be clustered together. Performance prediction scores (or their contributing components) can be based on the degree to which clusters are separated in feature space, the degree to which the synthetic singleplex images corresponding to a given color vector / stain are clustered together, and / or the degree to which the images assigned to a given cluster are close together in feature space. Such degrees can be quantified using (e.g.) silhouette scores, Davies-Bouldin indices, or distances (e.g., Euclidean distance, Mahalanobis distance, or Manhattan distance).

[0087] The performance prediction score may, additionally or alternatively, be based on estimated correlations between one or more synthetic singleplex images and their corresponding multiplex images. These correlations can be estimated in RGB space, optical density space, feature space, etc. This method can account for variations in staining protocols, image acquisition settings, and tissue characteristics, thereby providing a consistent basis for comparison. With respect to optical density space, the values ​​are inherently in the non-negative to positive range, thereby aligning well with the physical constraints of staining intensity.

[0088] For unmixing purposes, in one aspect of the present disclosure, constraints may be introduced to simplify the stain analysis, thus reducing the complexity associated with stain unmixing. This can facilitate greater accuracy, precision, and / or reliability in generating a composite singleplex image from a given multiplex image. Each pixel in the multiplex image may be mapped to a position in a multidimensional color map. Pixels within a particular portion of the color map (e.g., quadrants, portions defined by y values ​​greater than / less than and x values ​​greater than / less than, wedge shapes, etc.) may be assigned and characterized as representing signals corresponding to only a single particular stain. For example, in optical density space, a given angular range may be defined so as to be associated with a particular stain. For each pixel associated with a position within the angular range, it can be inferred that the pixel depicts a representation of a given stain. Furthermore, the intensity of the stain may be estimated (at least partially) based on the distance of the pixel representation's position from the axis. For pixels outside the angular range, the unmixing technique can predict the expression levels of other biomarkers and maintain predefined expression levels, such as "0" or another predefined number for the first biomarker.

[0089] To facilitate the extraction of specific portions from a color space, the GUI may provide a set of tools for interactively defining portions of a multidimensional space (e.g., OD space, feature space, RGB space, etc.) that map to corresponding rules for defining signal components. These tools may be configured to define regions of a multidimensional space corresponding to (e.g.) a wedge, facet, exterior, cylinder, curve, or ellipse. Alternatively or additionally, the tools may be configured to receive free-form input that identifies part or all of the boundary of the region. Some examples include a wedge tool being configured to receive input that identifies a center point and angle, an exterior tool being configured to receive input that selects one or more points along the boundary of the region to be defined, and a brush tool being configured to directly receive input corresponding to "painting" onto a color diagram in order to define one or more regions in a multidimensional space. In addition, the tools may also be provided to incorporate thresholding techniques that allow the user to specify thresholds for one or more axes (e.g., one or more polar axes or one or more color channel axes in OD space). Once a portion is defined or selected within a color space, certain operations may be performed on each pixel representation assigned to (or not assigned to) that portion. For example, if a pixel representation lies within that portion of space, a specific algorithm may be used to convert its coordinates to a predicted intensity of a particular staining corresponding to that portion. As another example, when a pixel representation lies outside that portion of space, it may be inferred that the pixel does not contain signals from a particular staining associated with that portion (for example, unmixing may be performed based on this inference).

[0090] Figure 1 shows a workflow 100 for acquiring and processing multiplex images. The image generation system 105 may be configured to collect images of one or more stained samples. The stained samples may be stained with (for example) one or more biomarker stains and / or one or more reference stains. The collected images may include pure color images (when the sample is stained with only one stain), singleplex images 108a-m (when the sample is stained with a single biomarker stain and a reference stain), or multiplex images 110a-n (when the sample is stained with two or more biomarker stains and reference stains). The collected images may be transmitted to a computer system 115 via a communication network 120.

[0091] The computer system 115 can process the image to generate one or more outputs 135a~p. In some cases, the computer system 115 receives a multiplex image depicting a sample stained with multiple biomarker stains (two or more stains or three or more stains) and a reference stain, and the computer system 115 generates an output predicting the signal from at least one of each of the stains. For example, if a triplex image is received, the computer system 115 can generate an output containing one or more synthetic singleplex images corresponding to the biomarker stains and / or reference stains used to prepare the sample slices of the image.

[0092] Outputs 135a-p may be generated, for example, using automated techniques and / or inputs received via interface 112. For example, interface 112 may be configured to dynamically display a composite singleplex image and / or associated metrics generated based on the current color vectors assigned to multiple stains represented in the input multipleplex image. Interface 112 may also be configured to receive inputs that directly or indirectly adjust the color vectors for each of one or more of the multiple stains (for example, thereby triggering an automatic update to interface 112).

[0093] Images used by the computing system 115 may include image data and / or may be converted to image data (e.g., via the computing system 115), which may include data characterizing one or more intensities for each of one or more pixels (e.g., each intensity corresponds to a given color channel or a given frequency band). For example, biological specimens, such as tissue sections, are stained by applying staining assays that include one or more chromogenic stains (for bright-field imaging), fluorophores (for fluorescence imaging), quantum dots, or combinations thereof. In the analysis of biological specimens, such as cancerous tissue, different stains are specified to identify one or more types of biomarkers, such as immune cells.

[0094] The communication network 120 may include the Internet, intranet, wired LAN (Local Area Network), wireless LAN (WiLAN), WAN (Wide Area Network), MAN (City-Scale Network), PSTN (Public Switched Telephone Network), and other types of communication networks. The communication network 120 may further include one or more communication devices such as gateways, routers, or bridges. As just one example, the communication network 120 may have one or more servers and one or more websites accessible by users to send and receive information usable by one or more computer systems 115. The communication network 120 may be any type of network well known to those skilled in the art that can support data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk®, etc.

[0095] An exemplary computer system 115 of system 100 may include a processing system 125 having one or more high-speed central processing units (CPUs), processors, and one or more memories. Computer system 115 may also include memory for storing processing modules or logical instructions executed by one or more combined processors. The computer memory for storing data may also be maintained on computer-readable media including magnetic disks, optical disks, organic memory, and any other volatile (e.g., random access memory (RAM)) or non-volatile (e.g., read-only memory (ROM), flash memory, etc.) mass storage systems readable by the CPU. The computer-readable media may include collaborative or interconnected computer-readable media that may reside exclusively on the processing system or be distributed among multiple interconnected processing systems that may be local or remote to the processing system.

[0096] One or more databases 130 can store images collected by the image generation system 105 and / or one or more image processing results (e.g., composite singleplex images and / or composite multiplex images).

[0097] The computer system 115 may include one or more servers or client terminals that communicate with personal information terminals / data assistants (PDAs), laptop computers, mobile computers, internet appliances, one-way or two-way pagers, mobile phones, or other similar desktop, mobile, or handheld electronic devices. A client terminal may be configured to transmit and / or receive information to and from one or more client systems. For example, a client terminal may provide an interface that receives inputs that partially or completely define one or more color vectors or other components of an unmixing protocol. The interface may further or alternatively display representations of one or more received images (e.g., in optical density space) and / or one or more composite images (e.g., generated using a set of color vectors that may be at least partially generated using inputs received through the interface).

[0098] Figure 2 shows an exemplary network 200 of the digital pathology image generation system 105 of Figure 1. The image generation system 105 may include a fixation / embedding system 205 for fixing and / or embedding tissue samples treated with a fixative (e.g., a liquid fixative such as formaldehyde solution) and / or embedding materials (e.g., histological waxes such as paraffin wax and / or one or more resins such as styrene or polyethylene). Each slice may be fixed by dehydrating the slice after exposing it to the fixative for a predetermined period (e.g., at least 3 hours) (e.g., via exposure to an ethanol solution and / or a clearing intermediate). The embedding material can penetrate the slice when it is in a liquid state (e.g., when heated).

[0099] The image generation system 105 may further include a tissue slicer 210 that slices a fixed and / or embedded tissue sample (e.g., a tumor sample) to obtain a series of sections, each section having, for example, a thickness of 4-5 microns. Such sectioning may be performed by first cooling the sample and then slicing the sample in a hot water bath. The tissue may be sliced ​​using (e.g.) a vibratome or a compressstorm.

[0100] Since tissue sections and the cells within them are nearly transparent, preparation generally involves staining the tissue sections (e.g., automatically) to make the relevant structures more visible. In some cases, staining is performed manually. In other cases, staining is performed semi-automatically or automatically using the staining system 215.

[0101] Staining may involve exposing individual sections of tissue to one or more different stains (e.g., sequentially or simultaneously) to express different characteristics of the tissue. For example, each section may be exposed to a predetermined amount of stain for a predetermined period of time. The stains may include (e.g.) RNA probes, protein probes (e.g., nuclear protein probes or cytoplasmic protein probes), immunohistochemical stains, secretion probes, etc. In some cases, the staining may be for staining KAPPA mRNA or LAMBDA mRNA.

[0102] One typical type of tissue staining is histochemical staining, which uses one or more chemical dyes (e.g., acidic dyes, basic dyes) to stain tissue structures. Histochemical staining can be used to show general aspects of tissue morphology and / or cellular microanatomy (e.g., distinguishing the cell nucleus from the cytoplasm, showing lipid droplets, etc.). An example of a histochemical stain is hematoxylin and eosin (H&E). Other examples of histochemical stains include trichrome stain (e.g., Masson's trichrome), Schiff periodate (PAS), silver stain, and iron stain. The molecular weight of histochemical staining reagents (e.g., dyes) is generally about 500 kilodaltons (kD) or less, although some histochemical staining reagents (e.g., Alcian blue, phosphomolybdic acid (PMA)) can have molecular weights up to 2000 or 3000 kD. An example of a high molecular weight histochemical staining reagent is α-amylase (approximately 55 kD), which is sometimes used to indicate glycogen.

[0103] Another type of tissue staining is immunohistochemistry (IHC, also called "immunostaining"), which uses a primary antibody that specifically binds to a target antigen (biomarker) of interest. IHC can be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or fluorophore). In indirect IHC, the primary antibody first binds to the target antigen, and then a secondary antibody conjugated to a label (e.g., a chromophore or fluorophore) binds to the primary antibody. Because antibodies have a molecular weight of approximately 150 kD or more, the molecular weight of IHC reagents is much larger than that of histochemical staining reagents.

[0104] The sections may then be individually mounted on corresponding slides, after which the imaging system 225 may scan these to generate raw multiplex and / or singleplex digital pathology images (e.g., 110a-n, 108a-m). Each section may be mounted on a slide, after which the slides may be scanned to create digital images, which may then be evaluated using automated digital pathology image analysis and / or input from a human pathologist (e.g., using image viewer software). Input and / or results from the automated analysis may (e.g.) identify annotations that identify one or more segments corresponding to physiological categories (e.g., tumor area, necrosis, etc.). Additionally or alternatively, input and / or results may identify some or all of the color vectors or related variables to facilitate unmixing of the same or different slides.

[0105] Digital histopathology images (e.g., 110 or 108) typically contain an array of pixels, usually a rectangular matrix. Each “pixel” is a single pixel, a digital quantity representing some property of the image at a location in the array corresponding to a specific location in the image. If the digital pathology image is a grayscale image, the pixel values ​​of the digital image typically correspond to a specified range. For example, each array element may be one byte (e.g., 8 bits) representing a pixel value in the range of 0 to 255. In a grayscale image, “255” may represent absolute white, and 0 (“0”) may represent absolute black (or vice versa). Color images may contain multiple (e.g., three) color channels, such as red, green, and blue (RGB) channels. For a particular pixel, there is typically one value for each of these color channels (e.g., a value representing the red component, a value representing the green component, and a value representing the blue component). By varying the intensity of these three components, typically all colors of the color spectrum are created. In some cases, digital histopathology images may contain signals corresponding to one or more wavelengths outside the visible spectrum (e.g., in the ultraviolet or infrared spectrum).

[0106] Figure 3A shows an exemplary workflow 300-A for defining color vectors associated with stains from digital pathology images and performing stain unmixing using the color vectors. In block 302, one or more color vectors are defined and / or adjusted to be associated with one or more corresponding stains. One or more single-stain (or pure-color) slides (e.g., slides 305a, 305b, and 305c) are accessed.

[0107] Slide 305, stained with a pure color, may be an IHC image depicting a slide stained with a single stain without counterstaining, stained by replacing the buffer of other key biomarkers in a multiplex IHC staining protocol. The user interface can present one or more of slides 305a-c and can receive user input identifying one or more regions of interest in a given depiction of at least a portion of the slide.

[0108] As an exemplary example, Figure 3A depicts three pure-color stained slides (single yellow stain (Dabsyl) 305c, single purple (TAMRA) 305b, and single blue (hematoxylin) 305a). These pure-color images 305 can also include one or more markers superimposed at corresponding specific locations within the image. This superimposition can provide reference points by strategically placing markers at locations likely to provide pure staining or representative regions of interest within the image. These locations can be selected based on prior knowledge of the staining process, tissue characteristics, user input, or by empirical observation of image features. Color vectors can be defined based on these specific locations.

[0109] Pure-color stained slides 305 can be processed using linear techniques such as non-negative matrix factorization (NMF) 310, which can yield two non-negative matrices. In this technique, pure-color stained RGB images (e.g., 305a-c) can be transformed into optical density (OD) regions based on the Lambert-Beer law. According to this law, optical density is linearly related to stain density. Following this law, a two-dimensional (2D) matrix (D) is obtained, which can be further factorized by the NMF 310 technique to find the initial color vectors 315 associated with the marker positions. Mathematically, this can be expressed in matrix form such as D = WH. For staining applications, D is the optical density matrix, W is the non-negative basis matrix also called the "color vector" matrix, and H is the non-negative coefficient matrix also called the stain intensity matrix. For RGB stained images, the columns of the W matrix may correspond to the initial color vectors 315 for each constituent stain based on specific positions.

[0110] A color vector (e.g., of size (1×3)) derived from a pure-color stained slide can correspond to a monochromatic representation in a three-dimensional color space such as RGB. For example, in the case of Dabsyl staining, the extracted color vector may have an RGB composition of [0.248, 0.374, 0.894]. A matrix W derived from a multiplex image, such as a duplex, may be of size (3×3) with two stains and one counterstain, or of size (3×4) for a triplex W. The initial color reference matrix W obtained from NMF310 may not function adequately to unmix the stains of a given multiplex image 330. This may result in the presence of errors (e.g., white space or faded counterstain hematoxylin) or background noise after unmixing the multiplex image 330 from the initial color vector 315. It will be understood that the initial color vectors 315 arranged in columns constitute the initial reference matrix W. To mitigate errors in the initial color vectors 315, calibration of the initial color vectors may be performed. Calibration may be performed to identify adjusted color vectors that produce high-quality composite singleplex and / or composite multiplex images. For this purpose, as shown in Figure 3A, an interactive graphical user interface (GUI) 112 and / or automated techniques may be provided to facilitate the fine-tuning of one or more initially defined color vectors 315. For reference, an actual multiplex image 330 may also be provided in interface 112 for the fine-tuning of the initial color vectors. Interface 112 can receive user input to define or adjust one or more color vectors, resulting in a color matrix W being dynamically defined or adjusted. As shown in Figure 3A, interface 112 may be configured for the user to fine-tune the color vectors by adjusting the contribution of a given color channel (e.g., red, green, or blue channel), but interface 112 may also show each of the representations of one or more color vectors in optical density space.

[0111] Using the color matrix W, one or more composite singleplex images 340 are dynamically generated from the actual multiplex image 330 or different multiplex images. The composite singleplex images 340 may be displayed on the interface 112 and may be dynamically updated as the color vector 325 is adjusted. Once the fine-tuning is complete, the color vector 325 may be locked and used to perform staining unmixing 335 of the same or different multiplex images.

[0112] In some cases, staining unmixing 335 may be performed linearly, for example by using an NMF technique that leverages the updated color matrix W325 and staining coefficient matrix H of the input (same or different) multiplex images 330, thereby generating a synthetic singleplex image 340. Alternatively, it may be performed non-linearly (for example by leveraging a machine learning model described below with reference to Figures 3D and 3E).

[0113] Figure 3B illustrates exemplary linear unmixing techniques according to several embodiments of the present disclosure. Such techniques may be used to extract an initial color vector 315 from a given pure slide and / or to perform staining unmixing 335. A particular linear unmixing technique shown in Figure 3B is non-negative matrix factorization (NMF) 310. NMF 310 may be performed in the optical density (OD) region, where color / staining is expressed as absorbance values ​​rather than raw RGB pixel values. Thus, for each pixel in the image, a preprocessing may be performed to convert the RGB intensity to an OD value. In this case, the processing of the OD value is based on the physical properties of light absorption by different stains or fluorophores present in the sample, resulting in more accurate and significant results.

[0114] To convert actual / synthetic RGB images (e.g., IHC pure stained images such as 305a-c) into OD domains, it can be assumed that the stained images are absorbent and satisfy the Lambert-Beer law. According to the Lambert-Beer law, the intensity of light absorbed or transmitted through a medium is proportional to the thickness of the medium and the density of the transmitting material. Mathematically, Lambert's law states that for the intensity of light (I) after passing through the medium, It can be formulated as TIFF2026516975000002.tif8170, where I0 is the initial intensity of light before it enters the medium, α is the absorption coefficient of the medium, c is the concentration of the absorbing material or the amount of staining per unit area, and d is the thickness of the medium. In the context of a digital IHC image (e.g., 305), each color channel (e.g., red, green, and blue) has the respective values ​​for the absorption coefficient of the sample, the concentration of staining in the sample, and the thickness of the sample. It has TIFF2026516975000003.tif7170. Therefore, Lambert's Law can be applied separately to each color channel, describing how each color component is attenuated differently as light passes through the medium, resulting in the final color appearance of a multiplexed image.

[0115] Lambert's law states an exponential (or nonlinear) relationship between the intensity (I) of light passing through a medium and the product of c, α, and d. Due to this nonlinear relationship, the intensity values ​​of RGB (digital) images cannot be directly used to unmix each stain. To simplify data analysis and interpretation, calculations may be performed in the optical density region, which utilizes a linear relationship and dynamic range compression when the intensity range is large. Optical density (OD), often represented by D, is a measure of how much a material attenuates light. This can be formulated as TIFF2026516975000004.tif10170, which shows a direct relationship between optical density and variables α, c, and d. A higher OD value may suggest a greater amount of staining in the sample. For each color channel, the OD vector is: It can be formed to be TIFF2026516975000005.tif6170.

[0116] NMF310 operates under the assumption that the observed color / staining in an image is a linear combination of the individual component colors / stainings. This assumption allows for the separation of mixed stains using linear transformations. NMF utilizes an iterative optimization algorithm to factorize the observed data matrix into non-negative matrices representing the spectral signatures (basis matrix) and abundance maps (coefficient matrix) of the components.

[0117] Using NMF can be advantageous in that it uses an intuitive non-negative constraint that aligns well with the physical constraints of staining intensity in pathological images (for example, compared to using other linear techniques). Furthermore, given that NMF uses basis vectors that represent pure staining, the results are interpretable. NMF is also configured to flexibly accept constraints or prior knowledge and to be robust to variations in noise and contamination.

[0118] In NMF310, the acquired data matrix is ​​in the optical density region. TIFF2026516975000006.tif6170 Here, d is the dimension of each data point (for example, in the case of an RGB image, this value is 3), and m is the number of data points, which is assumed to be a non-negative matrix. In other words, for each pixel, there is an RGB configuration in OD space. This matrix is, It can be broken down into TIFF2026516975000007.tif5170, where, TIFF2026516975000008.tif6170 is the desired rank of matrix D311, which represents the number of stains. The non-negativity constraint is also imposed on both matrices, namely W(312) and H(313). Mathematically, This can be solved by the optimization problem described below in TIFF2026516975000009.tif4170.

number

[0119] To achieve convergence to the optimal solution, the color vector matrix 312 and the coefficient matrix 313 may be initialized. Initialization may be performed by various techniques such as random initialization, singular value decomposition (SVD), sparse initialization, k-means, or induced initialization. These techniques may be used individually or in combination, and the choice of initialization technique may depend on the specific characteristics of the data and the desired characteristics of the factorization. In NMF310, the objective function is iteratively optimized using a multiplicative update rule. The updates of the basis matrix and coefficient matrix are performed, respectively: It can be formulated as TIFF2026516975000012.tif13170. To avoid the scale dispersion problem and non-unique solutions, NMF310 can be extended to sparse NMF by adding regularization and sparse terms.

[0120] For staining unmixing 335, a composite singleplex OD image can be reconstructed from the color vector matrix W 312 and the staining intensity matrix H 313. For the reconstruction of the i-th stain, TIFF2026516975000013.tif7170 can be multiplied to produce a composite singleplex OD image (e.g., 314a). A color vector matrix W 312 (e.g., defined based on user input) may be used. These singleplex OD images (e.g., 314a and 314b) can be converted to the RGB domain as needed. To convert the OD images to the RGB domain, the composite / actual OD image (e.g., 314a or 314b), associated with a single stain of singleplex or multiple stains, is converted to its respective composite / actual singleplex RGB image by applying Lambert's law, which indexes the OD values ​​and performs scaling. The mathematical formula for the conversion is: It may be written as TIFF2026516975000014.tif6170. The conversion is applied to each pixel, and the corresponding intensity values ​​of the composite / actual RGB singleplex or multiplex image can be obtained.

[0121] Figure 3C shows interface components that facilitate fine-tuning of color vectors. The RGB model may not be very useful in the fine-tuning process because the information of interest, such as the color of a dye (determined by its absorption properties), is mixed with variations in the amount of dye. One technique that can be used to extract chromaticity (color) information from RGB data is to use the Hue-Saturation-Intensity (HSI) model. The RGB-HSI conversion separates intensity information from color information. In the HSI model, the hue of a color is its angle measured on a color wheel ranging from 0 to 360 degrees. For example, the hue of pure red is 0°, the hue of pure green is 120°, and the hue of pure blue is 240°. For convenience, intermediate colors such as white, gray, and black are set to 0°. The HSI definition of saturation is a measure of the purity / grayness of a color, which can be estimated by the ratio of the difference between the maximum RGB value and the minimum RGB value to the maximum RGB value. In the HSI color model, saturation can be thought of as the distance from the center of the color wheel. Purer colors have higher saturation values ​​that are further from the center, while grayer colors have saturation values ​​that are closer to the center.

[0122] Intensity is the average of equivalent RGB values, i.e. OD is the overall lightness or brightness of a color, numerically defined as TIFF2026516975000015.tif5170. However, much of the perceived intensity variation in transmission optical microscopy can be caused by variations in staining density. Therefore, the Hue-Saturation-Density (HSD) conversion was defined as an RGB-HSI conversion applied to optical density values ​​rather than the intensity of individual RGB channels. For a single pixel, the measure of OD is It can be defined as TIFF2026516975000016.tif10170.

[0123] RGB-HSD conversion is, It can be defined as TIFF2026516975000017.tif8170. It will be understood that the color coordinates of the HSD model are not equal to the color coordinates of the HSI model because the OD is separated. For the HSD model, the resulting cx-cy plane is such that a single point is It has the characteristic of corresponding to RGB points with the same ratio between TIFF2026516975000018.tif7170. Therefore, all information regarding the absorption curve is represented within a single plane. Similar to the HSI model, hue and saturation values ​​can be calculated from the chromaticity triangle. This is because the staining mixture exhibits a linear pattern in the cx-cy plane of the HSD model.

[0124] In the chromaticity plane (cx-cy), the RGB cube can be represented by an equilateral triangle 321d that limits the spread of the cx-cy coordinates. The cx-cy plane is a 2D coordinate system represented by the equilateral triangle 321d, where the centers of each side represent red 321a, green 321b, and blue 321c. In this plane, each color vector can be represented as a point in the cx-cy plane, and the location of the point can correspond to the relative proportions of the primary colors (i.e., red, green, and blue) in the color vector. For example, if a color vector has a higher intensity in the green channel, the corresponding point will be closer to the green center point 321b. The staining properties of a dye can be modified by adjusting the location of the color vectors in the chromaticity plane 321 via GUI 112. This can also be modified by adjusting the proportions of R, G, and B from the slider bar 322.

[0125] In one example, GUI323 may be configured to blend two or more staining colors synthetically and interactively in different ratios to obtain a targeted stain. The resulting synthetic pixel may be displayed in the chromaticity plane cx-cy 321 via GUI323. Such a synthetic color pixel may be generated by selecting which color primarys to blend via user interaction from a given list of color primarys. The amount of staining for each color primary (e.g., relative to other color primarys) may then be set. The stained colors may also be blended by adding the result of multiplying the amount of staining / color primary by the corresponding color vector in OD space, which may then be converted back to RGB space for display purposes. Since the chromaticity planes cx and cy represent only hue and saturation, to determine the conversion back to RGB... You may need TIFF2026516975000019.tif5170. This is because you first need it. It is obtained as TIFF2026516975000020.tif6170, and then This can be done by calculating TIFF2026516975000021.tif6170.

[0126] As an exemplary embodiment, a user can generate a composite pixel 323b in the cx-cy plane by first selecting a set of chromophores (e.g., 323c) and then manipulating slider 324 to set the relative amount of staining for each chromophor. In Figure 3C, multiple sliders 324 are shown, and by setting "Teal" to "0.8", "Tamra" to "0.4", and the remaining chromophores, e.g., Dabsyl and hematoxylin (HTX), to "0", a blue composite pixel 1 323b (marked with "*" in GUI323) is generated. Similarly, another pixel 2 323d can be generated for another set of chromophores 323f by setting "Green" to "0.8" and "Tamra" to "0.4" from slider 324. Composite pixel 2 is marked with "X" in GUI323. Pure hematoxylin staining 323a can also be provided as a reference in Figure 3C for fine-tuning of the composite pixels. It can be observed that composite pixel 2 is visually closer to pure hematoxylin staining 323a than composite pixel 1. The location of the blended stain colors in the cx-cy plot indicates how close the two composite pixels are in hue and saturation. Increasing the amount of staining while maintaining the relative ratio of the chromogens may not change the pixel locations in the cx-cy plot, but the appearance of the composite pixels as displayed in the interface may change, which is consistent with the design of the cx-cy space, which counts only hue and saturation while keeping the density the same.

[0127] Figure 3D shows an exemplary architecture 300-D for generating one or more synthetic singleplex and / or synthetic multiplex images by leveraging multiple machine learning models. To generate such synthetic images, architecture 300-D includes a staining unmixing module 335, a color vector 325, and a remixing module 345. To generate the synthetic image, the synthetic chromogen may be controlled by adjusting the color vector, and cell / tissue-level biomarker staining patterns may be generated to mimic real-world images using a trained machine learning model. The machine learning model may include a generative model (e.g., a generative adversarial network (GAN), a diffusion model, or an autoencoder) trained to generate singleplex images for a particular stain. In an exemplary embodiment, the staining unmixing module 335 may include a conditional GAN ​​(cGAN) for generating a synthetic singleplex image conditioned on an input color vector. For example, separate cGANs (e.g., 338a, 338b, and 338c) can be trained to generate individual singleplex images (e.g., 340a, 340b, and 340c) corresponding to specific targeted stains (or composite pixels), as shown in Figure 3D. The number of models may depend on the number of constituent stains in the multiplex image.

[0128] In one embodiment, the architecture 300-D can be utilized to generate a composite image from a composite pixel or associated tuned color vector obtained using the techniques disclosed above. For example, to generate a composite singleplex image, a cGAN model (e.g., 338a) can acquire a counterstain image, such as hematoxylin, as an input image 332 conditioned on the color vector (e.g., 325a) of a target composite pixel (stain). The generated singleplex image 340 can be used to verify the accuracy of the composite pixel against the target stain. In another example, the composite singleplex image corresponding to the target composite pixel can be used to generate a composite multiplex image. The generated composite multiplex image 350 can be displayed simultaneously with the actual input multiplex image used to generate the composite singleplex images 340a-c. The user can then evaluate the degree to which the actual multiplex image and the composite multiplex image look identical (e.g., compared to cases where some or all of the signals from the actual multiplex image are not present in the composite multiplex image). This facilitates quality control and / or additional fine-tuning of one or more color vectors. Furthermore, once the cGAN models 338a-c are approved, they can be used to generate multiplexed images, thereby creating additional training data for machine learning models in a faster and more cost-effective manner than performing actual staining experiments in the laboratory. This may also allow pathologists to control the appropriate combination of different staining conditions, intensities, and biomarkers in synthetic multiplexed images. These synthetic multiplexed images can be tailored to specific needs and applications.

[0129] In another example, each cGAN may receive an actual multiplexed image (e.g., 330) as input image 332 and a color vector (e.g., 325a, 325b, or 325c) obtained from module 302. In this setup, architecture 300-D may be used to filter the actual multiplexed image according to the adjustment of the color vector. Such a generative model can be trained to filter a given multiplexed image, thereby producing an output containing a predictive signal for a particular stain associated with the model (such conditions are defined for the cGAN based on the color vector of a particular stain). These synthetic singleplexed images may be further combined by the stain remixing module 345 to produce a synthetic multiplexed image 350. The synthetic multiplexed image 350 may be compared (e.g., computationally, automatically, and / or via user review) with the actual multiplexed image provided as input 332. This comparison may be used during training and / or as an indicator of the reliability of the quality of the generated synthetic singleplexed images 338a-338c. Image quality display may be incorporated into GUI112 as feedback that can inform the user of their decision regarding whether to further adjust one or more color vectors. It should be understood that the number of generative models shown in Figure 3D is for illustrative purposes only. Depending on the nature of the multiplexed image, aspects of this disclosure are intended to include or otherwise cover any number of generative models.

[0130] Figure 3E shows an example of another method for generating one or more synthetic singleplex images 340 using a single model (e.g., a single cGAN). The single model 339 may be configured to receive as input an actual multiplex / counter-stained image 332 and identification information of a specific biomarker that characterizes the requested synthetic singleplex image (e.g., by receiving a corresponding color vector, e.g., 325a). Furthermore, as described above, the synthetic singleplex images 340 may be combined to generate a synthetic multiplex image 350. Comparison of the synthetic multiplex image 350 with the actual multiplex image may be used during training and / or as an indicator of the reliability of the quality of the synthetic singleplex image.

[0131] The staining and remixing module 345 can combine synthetic singleplex images (e.g., 340) to generate a synthetic multiplex image 350. The synthetic multiplex image 350 can be generated linearly in the optical density (OD) domain, which involves merging the intensity values ​​of each pixel from the individual singleplex images 340a-c to generate a composite multiplex image. This process can be achieved by various mathematical operations such as addition, subtraction, multiplication, or weighted averaging, depending on the desired result. Mathematically, this is: It can be formulated as TIFF2026516975000022.tif7170, where, TIFF2026516975000023.tif6170OD represents a singleplex matrix (e.g., 314a, 314b), These are the weight coefficients assigned to each singleplex image in TIFF2026516975000024.tif6170. These weights can control the contribution of each stain to the final multiplex image 350.

[0132] Alternatively, for staining remixing 345, a generative model such as a GAN or autoencoder may be trained to learn the complex mapping between the synthetic singleplex image and its corresponding multiplex counterpart. By training the generative model on a dataset containing input-output pairs (e.g., singleplex image and multiplex image), the model can capture the complex relationship between staining and cellular structure. This process may include learning to fuse features extracted from individual singleplex images 340 to create a coherent and visually realistic multiplex image. The adversarial loss for training the generator G and discriminator D to transform the synthetic singleplex images 340a-c into a synthetic multiplex image 350 can be formulated as follows:

number

[0133] Figure 4 shows a flowchart of an exemplary process 400 for determining one or more color vectors for stain unmixing according to some embodiments of the present disclosure. Process 400 relates to stain unmixing of digital pathology images by finding an initial color vector 315 and adjusting the color vector using a graphical user interface (GUI) 112. Process 400 begins in block 405, where color vectors associated with digital pathology stains or colored chromogens (e.g., at least 3) are determined. The color vectors may be default color vectors associated with dyes (which may be identified, for example, using lookup tables and / or predefined variables). For example, the color vector for the "green" dye may be defined as [0,1,0] in the RGB space.

[0134] Alternatively, the initial color vectors determined in block 405 may be determined using initial processing of one or more images received from the user device (or other devices associated with the user device). For example, non-negative matrix factorization (NMF) may be performed to transform a given OD matrix into two non-negative matrices, e.g., W color vector matrix and H abundance or coefficient matrix. The determined color vectors 315 in W can accurately represent the true spectral characteristics of the staining components, but alternatively, such characteristics may not be captured due to (e.g.) noise, artifacts, or limitations of the imaging system. Therefore, the interface can provide dynamic data to facilitate the fine-tuning of one or more color vectors.

[0135] In block 410, the interface is made available to the user device. For example, a server (e.g., a web server) may send a communication to the user device, which includes code containing instructions for generating and displaying the interface on the user device. Alternatively, local code may be executed to generate and display the interface.

[0136] The interface may include each of the determined color vectors, an actual multiplex digital pathology image, at least one composite singleplex image, and one or more color vector adjustment tools. Each of the at least one composite singleplex image may be generated using the actual multiplex digital pathology image and the color vectors determined in block 405. Each of the at least one composite singleplex image may be generated by processing the actual multiplex image using the techniques herein, such as linear unmixing techniques (NMF) or nonlinear unmixing techniques (e.g., machine learning models). One or more color vector adjustment tools may be configured to receive inputs that adjust the contributions or weights associated with each of one or more contribution axes for a given color vector. For example, a color vector adjustment tool may include sliders or numerical inputs that define weights to be assigned to a given color channel (e.g., a red, green, or blue channel), a polar coordinate channel (e.g., in optical density space), or a channel in another space.

[0137] In block 415, an input corresponding to a specific adjustment of the color vector represented in the interface is detected. The input may include interaction with at least one of one or more color vector adjustment tools. The input may include (e.g.) positioning a slider and / or inputting a number indicating the absolute or relative contribution of a channel (e.g., a color channel) to a given stain representation. For example, the input may include a number or slider position indicating that a given dye should contain 5% of the red channel instead of 0% of the red channel of the “green” dye (the percentage may be absolute or relative to the cumulative percentage of the entire channel). As another example, the input may identify a position in optical density space that should be used as the definition of the color vector of a given stain.

[0138] In block 420, interface 112 may be automatically updated in response to input detection. The automatic update may update the displayed representation of the color vector representing a particular stain. Additionally or alternatively, the update may update one or more composite images (e.g., one or more composite singleplex images and / or composite multiplex images) using the adjusted color vector. One or more metrics (e.g., characterizing absolute or relative statistics for the singleplex or multiplex images) may also be updated.

[0139] Blocks 415 and 420 may be repeated multiple times (for example, until no more input is received within a threshold time, until the session ends, until the user indicates that the color vector is determined / defined, until the automatic quality control conditions are met, etc.).

[0140] Figure 5 shows an exemplary architecture of a system 500 that determines an adjustment of an initial color vector based on a given multiplex digital pathology image, according to some embodiments of the present disclosure. The initial color vector 315 may be determined using an initialization technique (e.g., by leveraging an NMF technique 310), and a specific adjustment of the initial color vector can be found based on the actual multiplex image. In this setup, the actual multiplex image 502 may be stained using one or more stains from the initial color vector 315, but at least one initial reference stain may not be present (in that the corresponding slice was not stained with at least one initial reference stain).

[0141] At least one stain not represented in the actual multiplex image 502 is referred to in the ongoing discussion as an "excluded stain." The initial color vector 315 can be fed into a filter 504 configured to produce a filtered output from the actual multiplex image 502 based on the excluded stain. Similar to the process in Figure 3D, filtering can be facilitated by utilizing one or more generative models (e.g., autoencoders (AEs), image-to-image transformation networks, generative adversarial networks (GANs)) that can be trained to learn mappings from a given multiplex image to its constituent singleplex image conditioned on its constituent color vectors. This technique is motivated by the possibility of a well-trained machine learning model to produce a null (zero) image when conditioned on a color vector that does not exist in the input multiplex image 502. This behavior is expected because the model is trained to understand the relationship between the color vector and the corresponding stain present in the input image. If the model encounters a color vector that does not exist in the input multiplexed image, it may be unable to generate any meaningful output related to that staining, potentially resulting in a zero or null image for the excluded staining associated with that color vector.

[0142] Metrics can be calculated to evaluate the quality and characteristics of the filtered output generated by the machine learning model (filter 504). For example, if it is predicted (or known) that there is no biomarker corresponding to a given stain (or color vector) in the actual multiplex image 502, then the mean, median, mode, variance, standard deviation, and / or range in the synthetic singleplex image corresponding to the given stain can be predicted to be ideally very low (or zero). Thus, the metric can be generated such that the score is negatively dependent on the statistics in the synthetic singleplex image (e.g., mean, median, mode, variance, standard deviation, and / or range) that characterize the presence of the signal for the corresponding stain in the actual multiplex image.

[0143] For such metrics, a pixel cumulative statistic (e.g., mean or average) can be calculated using the pixel intensity values ​​of the composite singleplex image in OD space (e.g., matrices 314a and 314b in Figure 3B) and then dividing by the total number of pixels. Referring to Figure 3B, for example, if staining 1 is not present in matrix 314a, the corresponding row of the H matrix will be approximately 0, thus producing a null image in OD space. Subsequently, the mean of such a composite OD space matrix will be lower. Alternatively, the median may be selected by sorting all pixel intensities of a given OD matrix associated with the excluded stain (314a in this example) in ascending / descending order and identifying the midpoint.

[0144] Once the metric is determined, a spatial traverse technique 508 may be employed to find color vector adjustments 510 associated with the excluded stains in order to quantify the filtered output based on the extent to which the stains are present in the actual multiplex image 502. The spatial traverse technique 508 can systematically explore the space of possible adjustments to the color vectors representing the excluded stains. Examples of such techniques may include, but are not limited to, gradient descent, Monte Carlo methods, genetic algorithms, or other stochastic optimization techniques such as annealing, which iteratively adjusts the color vectors to optimize a particular criterion, such as minimizing a calculated metric. Gradient descent is an optimization algorithm commonly used to minimize a function by iteratively moving in the direction of the steepest descent of the function. In this example, the objective that may be targeted for minimization may be a metric calculated based on a synthetic singleplex image. The algorithm can start with an initial color vector representing the excluded stains and calculate the gradient of the metric with respect to the color vector. This gradient indicates the direction of the steepest ascent of the metric. The color vector may be scaled with small step sizes (learning rate) and adjusted in the opposite direction of the gradient to minimize the metric. This adjustment is repeated iteratively until convergence or stopping criteria are met. The objective may be defined as finding the optimal color vector that minimizes the metric and yields a synthetic singleplex image that accurately represents the excluded biomarkers.

[0145] The Monte Carlo method is a probabilistic simulation technique that uses random sampling to estimate numerical results. In this context, Monte Carlo simulation can be used to explore the space of possible adjustments to color vectors representing excluded stains. Following this technique, random adjustments are made to color vectors representing excluded stains within a specified range or distribution. A metric is calculated for each randomly adjusted color vector. Depending on the metric value and the optimization objective (minimize or maximize), adjustments may be probabilistically accepted or rejected, leading to a search toward a better solution. This process may be repeated over several iterations, enabling a comprehensive exploration of the adjustment space. By iteratively sampling and evaluating adjustments, the Monte Carlo method can efficiently explore the adjustment space and identify promising regions or solutions that can be incorporated via interface 112.

[0146] When the color vector associated with the excluded stains (e.g., 510) is adjusted by minimizing a metric, a new multiplex image 512 may be generated and / or utilized. This multiplex image 512 may be stained with stains associated with the initial color vector 315 (including one or more excluded stains whose color adjustments are calculated based on spatial shift techniques and metrics). By leveraging the aforementioned stain unmixing process 335, one or more new synthetic singleplex images 514 associated with the excluded stains may be generated.

[0147] Figure 6 shows an exemplary flowchart of process 600 for generating a composite singleplex image using finely tuned color vectors. In block 605, color vectors are determined for each of at least three digital pathology stains. The color vectors may be determined using the techniques described in relation to block 405 of process 400 (or other techniques disclosed herein).

[0148] In block 610, a digital pathology image may be accessed, and the image depicts a sample stained using one or more stains associated with the initial color vector 315 but without at least one of these stains. Each stain that is not present in the sample but is one of the at least three stains for which the color vector has been determined is referred to herein as an “excluded stain.” The digital pathology may be a multiplex (e.g., duplex) image or a singleplex image.

[0149] In block 615, the initial color vector 315 can be supplied to a filter 504 configured to produce an output filtered from the digital pathology image based on excluded stains. Filtering may be performed using a linear technique (e.g., NMF) or a nonlinear technique (e.g., a machine learning model).

[0150] In block 620, a metric is generated that characterizes the signal characteristics in the filtered output. Since the sample depicted in the digital pathology image is known not to be stained by the second stain, the optimal filtered output will contain no signal and will be blank. The metric can include any metric that indicates whether or not a signal is present. For example, the metric may include statistics on intensity values ​​such as mean, median, mode, variance, standard deviation and / or range.

[0151] In block 625, the metric is used to generate a modified color vector for the second staining. In some cases, the modified color vector is generated automatically using the modified color vector. For example, a spatial traverse technique (e.g., gradient descent, Monte Carlo method) may be used, and the filtered output and metric are dynamically updated as the space is traversed. As another example, the interface and backend system may be configured so that the filtered output and metric are dynamically updated when the interface user modifies the definition of the color vector for the second staining.

[0152] In block 630, a new image is received showing the sample stained with the second stain. The sample may also be stained with one or more other biomarkers and / or reference stains (e.g., one or more of the three other stains), or it may not be stained at all.

[0153] In block 635, a composite singleplex image is generated using the adjusted color vectors and the new image. For example, the new image may be processed using a linear or nonlinear technique to generate the composite image. The linear or nonlinear or nonlinear technique (e.g., and its associated parameters) may be the same as the one used to generate the filtered output in block 620.

[0154] In block 640, a composite singleplex image is output. For example, the composite singleplex image may be sent to and / or displayed on a user device. In some cases, it will be understood that multiple composite singleplex images are generated and output in blocks 635 and 640, and each composite singleplex image is generated using a different color vector. In some cases, the other color vector is modified after the metric is generated. For example, in block 625, the interface may be configured to dynamically generate and present the metric (e.g., and the composite singleplex image) in response to changing the color vector representing a second stain and / or changing one or more other color vectors representing one or more other stains of at least three stains. In some cases, the other color vector is determined initially in block 605.

[0155] Figure 7 shows an exemplary flowchart of process 700 for identifying the recommended color vector. In block 705, a color vector is determined for each of at least one stain. The color vector may be determined using the technique described in relation to block 405 of process 400 (or another technique disclosed herein).

[0156] In block 710, actual multiplex images stained with at least one stain associated with an initial color vector are accessed. For example, a duplex image stained with two biomarker stains along with a counterstain (e.g., hematoxylin) may be accessed. As another example, a singleplex image stained with one biomarker stain and a counterstain may be accessed. The objective may be to identify potential additional stains that are effectively distinguishable among existing stains so that a triplex image using (e.g.) two existing biomarker stains and a potential additional stain is reliably and accurately unmixed into three composite singleplex images.

[0157] In block 715, initial color vectors for additional potential stains may be identified. Such identification may be performed automatically or based on user input. For example, based on the color vector determined in block 705, the position of each of at least one digital pathology stain in optical density space may be determined. An automated technique may identify another position in optical density space using an objective function that prioritizes maximizing the distance in space (or maximizing the minimum distance) to a position associated with at least one digital pathology stain. As another example, the interface may display the position and / or vector of at least one digital pathology stain and receive user input defining another position and / or vector to be associated with the initial color vector.

[0158] In block 720, the filtered output is generated by filtering the actual multiplexed image using the initial color vector. The filtering may include linear or nonlinear filtering. For example, filtering may use NMF or a machine learning model.

[0159] In block 725, a metric is generated to characterize the signal characteristics in the filtered output. Since the illustrated sample was not stained with additional staining, the objective function can be defined such that the filtered output lacks signal and / or information. This may indicate that the signal that would be detected through additional staining is independent of at least one stain.

[0160] Signal characteristics can (for example) characterize the quantity, variation, or complexity of the signal. Signal characteristics may include (for example) the mean, median, mode, variance, standard deviation, and / or intensity range, and a spatial contrast metric. Additionally or alternatively, signal characteristics may characterize the degree to which the filtered output corresponding to an initial color vector differs from another filtered output corresponding to another color vector (for example, one of at least one vector).

[0161] In block 730, the metric is used to identify a recommended color vector. The color vector may be the same as the initial vector, or it may be a different vector. In some cases, the metric is used to determine whether or not to adjust the recommended color vector. For example, an automated algorithm may use the metric to iteratively evaluate the metric and adjust the color vector of an additional potential dye until a predefined condition is met (e.g., the target metric is achieved, the iterative improvement of the metric falls below an improvement threshold, a predefined number of iterations occurs, etc.). As another example, the metric and color vector of an additional potential dye may be displayed within the interface and dynamically updated, and user input may be received which may be adjusted to ultimately accept a given color vector for the additional potential dye.

[0162] A recommended color vector may be output (e.g., upon decision, upon acceptance, during iteration, etc.). The recommended color vector may be used to notify or select additional potential staining configurations.

[0163] Figure 8 shows an exemplary flowchart of process 800, which determines a performance prediction score representing the predicted degree to which at least three digital pathology stains are actually sufficiently separable, in order to reliably support the generation of a synthetic singleplex image.

[0164] In block 805, a color vector 315 is determined for each of at least one staining. The color vector may be determined using the technique described in relation to block 405 of process 400 (or another technique disclosed herein).

[0165] In block 810, an actual digital pathology image is accessed that depicts a sample stained using one or more stains (referred to as "excluded stains") that are associated with the initial color vector 315 but do not include at least one of these stains. The digital pathology image may be (for example) a duplex or singleplex image.

[0166] In block 815, the initial color vector 315 is supplied to a filter 504 configured to produce an output filtered from the actual digital pathology image 502 based on excluded stains. One or more machine learning models (e.g., one or more generative models) may be trained to learn a mapping from a given multiplex image to its constituent singleplex image for filtering purposes. As an example, a conditional GAN ​​may be used as a filter such that if the model is conditioned on a color vector that does not exist in the input multiplex image, it cannot produce any meaningful output related to that stain, resulting in a result of 0 or a null image. Conversely, if such a model is given a color vector that exists in a given multiplex image, it can produce a constituent composite singleplex image associated with that color vector.

[0167] In block 820, performance prediction scores are generated for the filtered output and / or for other composite singleplex images that make up the actual image. Finally, in block 825, the performance prediction scores are output (e.g., sent to and / or displayed on the user device). If it is predicted (or known) that there is a biomarker corresponding to a given stain in a given depicted sample or multiplex image, then it can be predicted that the performance prediction scores, e.g., mean, median, mode, variance, standard deviation, and / or range, may be relatively higher when accurate color vectors are used compared to when less accurate color vectors are used. If it is predicted (or known) that there is no biomarker corresponding to a given stain in a given depicted sample, then it can be predicted that the mean, median, mode, variance, standard deviation, and / or range, when accurate color vectors are used, may be relatively lower compared to when less accurate color vectors are used. Therefore, when the presence of a biomarker for the corresponding stain in the depicting sample is known or predicted, the performance prediction score may be generated such that the score positively depends on the mean, median, mode, variance, standard deviation, range, and / or the degree to which the stain can be effectively distinguished in the synthetic singleplex image.

[0168] In one example, the performance prediction score may be estimated by grouping similar stains together based on staining features. For example, staining features may include optical density values, color histograms, or any other features that can effectively capture the staining pattern. These features can be clustered using clustering techniques, e.g., k-means, hierarchical clustering, or spatial clustering of applications with density-based noise (DBSCAN). For example, k-means clustering may be used if the number of clusters is known in advance. Such a clustering algorithm divides the feature space into clusters, each cluster representing a group of stained regions having similar staining patterns. The clustering process aims to minimize distances within clusters (distances between points within the same cluster) and maximize distances between clusters (distances between points in different clusters). Finally, a performance prediction score that assesses the quality of the clusters may be estimated by metrics such as silhouette score, Davies-Bouldin index, distance (e.g., Euclidean distance, Mahalanobis, or Manhattan), or visual inspection.

[0169] In another case, performance prediction scores can be calculated for a synthetic singleplex image by estimating the correlation between each staining pattern observed in the multiplex image. For this purpose, a correlation coefficient (ρ) providing a standardized quantitative representation of staining intensity can be calculated for OD singleplex images derived from RGB by measuring the absorbance by the stained tissue. This method takes into account variations in staining protocols, image acquisition settings, and tissue characteristics, enabling a consistent basis for comparison. Additionally, OD values ​​are inherently in the non-negative to positive range, which aligns well with the physical constraints of staining intensity. The correlation coefficient between two singleplex OD images A and B is calculated using the Pearson correlation. It can be calculated by obtaining it as TIFF2026516975000028.tif17170, where, TIFF2026516975000029.tif7170 These are the averages of each. The absolute value of the correlation coefficient is in the range of 0 to 1, where 1 indicates a perfectly linear relationship, and values ​​closer to 0 indicate that the stains are well separable. This score represents the degree to which the stains of a synthetic singleplex image are separable and can be used as a measure of the suitability of the synthetic image for various applications such as image analysis, pathology, and medical diagnosis.

[0170] Multiplex digital pathology images can represent the complexity involved in visually examining multiple staining intensities co-localized within a cell. Unmixing multiplex images becomes even more difficult when multiple biomarkers, e.g., three or more biomarkers, are co-localized. For example, an input real / synthetic triplex image may contain multiple distinct stains configured to be absorbed by the progesterone receptor (PR), human epidermal growth factor receptor (HER), and estrogen receptor (ER). Additionally, real and / or synthetic multiplex images may contain signals from counterstained biomarkers configured to stain the nucleus and / or hematoxylin. Regarding staining, PR may be stained with carboxytetramethylrhodamine (TAMRA), HER2 may be stained with Green, and ER may be stained blue with benzenesulfonyl (Dabsyl) and a counterstained IHC marker, which is nuclear staining by hematoxylin.

[0171] Estrogen is a hormone that can be a contributing factor, particularly in breast cancer and endometrial cancer. Estrogen binds to estrogen receptors (ERs), triggering a series of cellular responses involving the proliferation and differentiation of certain cells. Estrogen receptors (ERs) and progesterone receptors (PRs) are biomarkers used in cancer pathology to assess the presence of estrogen and progesterone receptors in tumor cells. ERs and PRs are nuclear receptors primarily located within the nucleus of cancer cells. Staining patterns for ERs and PRs can help identify the intracellular localization of these biomarkers. For ERs, the antibody commonly used is ER-α. Staining is usually visualized with a chromogen, such as DAB. Progesterone staining may involve the use of PR antibodies, and the resulting stain can also be visualized using DAB.

[0172] For unmixing purposes, in one aspect of the present disclosure, constraints may be introduced to simplify the staining analysis, thus reducing the complexity associated with staining unmixing. This technique can facilitate, for example, higher accuracy, precision, and / or reliability in generating a synthetic singleplex image from a given multiplex image depicting a sample stained with three or more dyes / stains. In the disclosed technique, each pixel of the multiplex image may be mapped to a position in a multidimensional color map. Pixels within a particular part of the color map (e.g., quadrants, parts defined by y-values ​​greater than / less than and x-values ​​greater than / less than, wedge shapes, etc.) may be assigned a pixel-specific color vector that predicts the expression level of a first biomarker corresponding to that part (e.g., based on the grayscale optical density of the particular part) and a "0" (or other predefined expression level) for each other biomarker corresponding to the multiplex image. For pixels outside this particular part, the unmixing technique may predict the expression levels of other biomarkers that maintain a predefined expression level, e.g., a "0" for the first biomarker or a other predefined number. In some cases, a particular part may be defined by inequalities relating to the x and y coordinates, such as x > 25 and y < -15.

[0173] To extract specific portions from a color space, a GUI may be provided that interactively offers a set of tools for defining portions of a multiplexed image mapped to a multidimensional color space. These tools may include, but are not limited to, wedge, facet, exterior, cylindrical, curve, ellipse, brush tools, or freeform selection. For example, wedge may enable wedge-shaped portions by selecting a center point and angle, exterior may enable the selection of points along the boundary of a target area, and brush tools may enable painting directly onto a colored diagram to define portions by adjusting the size and shape of the brush to select areas of interest with varying levels of granularity. The tools may also be provided to incorporate thresholding techniques, allowing the user to specify threshold values ​​for x and y values ​​to define the portions. Furthermore, freeform tools can provide flexibility for demarcating portions where predefined shapes may not adequately capture the target area. Once a portion is defined or selected in the color space, the GUI may be configured to perform actions such as assigning specific values ​​to the remainder of the portion. The GUI may be configured to provide a corresponding matrix for applying an unmixing technique (such as one disclosed) to the remainder if the extracted portion is assigned "0".

[0174] In some embodiments, multidimensional color spaces include the International Commission on Illumination (CIE) color space (also known as the CIE XYZ color space). This color space is a standardized system for representing colors based on human perception. It defines three primary colors, X, Y, and Z, where Z represents luminance (brightness) and X and Y represent chromaticity (e.g., hue and saturation). For applications such as dyeing or color analysis, only XY may be used.

[0175] As an exemplary example, Figure 9A shows an example of an ER-PR-HER2 triplex image, where each pixel in the triplex image is mapped to a position in a cx-cy plot (used as a multidimensional color map). For illustrative purposes of the disclosed technique, an example of an ER-PR-HER2 triplex image 910 and the corresponding cx-cy plot 915 is shown in Figure 9A. In the distribution plot 915, pixels 915a, 915b, 915c, and 915d represent color vectors associated with Dabsyl (ER), TAMRA (PR), Green (HER2), and hematoxylin. From plot 915, it can be observed that hematoxylin, TAMRA, Dabsyl, and HER2 are distributed in the first, second, third, and fourth quadrants, respectively. Thus, by utilizing the disclosed constraint method, optical densities of different color distributions can be separated. These extractions may be performed by using different constraints such as linear, cylindrical, and wedge-shaped. For example, in plot 920, only the Green stain is extracted within the fourth quadrant. Depending on the constraint method, linear or other constraints may be used to extract the Dabsyl, TAMRA, and hematoxylin signals from the triplex image 910.

[0176] Figure 9B illustrates the staining unmixing of an exemplary triplex ER-PR-HER2 image 910 from Figure 9A according to several embodiments of the present disclosure. As shown in the cx-cy plot 925 of Figure 9B, the remaining distribution of ER-PR-HER2 includes TAMRA, Dabsyl, and hematoxylin, thereby yielding a duplex image. This plot can be achieved by using a two-facet wedge 960b from the constraint toolbox 960 to separate Green from the remaining stain. In the cx-cy plot 930, the hematoxylin signal can be extracted from the remaining distribution of ER-PR-HER2 using a facet (line) 960d connecting the color vectors associated with TAMRA pixels 915b and Green 915c. Similarly, in the cx-cy plot 935, the remaining distribution is the same as the distribution in plot 925, having a different facet 960d connecting Dabsyl to hematoxylin. The resulting distribution can be seen in plot 940, which can be achieved by applying the dye unmixing technique 335 described above. Using the constraint toolbox 960, the distribution can be divided into four quadrants by selecting the xy quadrant separation constraint 960a, as shown in plot 915.

[0177] Figure 9C shows an example of staining unmixing results for ER-PR-HER2 triplex and one or more singleplex images using the disclosed constraint technique. Figure 9C shows ER-PR-HER2 triplex image 962 stained with Dabsyl, TAMRA, Green, and counterstain hematoxylin for the cell nucleus. By utilizing the constraint technique, triplex image 962 is unmixed into singleplex images of its constituent Dabsyl(ER)964, TAMRA(PR)966, Green(HER2)968, and hematoxylin970. Similarly, Dabsyl singleplex image 974, TAMRA singleplex image 976, and Green singleplex image 978 are not unmixed from adjacent registered singleplexes in the bottom row of Figure 9C using the disclosed constraint technique. These results demonstrate that this technique can be effectively used to obtain staining unmixing for different expression levels of low / medium / high HER2(Green).

[0178] Figure 9D shows an example of staining and remixing results for ER-PR-HER2 triplex and one or more singleplex images using the disclosed constraint technique. Staining and remixing may be performed by process 345 described in Figure 3D. The top row 980 shows the remixing result for the ER-PR-HER2 triplex, and the bottom row 982 shows a ground truth triplex image and adjacent, registered real singleplex images for comparison.

[0179] Figure 9E shows the staining and remixing results of a triplex ER-PR-HER2 according to several embodiments of the present disclosure. In this example, the ER-PR-HER2 triplex image 992 is not unmixed into its constituent colorants using the disclosed constraint technique. The disclosed constraint technique allows for the extraction of individual signals by applying constraints, such as linear, wedge-shaped, and cylindrical constraints, from a provided interface. The extracted staining signals are then remixed and counterstained with hematoxylin to obtain synthetic remixed Dabsyl 994, synthetic remixed TAMRA 996, and synthetic remixed Green 998, as shown in Figure 9E.

[0180] Figure 10A shows an exemplary flowchart of process 1000-A for performing stain unmixing. The constraint technique can support the more accurate, precise, and / or reliable generation of synthetic singleplex images from multiplex images (e.g., depicting sections of a sample stained with three or more dyes or four or more dyes). For staining, an unmixing constraint may be added, which may have the effect of reducing the complexity of potential color analysis.

[0181] In block 1005, a color vector is determined for each of at least four digital pathology stains. The color vector may be determined using the technique described in relation to block 405 of process 400 (or another technique disclosed herein). The color vector may be adjusted according to the technique described above in Figure 3A. In some cases, each pixel in the digital pathology image may be mapped to a position in a multidimensional color space. Of these four stains, a particular stain may be selected in block 1010 such that in block 1015 it is due to a portion of the color space (e.g., quadrants, portions defined by being greater than / less than some y value and greater than / less than some x value, wedge-shaped, cylindrical, etc.). The particular stain may be one that is not expected to co-express with one, more than, or all of the other stains among the at least four stains. For example, a particular stain may include a stain configured to be absorbed by the cell nucleus (e.g., having a given biological property), while other stains may be configured to be absorbed by the cell membrane (e.g., having a corresponding other biological property). As another example, certain stains may be configured to be absorbed by the cell membrane (e.g., having a given biological property), while other stains may be configured to be absorbed by the cell nucleus (e.g., having a corresponding other biological property).

[0182] In block 1020, an actual multiplex image depicting a specimen (e.g., a tissue section) stained with at least three digital pathological stains is accessed. In block 1025, each pixel of the actual multiplex image may be mapped to a point in multidimensional space. In block 1030, for each pixel, a pixel-specific vector may be generated that predicts the degree of expression of each of the at least four stains in the portion of the biopsy section depicted in the pixel. Finally, in block 1035, one or more composite singleplex images may be generated using the pixel-specific color vectors.

[0183] Figure 10B further illustrates an exemplary flowchart of component 1030 of Figure 10A. In block 1030a, it is determined that each of the first subsets of the set of pixels maps to a point in a particular part of the color space. In block 1030b, for each pixel associated with a particular part, the expression level of the biomarker associated with the particular stain associated with that part is predicted based on the optical density of that pixel. For example, the part of the color space may be a quadrant or wedge associated with the green channel, and the predicted expression of the biomarker associated with the green stain assigned to each pixel in the quadrant or wedge may be defined as the optical density of the pixel. In some cases, the predicted expression levels of the biomarkers for each of the other four stains may be set to 0 or another constant.

[0184] In block 1030c, a second subset of the set of pixels is defined, and each pixel in the second subset is mapped to a position outside the portion of the color space. In block 1030d, for the pixels in the second subset, an unmixing technique (such as NMF) is performed to predict the expression level of each biomarker associated with other stains in at least four stains (excluding the stain associated with the portion). In some cases, the expression of a biomarker associated with a stain associated with a portion of the color space may be defined as 0.

[0185] In multiplex immunohistochemistry (mIHC), digital pathology images can be referred to as, for example, singleplex, duplex, or triplex, depending on the number of different markers or stains used for staining. For example, singleplex staining may use a single marker or stain on a tissue section to visualize a specific target or protein along with a counterstain. Similarly, duplex and triplex staining may apply two and three different markers, respectively, along with counterstains, to simultaneously detect a number of different antigens (target proteins) within a single tissue sample. This technique can be used to study multiple biomarkers or antigens in the same tissue section and to provide comprehensive information about the cell interactions, heterogeneity, location, function, and visualization of these antigens. Such multiplex staining involves multiple primary antibodies, each recognizing a specific target, followed by the application of corresponding secondary antibodies labeled with different chromophores or fluorophores for visualization. Furthermore, multiplex staining, such as triplex staining, saves time, uses less material, and preserves valuable samples compared to three simple stains, allowing detection to be performed on the same tissue section.

[0186] Exemplary embodiment: Exemplary embodiments of the disclosed technique are provided for staining unmixing of multiplex digital pathology images 110a-n or singleplex images 108a-m. In the following embodiments, stained slides were scanned at 20x magnification with a VENTANA DP200 scanner and annotated with 10 fields of view (FOV) per slide using HALO image analysis software. To maintain consistency in FOV placement throughout the slides, all FOVs underwent quality control (QC) by independent team members.

[0187] As mentioned above, the color vector (initial W matrix) 315 obtained from non-negative matrix factorization (NMF) 310 may not work well for staining unmixing. Figure 11A depicts a comparison of staining unmixing 335 of a duplex image 1105a using the initial color matrix 315 and the adjusted color matrix, according to an exemplary embodiment. The first row 1105 in Figure 11A represents the unmixing performance using the conventional NMF 310 method. White space or noise may be observed from the synthesized TAMRA 1105b (e.g., the faint / blurred nucleus (hematoxylin) problem seen in the synthesized TAMRA 1105b). The second row 1110 in Figure 11A represents the unmixing performance of the duplex image 1105a with the adjusted color matrix 325.

[0188] In other synthesized TAMRA 1110a images, a clearer depiction of the nuclei may be observed, which is obtained by shifting the Dabsyl vector to the left or outward from the hematoxylin vector in cx-cy space using the disclosed technique. This color vector modification enhances the nuclear hematoxylin intensity and provides better nuclear signaling (e.g., visibility of nucleoli, chromatin, etc.). The improved nuclear signaling in the synthesized images is quite comparable in signal quality to that of ground truth images 1110b and H&E images 1110c. It will be understood that ground truth singleplex multiplex images are from sequential tissue sections representing the corresponding adjacent singleplex images. In these ground truth images, the tissue morphology does not match due to the fact that the images are from adjacent slides and not the same slide. Thus, differences in tissue morphology remain.

[0189] Figure 11B depicts a comparison of staining unmixing of another duplex image 1115e and singleplex image 1115a using the initial and adjusted color matrices. While unmixing singleplex Dabsyl 1115a and duplex image 1115e with very weak TAMRA, as shown in the exemplary examples in the first row of Figure 11B, we investigated the color vector that detected the TAMRA stain of, for example, 1115d (image indicated by the red arrow). The original TAMRA color vector was adjusted until a color vector was obtained that unmixed singleplex Dabsyl image 1115a showing a very low TAMRA background, i.e., a low TAMRA signal ~ no TAMRA signal. “Very low” means an intensity close to the intensity of tissue where no cells are present. The same color vector can be used to accurately detect the TAMRA signal in an image of a sample where such a signal is present, as shown in the second row 1120 of Figure 11B. In these examples, the singleplex Dabsyl image 1115a contains blue (counter-stained, such as hematoxylin 1115b) and yellow channels 1115c, which are predicted to have a very small signal for TARMA (1115d). Starting from the initial color vectors, calibration or fine-tuning of the color vectors is performed via an interactive graphical user interface (GUI) 112 as a semi-automatic method for adjusting color vectors that can unmix the multiplex image with good quality. The second row 1120 of Figure 11B shows improved background noise in the TARMA channel using the adjusted color vectors.

[0190] Figure 12A shows an example of a duplex image 1202 superimposed with candidate species in each nucleus (marked with red dots) detected by automated nuclear segmentation. In this embodiment, automated nuclear segmentation was performed based on the iterative modified radial symmetry method, Parvin et al., 2007 (see references). The algorithm was performed on the hematoxylin image 1206 channel after unmixing the duplex image 1202. Duplex image 1202a provides a magnified view of the segments from 1202, while duplex image 1202b displays 1202a with the marked candidate species in each nucleus represented by red dots. The candidate species can serve as initial markers or reference points for nuclear segmentation. As shown in Figure 12A, these candidate species and species labels were detected and segmented from the duplex image 1202 using the unmixed hematoxylin 1206 (e.g., the segmented image provided in 1212). Next, the intensities of Dabsyl 1210 and TAMRA 1208 were assigned to each candidate seed. Simple filtering was applied to remove some stromal cells or cells with very low Dabsyl 1210 and TAMRA 1208 intensities. The intensities of Dabsyl 1210 and TAMRA 1208 were measured for each FOV.

[0191] Figure 12B depicts a comparison of nuclear segmentation results for hematoxylin images obtained by unmixing duplex images using linear inverse convolution (e.g., 1214) and NMF (e.g., 1216) techniques. From these images, it can be observed that the hematoxylin image channels unmixed using linear inverse convolution (e.g., 1214) are stained, and the cellular regions are not well delineated. In contrast, NMF more accurately unmixed the nuclear regions, resulting in improved nuclear definition that is separated from the background (as shown in 1216).

[0192] Singleplex slides were also investigated using linear deconvolution and NMF (e.g., 1218 and 1220, respectively), and it was determined that the nuclear segmentation results obtained from both unmixing methods showed comparable performance, as shown in the second row of Figure 12B. Table 1 lists the number of nuclei derived from duplex and singleplex images using the linear deconvolution method and the NMF method (with fine-tuning), respectively. This shows that the number of nuclei derived from duplex images using linear deconvolution is much higher than that of the NMF method, but there is little difference in singleplex images using both methods. The first duplex image was generated using the linear unmixing method to produce a composite singleplex image. As shown in Table 1, 784 nuclei were detected in the composite singleplex image. On the other hand, the actual adjacent singleplex image depicted 563 nuclei, showing a substantial discrepancy. Meanwhile, a second duplex image was generated using the NMF method (including unmixing) to produce a composite singleplex image. As shown in Table 1, 624 nuclei were detected in the synthesized singleplex image. In contrast, the actual adjacent singleplex image depicted 533 nuclei. Therefore, the NMF results are estimated to be more accurate than the linear unmixing method. [Table 1]

[0193] Figure 13 shows an exemplary graphical user interface (GUI) for generating composite pixels. As previously mentioned, the interface can be configured to interactively blend two or more staining colors in different ratios and display them in a cx-cy plot. Additionally, a given color vector associated with a particular chromophor can be adjusted using the interface. For example, the exemplary interface 1300 shows a set of four chromophores (1305) along with their associated RGB values, for example, Dabsyl [0.7108, 0.5888, 0.3849], TAMRA [0.9082, 0.3621, 0.21], Teal [0.244, 0.8821, 0.403], and hematoxylin [0.145, 0.2969, 0.9438]. The corresponding color vectors are plotted as pixels in the cx-cy plot 1310, where the marker ("X") represents the initial color vector of Dabsyl, which can be adjusted by tuning various adjustment options from interface 1300. For example, the intensity ratio (amount) of each chromophor 1305 may be selected via interface 1312. In addition to the amounts of chromophores, interface 1300 can also enable hue saturation adjustment 1314 of Dabsyl. By utilizing the adjustment options 1312 and 1314 for Dabsyl, the adjusted Dabsyl 1315 can be observed with an updated color vector of [0.7882, 0.6784, 0.3686], each with intensity ratios from the respective chromophores [1, 0.05, 0.05, 0.05, 0.05, 0.05]. Further tuning of the density ratio to [1,0,0,0] by selecting options from 1312 can result in a tuned Dabsyl 1320 with updated RGB values ​​of [0.8667,0.7412,0.3882].

[0194] While adjusting the amount of stain / chromophor, the adjusted amount is multiplied by the corresponding color vector in OD space, which is then converted back to RGB space for display. In interface 1300, Dabsyl's tuned pixels are shown in a cx-cy plot represented by ("*"). The positions of two pixels, e.g., initial Dabsyl ("X") and tuned Dabsyl ("*") in the cx-cy plot, indicate how close the two pixels are in hue and saturation. Scale up the amount of stain while maintaining the relative ratio of chromophores (e.g., keeping the composition the same), and while their positions in the cx-cy plot do not change, the appearance of the composite pixels changes, which is consistent with the design of the cx-cy space, which counts only hue and saturation while keeping the density the same.

[0195] Such a user interface can enable (1) visual inspection of the range of colors produced by specific combinations of chromogens from biomarker assays for both pathologist users and algorithm developers; (2) providing ground truth for staining unmixing, when the components of each chromogen that produce a synthetic color stain are known, and therefore color unmixing of groups of synthetic pixels can be performed, and the results can be compared to known settings used to produce these synthetic pixels; (3) studying potential unmixing errors (e.g., loss of staining signal in some of the unmixed images) when applying various regularizations of NMF-based unmixing, such as wedge constraints; and (4) assisting in the selection and comparison of chromogens by evaluating which chromogens are more feasible for unmixing.

[0196] Figure 14A shows an exemplary GUI that uses a composite pixel to evaluate the range of color from a blend of multiple chromogens. In a multiplex image, different chromogen colors can be blended when multiple biomarkers are stained on the same or nearby structures of tissue structures (e.g., a portion of tissue expressing multiple proteins detected by an assay). Such color blends can produce a variety of colors depending on the properties of the chromogens and the relative amounts of each chromogen deposited on the tissue structure. Figure 14A includes a cx-cy plot, e.g., 1402, representing the pixels associated with the constituent chromogens of a triplex image. In this plot 1402, an exemplary color is produced by blending Green and QM-Dabsyl (Green row 1408 in the composite pixel), represented as the "*" marker (pixel 1) in the cx-cy plot 1402. Another exemplary color is produced by blending Teal and QM-Dabsyl (Teal row 1410 in the composite pixel), represented as the "x" marker (pixel 2) in the cx-cy plot 1402. By adjusting these color vectors (associated with composite pixel 1 and pixel 2) through various adjustment options within the interface, different ranges of Green and Teal can be achieved, along with the generated color vectors in rows 1408 and 1410, respectively, as shown in cx-cy plots 1404 and 1406.

[0197] Additionally, the composite pixel generation interface can facilitate the selection and comparison of color primarys by evaluating which color primary is more feasible for unmixing, as illustrated in process 700 in Figure 7. Specifically, against a pre-defined set of color primarys, the color blend of one color primary under examination with another color primary can be examined, and then it can be quantitatively (by calculating how close the blended colors are in the cx-cy space) and qualitatively (by visual inspection) determined which color primary will produce a color range that supports accurate color unmixing. For example, a candidate color primary may be an inferior choice if, when blended with other color primarys, the blended color is similar to that of the other color primary.

[0198] Figure 14A further illustrates the determination of a recommended color as the third chromophor for the triplex assay. In an example where the selection of the third chromophor for the triplex assay is between Teal and Green (excluding yellow QM-Dabsyl and purple TAMRA), while Teal (cyan) or a blend of Green and QM-Dabsyl (purple) can produce pixels with diverse appearances corresponding to a wide range of hues and saturations, it may be observed that the blend with Teal produces a staining color similar to hematoxylin, as shown in row 1410 of Figure 14A. Such color similarity to hematoxylin pixels can increase the difficulty of staining unmixing, as an unmixing error may occur where the algorithm incorrectly unmixes such pixels to Teal and QM-Dabsyl instead of hematoxylin when the blended pixel value is close to hematoxylin. The results shown in the exemplary embodiment of Figure 14A suggest supporting Green instead of Teal as the third chromophor.

[0199] Figure 14B shows a comparison of one or more blended colors synthesized from stains from different reagent sources, according to an exemplary embodiment. In this example, the same type of chromophor ("Green") from different reagent sources is examined. The interface shows the associated color of the synthesized pixels produced from the user interface, as discussed above and in Figure 13. Figure 14B includes blends of pure hematoxylin 1420, TAMRA, and Green from Source 1 (lot: H27689) and Source 2 (lot: H35597) in blocks 1415a and 1415b, respectively. Chromophores from different reagent sources may result in slightly different staining colors, and the disclosed method may help in selecting the most reliable and preferred reagent source.

[0200] Figure 14C shows an example of blending two or more chromogens to generate a range of colors, according to an exemplary embodiment. Potential unmixing errors may include, for example, the loss of staining signals in the unmixed singleplex image when applying various regularizations of NMF-based unmixing. Using the NMF constraint toolbox 960, the triplex image may be unmixed based on the localization of the biomarker. Figure 14C shows an exemplary triplex 1424 of MET-PDL1-EGFR, along with the associated cx-cy plot 1422. Using the wedge constraint of the NMF method, the hematoxylin signal 1422a can be separated from the other positive biomarker stains in the first quadrant of the cx-cy space 1422. The remaining signals can be unmixed to Dabsyl, TAMRA, and Green using a three-color unmix in all other quadrants. The wedge constraints applied to hematoxylin and the assigned pixels within the wedge can be seen in the cx-cy plot 1425, represented by the red triangle in Figure 14C. The range of colors assigned to hematoxylin within the wedge can be evaluated using the composite pixel generation user interface. These color ranges may be the result of blending TAMRA, Green, and QM-Dabsyl.

[0201] Specifically, in the cx-cy plot 1430 in Figure 14C, a blend of TAMRA and Teal can produce a range of colors, which can lie on the line connecting the TAMRA and Green color vectors (the dark red line in plot 1430). A small amount of QM-Dabsyl can pull the color inward in a wedge shape. Such blended colors can be assigned to hematoxylin and thus generate unmixing errors. The degree of error may depend on the relative amounts of each chromogen corresponding to the expression levels of each biomarker that can be detected by the biomarker assay.

[0202] Figure 14D illustrates how to evaluate the range of colors assigned to hematoxylin using wedge constraints. In Figure 14D, an interface can be used to visualize one or more exemplary colors that could be misassigned to hematoxylin. Quantitatively, how wide the range of colors assigned to hematoxylin can be calculated can be calculated, for example, using the L2 norm of their cx-cy values ​​between two colors intersecting the wedge line and the line connecting the TAMRA and Green color vectors (gray arrows in Figure 14C).

[0203] Some embodiments of this disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-temporary computer-readable storage medium containing instructions such that, when the instructions are executed on one or more data processors, the one or more data processors cause one or more data processors to execute some or all of one or more of the methods disclosed herein and / or some or all of one or more processes. Some embodiments of this disclosure include a computer program product tangibly embodied in a non-temporary machine-readable storage medium, which includes instructions configured to cause one or more data processors to execute some or all of one or more of the methods disclosed herein and / or some or all of one or more processes.

[0204] This description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, this description of preferred exemplary embodiments provides a possible description for carrying out various embodiments for those skilled in the art. It will be understood that the function and arrangement of the elements can be varied without departing from the idea and scope described in the appended claims.

[0205] In order to provide a complete understanding of the embodiments, specific details are given herein. However, it will be understood that embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as not to obscure the embodiments.

Claims

1. A computer implementation method, For each of at least three digital pathology stains, determine the color vector representing the stain, This involves utilizing an interface to a user device, wherein the interface is Each of the aforementioned determined color vectors, Actual multiplex digital pathology images depicting biopsy sections stained with two or more of the aforementioned three digital pathology stains, At least one composite singleplex image, each of which is generated by filtering the actual multiplex digital pathology image using a single color vector from the determined color vectors, and A color vector adjustment tool comprising one or more color vector adjustment tools, each of which is configured to receive user input corresponding to the adjustment of a color vector representing a corresponding stain among the at least three digital pathology stains. This includes using an interface, To detect input received via interaction with the interface corresponding to a specific adjustment of the color vector representing a specific stain among the at least three digital pathology stains, The interface is automatically updated in response to the detection of the aforementioned input. Computer implementation methods, including those mentioned above.

2. The computer implementation method according to claim 1, wherein each of the determined color vectors' representations includes a representation of a position in optical density space.

3. The computer implementation method according to claim 1, wherein the updated interface further includes the at least one composite singleplex image.

4. The computer implementation method according to claim 1, wherein the one or more color vector adjustment tools include at least three color adjustment tools.

5. The computer implementation method according to claim 1, wherein determining the color vectors involves processing one or more single-stain images depicting the same or other biopsy sections stained with only one of the at least three digital pathology stains.

6. The computer implementation method according to claim 1, wherein the one or more monostained images include markers superimposed at specific positions in a multidimensional color space, and the color vectors are defined based on the specific positions.

7. The computer implementation method according to claim 1, wherein the actual multiplex digital pathology image depicts the biopsy section stained with at least four stains.

8. The determined color vector lies in a two-dimensional color space, and the method is The further includes determining a portion of the color space that is expected to be due to a prominent signal corresponding to a particular stain among the at least three digital pathological stains, The computer implementation method according to claim 1, wherein the automatic updating of the interface is performed using an unmixing technique that selectively focuses on the at least three digital pathological stains while excluding the specific stains.

9. The computer implementation method according to claim 1, wherein the determination of the color vector is performed using non-negative matrix factorization.

10. Receiving a new multiplex image stained with at least one of the three digital pathology stains mentioned above, A new composite singleplex image is generated based on the new multiplex image and the adjusted color vector. Outputting the aforementioned new composite singleplex image The computer implementation method according to claim 1, further comprising:

11. The computer implementation method according to claim 1, wherein the actual multiplex digital pathology image is filtered by using the color vector and machine learning model.

12. It is a system, One or more data processors, Non-temporary computer-readable storage medium containing instructions and The system includes, and when the instruction is executed on one or more data processors, it causes one or more data processors to perform an action, and the action is For each of at least three digital pathology stains, determine the color vector representing the stain, This involves utilizing an interface to a user device, wherein the interface is Each of the determined color vectors includes a representation of the position in optical density space, Actual multiplex digital pathology images depicting biopsy sections stained with two or more of the aforementioned three digital pathology stains, At least one composite singleplex image, each of which is generated by filtering the actual multiplex digital pathology image using a single color vector from the determined color vectors, and A color vector adjustment tool comprising one or more color vector adjustment tools, each of which is configured to receive user input corresponding to the adjustment of a color vector representing a corresponding stain among the at least three digital pathology stains. This includes using an interface, To detect input received via interaction with the interface corresponding to a specific adjustment of the color vector representing a specific stain among the at least three digital pathology stains, Automatically updating the interface in response to the detection of the input, wherein the updated interface further includes the at least one composite singleplex image. A system that includes this.

13. The system according to claim 12, wherein determining the color vectors includes processing one or more single-stain images depicting the same or other biopsy sections stained with only one of the at least three digital pathology stains.

14. The system according to claim 12, wherein the one or more monostained images include markers superimposed at specific positions in a multidimensional color space, and the color vectors are defined based on the specific positions.

15. The determined color vector lies in a two-dimensional color space, and the action is The further includes determining a portion of the color space that is expected to be due to a prominent signal corresponding to a particular stain among the at least three digital pathological stains, The system according to claim 12, wherein the automatic updating of the interface is performed using an unmixing technique that selectively focuses on the at least three digital pathology stains while excluding the specific stains, and the automatic updating of the interface is performed using non-negative matrix factorization.

16. The aforementioned action, Receiving a new multiplex image stained with at least one of the three digital pathology stains mentioned above, A new composite singleplex image is generated based on the new multiplex image and the adjusted color vector. Outputting the aforementioned new composite singleplex image The system according to claim 12, further comprising:

17. The system according to claim 12, wherein the actual multiplex digital pathology images are filtered by using the color vectors and machine learning models.

18. A computer program product tangibly embodied in a non-temporary machine-readable storage medium, which includes instructions configured to cause one or more data processors to perform an action, wherein the action is For each of at least three digital pathology stains, determine the color vector representing the stain, This involves utilizing an interface to a user device, wherein the interface is Each of the determined color vectors includes a representation of the position in optical density space, Actual multiplex digital pathology images depicting biopsy sections stained with two or more of the aforementioned three digital pathology stains, At least one composite singleplex image, each of which is generated by filtering the actual multiplex digital pathology image using a single color vector from the determined color vectors, and A color vector adjustment tool comprising one or more color vector adjustment tools, each of which is configured to receive user input corresponding to the adjustment of a color vector representing a corresponding stain among the at least three digital pathology stains. This includes using an interface, To detect input received via interaction with the interface corresponding to a specific adjustment of the color vector representing a specific stain among the at least three digital pathology stains, Automatically updating the interface in response to the detection of the input, wherein the updated interface further includes the at least one composite singleplex image. Computer program products, including [this].

19. The computer program product according to claim 18, wherein determining the color vectors includes processing one or more single-stain images depicting the same or other biopsy sections stained with only one of the at least three digital pathology stains.

20. The aforementioned action, Receiving a new multiplex image stained with at least one of the three digital pathology stains mentioned above, A new composite singleplex image is generated based on the new multiplex image and the adjusted color vector. Outputting the aforementioned new composite singleplex image The computer program product according to claim 18, further comprising:

21. A computer implementation method, For each of at least three digital pathology stains, determine the color vector representing the stain, Accessing an actual multiplex digital pathology image depicting a biopsy section stained with at least one first stain from the at least three stains, wherein the depicted biopsy section is not stained with at least one second stain from the at least three stains, A filtered output is generated by filtering the actual multiplex digital pathology image using the color vector representing one of the at least one second stains. To generate a metric that characterizes the signal characteristics in the filtered output, Using the metric and spatial traverse techniques to identify the adjustment of the color vector representing the second stain, Receiving a new multiplex image stained with at least one of the three digital pathology stains mentioned above, A new composite singleplex image is generated based on the new multiplex image and the adjusted color vector representing the second stain, Outputting the aforementioned new composite singleplex image Computer implementation methods, including those mentioned above.

22. The computer implementation method according to claim 21, wherein for each of the at least three digital pathological stains, the color vector is a vector in optical density space.

23. The computer implementation method according to claim 21, wherein the spatial traverse technique includes a gradient descent technique.

24. The computer implementation method according to claim 21, wherein the spatial traverse technique includes the Monte Carlo technique.

25. The computer implementation method according to claim 21, wherein the metric includes the mean, median, or mode intensity.

26. The computer implementation method according to claim 21, wherein the metric characterizes the level of staining over all or part of the filtered output.

27. The computer implementation method according to claim 21, wherein the filtered output is generated by using a machine learning model.

28. It is a system, One or more data processors, Non-temporary computer-readable storage medium containing instructions and The system includes, and when the instruction is executed on one or more data processors, it causes one or more data processors to perform an action, and the action is For each of at least three digital pathology stains, determine the color vector representing the stain, Accessing an actual multiplex digital pathology image depicting a biopsy section stained with at least one first stain from the at least three stains, wherein the depicted biopsy section is not stained with at least one second stain from the at least three stains, A filtered output is generated by filtering the actual multiplex digital pathology image using the color vector representing one of the at least one second stains. To generate a metric that characterizes the signal characteristics in the filtered output, Using the metric and spatial traverse techniques to identify the adjustment of the color vector representing the second stain, Receiving a new multiplex image stained with at least one of the three digital pathology stains mentioned above, A new composite singleplex image is generated based on the new multiplex image and the adjusted color vector representing the second stain, Outputting the aforementioned new composite singleplex image A system that includes this.

29. The system according to claim 28, wherein for each of the at least three digital pathological stains, the color vector is a vector in optical density space.

30. The system according to claim 28, wherein the spatial traverse technique includes a gradient descent technique.

31. The system according to claim 8, wherein the spatial traverse technique includes the Monte Carlo technique.

32. The system according to claim 28, wherein the metric includes the mean, median, or mode intensity.

33. The system according to claim 28, wherein the metric characterizes the level of staining over all or part of the filtered output.

34. The system according to claim 28, wherein the filtered output is generated by using a machine learning model.

35. A computer program product tangibly embodied in a non-temporary machine-readable storage medium, which includes instructions configured to cause one or more data processors to perform an action, wherein the action is For each of at least three digital pathology stains, determine the color vector representing the stain, Accessing an actual multiplex digital pathology image depicting a biopsy section stained with at least one first stain from the at least three stains, wherein the depicted biopsy section is not stained with at least one second stain from the at least three stains, A filtered output is generated by filtering the actual multiplex digital pathology image using the color vector representing one of the at least one second stains. To generate a metric that characterizes the signal characteristics in the filtered output, Using the metric and spatial traverse techniques to identify the adjustment of the color vector representing the second stain, Receiving a new multiplex image stained with at least one of the three digital pathology stains mentioned above, A new composite singleplex image is generated based on the new multiplex image and the adjusted color vector representing the second stain, Outputting the aforementioned new composite singleplex image Computer program products, including [this].

36. The computer program product according to claim 35, wherein for each of the at least three digital pathological stains, the color vector is a vector in optical density space.

37. The computer program product according to claim 35, wherein the spatial traverse technique includes a gradient descent technique.

38. The computer program product according to claim 35, wherein the spatial traverse technique includes the Monte Carlo technique.

39. The computer program product according to claim 35, wherein the metric includes the mean, median, or mode intensity.

40. The computer program product according to claim 35, wherein the metric characterizes the level of staining over all or part of the filtered output, and the filtered output is generated by using a machine learning model.

41. A computer implementation method, For each of at least two digital pathological stains, determine the color vector representing the stain, Access to actual multiplex digital pathology images depicting biopsy sections stained with at least two of the aforementioned digital pathology stains, The recommended color vectors represent potential additional staining. Identifying the initial color vector, A filtered output is generated by filtering the actual multiplex digital pathology image using the aforementioned initial color vector. To generate a metric that characterizes the signal characteristics in the filtered output, and Use the metric and spatial traverse techniques to identify the aforementioned recommended color vector. Identifying by, Outputting the aforementioned recommended color vector Computer implementation methods, including those mentioned above.

42. The computer implementation method according to claim 41, wherein the spatial traverse technique is performed such that it includes one or more targets in the traverse to minimize the signal in the filtered output.

43. The computer implementation method according to claim 42, wherein minimizing the signal in the filtered output includes minimizing the mean, median, or mode intensity of the corresponding filtered output.

44. The computer implementation method according to claim 41, wherein the determination of the color vector is carried out using non-negative matrix factorization.

45. The computer implementation method according to claim 41, wherein for each of the at least two digital pathological stains, the color vector is a vector in optical density space.

46. The computer implementation method according to claim 41, wherein the filtered output is generated by using a machine learning model.

47. The computer implementation method according to claim 41, wherein the spatial traverse technique includes a gradient descent technique.

48. It is a system, One or more data processors, Non-temporary computer-readable storage medium containing instructions and The system includes, and when the instruction is executed on one or more data processors, it causes one or more data processors to perform an action, and the action is For each of at least two digital pathological stains, determine the color vector representing the stain, Access to actual multiplex digital pathology images depicting biopsy sections stained with at least two of the aforementioned digital pathology stains, The recommended color vectors represent potential additional staining. Identifying the initial color vector, A filtered output is generated by filtering the actual multiplex digital pathology image using the aforementioned initial color vector. To generate a metric that characterizes the signal characteristics in the filtered output, and Use the metric and spatial traverse techniques to identify the aforementioned recommended color vector. Identifying by, Outputting the aforementioned recommended color vector A system that includes this.

49. The system according to claim 48, wherein the spatial traverse technique is performed such that it includes one or more targets in the traverse to minimize the signal in the filtered output.

50. The system according to claim 49, wherein minimizing the signal in the filtered output includes minimizing the mean, median, or mode intensity of the corresponding filtered output.

51. The system according to claim 48, wherein the determination of the color vector is carried out using non-negative matrix factorization.

52. The system according to claim 48, wherein for each of the at least two digital pathological stains, the color vector is a vector in optical density space.

53. The system according to claim 48, wherein the filtered output is generated by using a machine learning model.

54. A computer program product tangibly embodied in a non-temporary machine-readable storage medium, which includes instructions configured to cause one or more data processors to perform an action, wherein the action is For each of at least two digital pathological stains, determine the color vector representing the stain, Access to actual multiplex digital pathology images depicting biopsy sections stained with at least two of the aforementioned digital pathology stains, The recommended color vectors represent potential additional staining. Identifying the initial color vector, A filtered output is generated by filtering the actual multiplex digital pathology image using the aforementioned initial color vector. To generate a metric that characterizes the signal characteristics in the filtered output, and Use the metric and spatial traverse techniques to identify the aforementioned recommended color vector. Identifying by, Outputting the aforementioned recommended color vector Computer program products, including [this].

55. The computer program product according to claim 54, wherein the spatial traverse technique is performed to include one or more targets in the traverse in order to minimize the signal in the filtered output.

56. The computer program product according to claim 55, wherein minimizing the signal in the filtered output includes minimizing the mean, median, or mode intensity of the corresponding filtered output.

57. The computer program product according to claim 54, wherein for each of the at least two digital pathological stains, the color vector is a vector in optical density space.

58. The computer program product according to claim 54, wherein the filtered output is generated by using a machine learning model.

59. The computer program product according to claim 54, wherein the determination of the color vector is carried out using non-negative matrix factorization.

60. The computer program product according to claim 54, wherein the spatial traverse technique includes a gradient descent technique.

61. A computer implementation method, For each of at least three digital pathology stains, determine the color vector representing the stain, Accessing an actual multiplex digital pathology image depicting a biopsy section stained with at least one first stain from the at least three digital pathology stains, wherein the depicted biopsy section is not stained with at least one second stain from the at least three stains, A filtered output is generated by filtering the actual multiplex digital pathology image using the color vector representing one of the second stains among at least one second digital pathology stain, The above-mentioned at least three digital pathology stains generate a performance prediction score that represents the predicted degree to which they are actually sufficiently separable in order to reliably support the generation of a synthetic singleplex image, Outputting the aforementioned performance prediction score Computer implementation methods, including those mentioned above.

62. The computer implementation method according to claim 61, wherein the performance prediction score is generated using the filtered output.

63. The computer implementation method according to claim 61, wherein the performance prediction score includes the mean, median, or mode intensity of the corresponding filtered output.

64. The computer implementation method according to claim 61, wherein the performance prediction score includes correlation coefficients between each pair of synthetic singleplex images associated with the filtered output.

65. The computer implementation method according to claim 61, wherein for each of the at least three digital pathological stains, the color vector is a vector in optical density space.

66. The computer implementation method according to claim 61, wherein the filtered output is generated by using a machine learning model.

67. The computer implementation method according to claim 61, wherein the color vector is adjusted via a graphical user interface (GUI) based on the performance prediction score.

68. It is a system, One or more data processors, Non-temporary computer-readable storage medium containing instructions and The system includes, and when the instruction is executed on one or more data processors, it causes one or more data processors to perform an action, and the action is For each of at least three digital pathology stains, determine the color vector representing the stain, Accessing an actual multiplex digital pathology image depicting a biopsy section stained with at least one first stain from the at least three digital pathology stains, wherein the depicted biopsy section is not stained with at least one second stain from the at least three stains, A filtered output is generated by filtering the actual multiplex digital pathology image using the color vector representing one of the at least one second stains. The above-mentioned at least three digital pathology stains generate a performance prediction score that represents the predicted degree to which they are actually sufficiently separable in order to reliably support the generation of a synthetic singleplex image, Outputting the aforementioned performance prediction score A system that includes this.

69. The system according to claim 68, wherein the performance prediction score is generated using the filtered output.

70. The system according to claim 68, wherein the performance prediction score includes the mean, median, or mode intensity of the corresponding filtered output.

71. The system according to claim 68, wherein the performance prediction score includes correlation coefficients between each pair of synthetic singleplex images associated with the filtered output.

72. The system according to claim 68, wherein the filtered output is generated by using a machine learning model.

73. The system according to claim 68, wherein for each of at least three digital pathological stains, the color vector is a vector in optical density space.

74. The system according to claim 68, wherein the color vector is adjusted via a graphical user interface (GUI) based on the performance prediction score.

75. A computer program product tangibly embodied in a non-temporary machine-readable storage medium, which includes instructions configured to cause one or more data processors to perform an action, wherein the action is For each of at least three digital pathology stains, determine the color vector representing the stain, Accessing an actual multiplex digital pathology image depicting a biopsy section stained with at least one first stain from the at least three digital pathology stains, wherein the depicted biopsy section is not stained with at least one second stain from the at least three stains, A filtered output is generated by filtering the actual multiplex digital pathology image using the color vector representing one of the at least one second stains. The above-mentioned at least three digital pathology stains generate a performance prediction score that represents the predicted degree to which they are actually sufficiently separable in order to reliably support the generation of a synthetic singleplex image, Outputting the aforementioned performance prediction score Computer program products, including [this].

76. The computer program product according to claim 75, wherein the performance prediction score is generated using the filtered output.

77. The computer program product according to claim 75, wherein for each of the at least three digital pathological stains, the color vector is a vector in optical density space.

78. The computer program product according to claim 75, wherein the performance prediction score includes the correlation coefficient between each pair of synthesized singleplex images.

79. The computer program product according to claim 75, wherein the filtered output is generated by using a machine learning model.

80. The computer program product according to claim 75, wherein the color vector is adjusted via a graphical user interface (GUI) based on the performance prediction score.

81. A computer implementation method, For each of at least four digital pathological stains, the color vector representing the stain is determined, wherein the determined color vector represents the stain in a multidimensional color space. Selecting a specific stain from at least four of the aforementioned digital pathology stains, To determine the portion of the color space that is predicted to be caused by a prominent signal corresponding to the particular staining, Accessing an actual multiplex digital pathology image depicting a biopsy section stained with at least three of the four digital pathology stains, wherein the actual multiplex digital pathology image includes a set of pixels. Mapping each pixel in the set of pixels in the actual multiplex digital pathology image to a point in the multidimensional color space, For each of the at least four digital pathology stains, the process includes generating a pixel-specific color vector for each pixel in the set of pixels that predicts the degree of staining expression in the portion of the biopsy section depicted in the pixel, wherein generating the pixel-specific color vector is Determining that each of the first subsets of the set of pixels is mapped to a point in the portion of the color space, Determining the optical density for each pixel in a first subset of the aforementioned pixels, wherein the pixel-specific color vector of the pixel identifies the degree of expression of the particular stain corresponding to the optical density. Determining that each of the second subsets of the set of pixels is mapped to a point outside the portion of the color space, For each pixel in the second subset and each of the at least four digital pathology stains, an unmixing technique is performed to predict the degree of expression of the stain in the portion of the biopsy section depicted in the pixel, wherein the at least four digital pathology stains do not include the particular stain, and the unmixing technique uses the color vectors determined to represent each of the at least four digital pathology stains. A computer implementation method comprising generating one or more composite singleplex images using the aforementioned pixel-specific color vectors.

82. The computer implementation method according to claim 81, wherein the particular stain is selected based on information regarding which part of a cell each of the at least four digital pathological stains is configured to stain.

83. The computer implementation method according to claim 81, wherein the color space includes the International Commission on Illumination (CIE) color space.

84. The computer implementation method according to claim 81, wherein the portion of the color space includes a wedge shape.

85. The computer implementation method according to claim 81, wherein the portion of the color space includes a portion of a space defined based on an inequality relating to the x-coordinate and an inequality relating to the y-coordinate.

86. The computer implementation method according to claim 81, wherein the portion of the color space includes a combination of primitives.

87. The computer implementation method according to claim 81, wherein the unmixing technique is performed by using non-negative matrix factorization (NMF).

88. The computer implementation method according to claim 81, wherein the color vector is determined based on one or more user inputs received using one or more color vector adjustment tools available within the interface.

89. It is a system, One or more data processors, Non-temporary computer-readable storage medium containing instructions and The system includes, and when the instruction is executed on one or more data processors, it causes one or more data processors to perform an action, and the action is For each of at least four digital pathological stains, the color vector representing the stain is determined, wherein the determined color vector represents the stain in a multidimensional color space. Selecting a specific stain from at least four of the aforementioned digital pathology stains, To determine the portion of the color space that is predicted to be caused by a prominent signal corresponding to a particular stain, Accessing an actual multiplex digital pathology image depicting a biopsy section stained with at least three of the four digital pathology stains, wherein the actual multiplex digital pathology image includes a set of pixels. Mapping each pixel in the set of pixels in the actual multiplex digital pathology image to a point in the multidimensional color space, For each of the at least four digital pathology stains, the process includes generating a pixel-specific color vector for each pixel in the set of pixels that predicts the degree of staining expression in the portion of the biopsy section depicted in the pixel, wherein generating the pixel-specific color vector is Determining that each of the first subsets of the set of pixels is mapped to a point in the portion of the color space, Determining the optical density for each pixel in a first subset of the aforementioned pixels, wherein the pixel-specific color vector of the pixel identifies the degree of expression of the particular stain corresponding to the optical density. Determining that each of the second subsets of the set of pixels is mapped to a point outside the portion of the color space, For each pixel in the second subset and each of the at least four digital pathology stains, an unmixing technique is performed to predict the degree of expression of the stain in the portion of the biopsy section depicted in the pixel, wherein the at least four digital pathology stains do not include the particular stain, and the unmixing technique uses the color vectors determined to represent each of the at least four digital pathology stains. A system comprising generating one or more composite singleplex images using the aforementioned pixel-specific color vectors.

90. The system according to claim 89, wherein the particular stain is selected based on information regarding which part of a cell each of the at least four digital pathological stains is configured to stain.

91. The system according to claim 89, wherein the color space includes the International Commission on Illumination (CIE) color space.

92. The system according to claim 89, wherein the portion of the color space includes a wedge shape or a combination of primitives.

93. The system according to claim 89, wherein the portion of the color space includes a portion of a space defined based on an inequality relating to the x-coordinate and an inequality relating to the y-coordinate.

94. The system according to claim 89, wherein the unmixing technique is performed by using non-negative matrix factorization (NMF).

95. The system according to claim 89, wherein the color vector is determined based on one or more user inputs received using one or more color vector adjustment tools available within the interface.

96. A computer program product tangibly embodied in a non-temporary machine-readable storage medium, which includes instructions configured to cause one or more data processors to perform an action, wherein the action is For each of at least four digital pathological stains, the color vector representing the stain is determined, wherein the determined color vector represents the stain in a multidimensional color space. Selecting a specific stain from at least four of the aforementioned digital pathology stains, To determine the portion of the color space that is predicted to be caused by a prominent signal corresponding to a particular stain, Accessing an actual multiplex digital pathology image depicting a biopsy section stained with at least three of the four digital pathology stains, wherein the actual multiplex digital pathology image includes a set of pixels. Mapping each pixel in the set of pixels in the actual multiplex digital pathology image to a point in the multidimensional color space, For each of the at least four digital pathology stains, the process includes generating a pixel-specific color vector for each pixel in the set of pixels that predicts the degree of staining expression in the portion of the biopsy section depicted in the pixel, wherein generating the pixel-specific color vector is Determining that each of the first subsets of the set of pixels is mapped to a point in the portion of the color space, Determining the optical density for each pixel in a first subset of the aforementioned pixels, wherein the pixel-specific color vector of the pixel identifies the degree of expression of the particular stain corresponding to the optical density. Determining that each of the second subsets of the set of pixels is mapped to a point outside the portion of the color space, For each pixel in the second subset and each of the at least four digital pathology stains, an unmixing technique is performed to predict the degree of expression of the stain in the portion of the biopsy section depicted in the pixel, wherein the at least four digital pathology stains do not include the particular stain, and the unmixing technique uses the color vectors determined to represent each of the at least four digital pathology stains. A computer program product comprising generating one or more composite singleplex images using the aforementioned pixel-specific color vectors.

97. The computer program product according to claim 96, wherein the particular stain is selected based on information regarding which part of a cell each of the at least four digital pathological stains is configured to stain.

98. The computer program product according to claim 96, wherein the portion of the color space includes a wedge shape, a combination of primitives, or a portion of a space defined based on inequalities relating to the x-coordinate and inequalities relating to the y-coordinate.

99. The computer program product according to claim 96, wherein the unmixing technique is performed by using non-negative matrix factorization (NMF).

100. The computer program product according to claim 96, wherein the color vector is determined based on one or more user inputs received using one or more color vector adjustment tools available within the interface.

Images