Virtual stain multiplexing model architecture for changeable staining color

Abstract

The present disclosure relates to methods and apparatuses for generating virtual staining for an image of a biologic sample. Such virtual staining comprises obtaining a virtual stain concentration image including applying an artificial intelligence model to the image of a biologic sample. Moreover, a staining control parameter is set. Then, a virtual stain image is generated by processing the virtual stain concentration image according to the staining control parameter. In some exemplary implementations, there may be more than one virtual stain image generated using different respective staining control parameters. The generated virtual stain image(s) may be combined with the original (unstained image) to obtain a stained image.

Description

VIRTUAL STAIN MULTIPLEXING MODEL ARCHITECTURE FORCHANGEABLE STAINING COLORCross-Reference to Related Application

[0001] The present application claims the benefit of priority to U.S. Provisional Application No. 63 / 679,010, filed August 2, 2024, the content of which is hereby incorporated by reference in its entirety.Technical Field

[0002] The present disclosure relates generally to methods and devices for use in detecting targets in biological samples aided by imaging. In particular, the present disclosure relates to virtual staining of biological samples.Background

[0003] Histological specimens are frequently disposed upon glass slides as a thin slice of patient tissue fixed to the surface of each of the glass slides. Using a variety of chemical or biochemical processes, one or several colored compounds may be used to stain the tissue to differentiate cellular constituents, which can be further evaluated utilizing microscopy. Bright- field slide scanners are conventionally used to digitally analyze these slides.

[0004] When analyzing tissue samples on a microscope slide, staining the tissue or certain parts of the tissue with a colored or fluorescent dye can aid the analysis. Staining refers to applying color to biologic (e.g., cellular) features. Dyes are typically soluble in water or an organic solvent and work by binding to objects that they have a molecular affinity for. In other words, colored compounds in the dye stick to the object being stained (which may be colorless), effectively transferring their color. A chromogen is a colorless chemical compound that is only converted to a colored compound through a chemical reaction.

[0005] The ability to visualize or differentially identify microscopic structures is frequently enhanced using histological stains. An example of a dye is hematoxylin. When applied to cells, the positively charged hematoxylin compounds stick to negatively charged chromatin in cell nuclei making the nuclei appear blue. Hematoxylin and eosin (“H&E”) stains are commonly used stains in light microscopy for histological samples. In addition to H&E stains, other stains or dyes have been applied to provide more specific staining and provide a more detailed view of tissue morphology.

[0006] In Immunohistochemistry (“IHC”), such reactions occurs by the action of enzymes. Typically, one or more enzymes are directly or indirectly attached to an antibody-antigen binding site in order to catalyze the conversion of chromogen features into different colored products. Common chromogens include 3,3 '-diaminobenzidine tetrahydrochloride (“DAB”), which reacts with the enzyme horseradish peroxidase (“HRP”) to produce a brown end product, or Fast Red, which reacts with the enzyme alkaline phosphatase (“AP”) to produce a red end product. Other chromogens are available to achieve other colors.

[0007] Thus, IHC stains using a peroxidase substrate or AP substrate for IHC staining, provide a uniform staining pattern that appears to the viewer as a homogeneous color with intracellular resolution of cellular structures, e.g., membrane, cytoplasm, and nucleus.

[0008] Formalin Fixed Paraffin Embedded (“FFPE”) tissue samples, metaphase spreads or histological smears are typically analyzed by staining on a glass slide, where a particular biomarker, such as a protein or nucleic acid of interest, can be stained with H&E and / or with a colored dye, hereafter “chromogen” or “chromogenic moiety.” IHC staining is a common tool in the evaluation of tissue samples for the presence of specific biomarkers. In situ hybridization (“ISH”) may be used to detect target nucleic acids in a tissue sample. ISH may employ nucleic acids labeled with a directly detectable moiety, such as a fluorescent moiety, or an indirectly detectable moiety, such as a moiety recognized by an antibody which can then be utilized to generate a detectable signal. Further approaches, such as Fluorescence ISH (“FISH”), have also been developed.

[0009] Compared to other detection techniques, such as radioactivity, chemo-luminescence or fluorescence, chromogens generally suffer from much lower sensitivity, but have the advantage of providing a permanent, plainly visible color which can be visually observed, such as with bright field microscopy.

[0010] In recent years, digital pathology has gained more popularity as many stained tissueslides are digitally scanned with high resolution (e.g., 40*) and viewed as whole slide images (“WSIs”) using digital devices (e.g., PCs, tablets, etc.) instead of standard microscopes. Having the information in a digital format enables digital analyses that may be applied to WSI for instance to facilitate diagnoses. Recently, quantitative staining approaches have been developed, which convert antibody / antigen complexes into dots. These dots may then be detected and they provide quantitative measure for expression of a desired molecules (e.g. proteins).

[0011] Moreover, the development of artificial intelligence (“Al”), has provided new possibilities for analyzing and generating images. For example, virtual staining may begenerated by an Al model that is trained to produce stained images from unstained images or from images stained in different ways. However, current options for multiplexed assays using chromogens, or virtually, remain limited.Brief Summary

[0012] The present disclosure provides computer-implemented systems and methods that facilitate configurable, customizable virtual staining. Various embodiments of such systems and methods, as well as other aspects of the disclosure shall become apparent in view of the following description and the accompanying figures.

[0013] In a first aspect, a computer-implemented method is provided for generating virtual staining for an image of a biologic sample, the method comprising: obtaining a virtual stain concentration image by applying an Al model to the image of a biologic sample; setting or receiving a staining control parameter; and generating a virtual stain image by processing the virtual stain concentration image according to the staining control parameter.

[0014] In a second aspect, in addition to the first aspect, the staining control parameter is a parameter specifying at least one color to be applied to the virtual stain concentration image.

[0015] In a third aspect, in addition to the first aspect or the second aspect, the staining control parameter is a parameter specifying at least one filter to be applied to the virtual stain concentration image.

[0016] In a fourth aspect, in addition to the third aspect, the staining control parameter is a parameter specifying at least one of level of quantization, contrast, brightness, or smoothness to be applied to the virtual stain concentration image.

[0017] In a fifth aspect, in addition to any of the second aspect to fourth aspect, the virtual stain concentration image is an image in an optical density space, in which an intensity of a pixel is correlated with stain concentration.

[0018] In a sixth aspect, in addition to any of the first aspect to fifth aspect, the method further comprises combining the image of a biologic sample with the virtual stain image.

[0019] In a seventh aspect, in addition to any of the first aspect to fifth aspect, the method comprises repetition of: obtaining the virtual stain concentration image; setting or receiving the staining control parameter; and the generating a virtual stain image a plurality, TV, of times, thereby obtaining N virtual stain images, and combining the image of a biologic sample with the N virtual stain images.

[0020] In an eighth aspect, in addition the seventh aspect, the repetition of the obtaining stain concentration image N times includes application of N different Al models trained to generaterespective N different stain concentration images. In some aspects, one or more (e.g., each) of the different stain concentration images may imitate the staining of a different antigen.

[0021] In a ninth aspect, in addition to any of the sixth aspect to eighth aspect, the combining comprises color processing of the image of a biologic sample and setting the staining control parameter, e.g., to imitate a result of a fluorescence-based staining.

[0022] In a tenth aspect, in addition to any of the first aspect to ninth aspect, said setting a staining control parameter is obtained as an output of an Al model (e.g., a neural network) processing one or more patches of said image of a biologic sample.

[0023] In an eleventh aspect, in addition to the tenth aspect, said one or more patches comprise a plurality of patches; and for each of the plurality of patches, the Al obtains more than one patch staining control parameters; and in said setting the staining control parameter, the staining control parameter is obtained as a statistic measure of the more than one patch staining parameters.

[0024] In a twelfth aspect, in addition to any of the first aspect to eleventh aspect, the method comprises generating a graphical user interface(“GUI”), the GUI comprising an input means for a user to set the staining control parameter; and controlling a display to display the GUI.

[0025] In 13-th aspect, further to the twelfth aspect, the GUI further comprises: an area for displaying the image of the biologic sample and the virtual stain image in response to receiving an input via said input means.

[0026] In a 14-th aspect, in addition to any of the sixth to eleventh aspect, the method comprises generating a GUI, the GUI comprising an input means for a user to set the staining control parameter; and controlling a display to display the GUI. The GUI further includes a widget that allows a user to: select said N virtual stain concentration images among L virtual stain concentration image, L being equal to or larger than N, for each of the N virtual stain concentration images, respectively specify said control parameter, and displays said N virtual stain concentration images superimposed on the image of the biologic sample.

[0027] In a 15-th aspect, a computer-implemented method is provided for generating virtual staining for an image of a biologic sample, the method comprising: repeating N times, A being an integer larger than 1, the following step 1=1... N: (i) obtaining an z-th virtual stain concentration image including applying an artificial intelligence model to the image of a biologic sample; (ii) setting an z-th staining control parameter; and (iii) generating an z-th virtual stain image by processing the z-the virtual stain concentration image according to the z-th staining control parameter; and the method further comprises combining the image of a biologic sample with the N virtual stain images.

[0028] In a 16-th aspect, a computer program is provided, stored on a non-transitory medium and including code instructions which, when executed on one or more processors, causes the one or more processors to perform steps of the method according any of first to 15-th aspect.

[0029] In a 17-th aspect, an apparatus is provided for generating virtual staining for an image of a biologic sample, the apparatus comprising: processing circuitry, which in operation: (i) obtains a virtual stain concentration image including applying an Al model to the image of a biologic sample; (ii) sets a staining control parameter; and (iii) generates a virtual stain image by processing the virtual stain concentration image according to the staining control parameter, and an output interface, configured to output the virtual stain image generated.

[0030] In an 18-the aspect, the staining control parameter is a parameter specifying at least one color to be applied to the virtual stain concentration image.

[0031] In a 19-th aspect, in addition to the 17-th or 18-th aspect, the staining control parameter is a parameter specifying at least one filter to be applied to the virtual stain concentration image.

[0032] In a 20-th aspect, in addition to the 19-th aspect, the staining control parameter is a parameter specifying at least one of level of quantization, contrast, brightness, or smoothness to be applied to the virtual stain concentration image.

[0033] In a 21-st aspect, in addition to any of the 18-th to 20-th aspect, the virtual stain concentration image is an image in an optical density space, in which an intensity of a pixel is correlated with stain concentration.

[0034] In a 22-nd aspect, further to any of the 17-th to 21-st aspect, the processing circuitry, in operation further performs combining the image of a biologic sample with the virtual stain image.

[0035] In a 23-rd aspect, in addition to any of the 17-th to 21-st aspect, the processing circuitry, in operation further performs: repetition of the obtaining virtual stain concentration image; setting a staining control parameter; and the generating a virtual stain image a plurality, N, of times, thereby obtaining N virtual stain images, and combining the image of a biologic sample with the N virtual stain images.

[0036] In a 24-th aspect, further to the 23-rd aspect, the repetition of the obtaining stain concentration image N times includes application of N different Al models trained to generate respective A different stain concentration images (e.g., imitating staining respectively different antigens).

[0037] In a 25-th aspect, further to any of the 22-nd to 24-th aspect, the combining comprises color processing of the image of a biologic sample and setting the staining control parameter, so as to imitate result of a fluorescence-based staining.

[0038] In a 26-th aspect, further to any of the 17-th to 25-th aspect, said setting a staining control parameter is obtained as an output of an Al, e.g., a neural network configured to process one or more patches of said image of a biologic sample.

[0039] In a 27-th aspect, in addition to the 26-th aspect, said one or more patches comprise a plurality of patches; for each of the plurality of patches, the Al obtains more than one patch staining control parameters; and in said setting the staining control parameter, the staining control parameter is obtained as a statistic measure of the more than one patch staining parameters.

[0040] In a 28-th aspect, in addition to any of the 17-th to 27-th aspect, the processing circuitry, in operation further performs: generating a GUI, the GUI comprising an input means for a user to set the staining control parameter; and controlling a display to display the GUI.

[0041] In a 29-th aspect, in addition to the 28-th aspect, the GUI further comprises an area for displaying the image of the biologic sample and the virtual stain image in response to receiving an input via said input means.

[0042] In a 30-th aspect, in addition to any of the 22-nd to 27-th aspect, the processing circuitry, in operation further performs: generating a GUI, the GUI comprising an input means for a user to set the staining control parameter; and controlling a display to display the GUI. The GUI may further include a widget that allows a user to: select said N virtual stain concentration images among L virtual stain concentration image, L being equal to or larger than A, for each of the N virtual stain concentration images, respectively specify said control parameter, and displays said N virtual stain concentration images superimposed on the image of the biologic sample.

[0043] In a 31-st aspect, an apparatus is provided for generating virtual staining for an image of a biologic sample, comprising: processing circuitry that, in operation, performs: repeating A times, Abeing an integer larger than 1, the following step z=l...A obtaining an z-th virtual stain concentration image including applying an artificial intelligence model to the image of a biologic sample; setting an z-th staining control parameter; and generating an z-th virtual stain image by processing the z-the virtual stain concentration image according to the z-th staining control parameter, and combining the image of a biologic sample with the A virtual stain images into a combined image. The apparatus further comprises an output interface, configured to output the combined image.Brief Description of the Drawings

[0044] A further understanding of the nature and advantages of particular embodiments may be realized by reference to the remaining portions of the specification and the drawings, inwhich like reference numerals are used to refer to similar components. In some instances, a sublabel is associated with a reference numeral to denote one or multiple similar components. When reference is made to a reference numeral without specification to an existing sub-label, it is intended to refer to all such multiple similar components.

[0045] FIG. 1 is a block diagram illustrating a virtual staining module and its inputs and output.

[0046] FIG. 2 is a block diagram illustrating an exemplary implementation of the virtual staining module with separated stain generation and parameter-based stain processing.

[0047] FIG. 3 is a block diagram illustrating an exemplary implementation of the virtual staining module with separated stain generation and parameter-based stain processing with parameter controlling color of the virtual stain.

[0048] FIG. 4 is a block diagram illustrating an exemplary implementation of the virtual staining module with stain generation and parameter-based stain processing not necessarily separated.

[0049] FIG. 5 is a block diagram an exemplary implementation of the virtual staining module with separated stain generation and parameter-based stain processing facilitating multiplexing of multiple stains.

[0050] FIG. 6 is a schematic drawing illustrating a GUI that may serve to a human user for interaction with the virtual staining module including setting of staining control parameter(s) and displaying results of virtual staining.

[0051] FIG. 7 is a flow diagram illustrating steps of a method for generating virtual staining.

[0052] FIG. 8 is a block diagram illustrating an exemplary hardware architecture that may be used to implement present disclosure.

[0053] FIG. 9 is a block diagram illustrating functional modules of a memory in the exemplary hardware architecture.

[0054] FIG. 10 is a system diagram illustrating a possible system for providing and using applications of virtual staining of the present disclosure.

[0055] FIG. 11 is a drawing showing results of an exemplary virtual staining.

[0056] FIG. 12 is a block diagram illustrating the model architecture and training loss structure for an exemplary implementation of the virtual staining module with separated stain generation and parameter-based stain processing with parameter controlling color of the virtual stain.

[0057] FIG. 13 is a flowchart showing an exemplary workflow for dataset preparation.

[0058] FIG. 14 is an image showing an example of tumor annotation using an Al model trained according to the methods described herein.

[0059] FIG. 15 is a table showing the architecture and training details of an exemplary Al model trained according to the methods described herein.

[0060] FIG. 16 is a set of images showing a comparison of multiplexed virtual stain and ground truth patches (left). The panels on the right show: (a) PD-L1+ cells annotated as CD68+ based on sequential CD68 staining by Pathologist 1 (ground truth); (b) the virtual stain- multiplexed CD68 model correctly stains the tumor-infiltrating macrophages; and (c) annotation of macrophages by Pathologist 2, based on PD-L1 staining only. The results are compared to ground-truth annotation and presented as True Positive (“TP”) and False Negative (“FN”).

[0061] FIG. 17 is a set of images showing qualitative effect of combined loss. Comparing (left to right) input, ground-truth, full model output and the output of an ablation model trained only using MSE loss. Dashed rectangles indicate a macrophage with clear membranal staining, which is partially stained in ground-truth patch, fully stained in full model outputs, and only faintly stained in the ablation model outputs

[0062] FIG. 18 illustrates a cell classification model evaluation. Confusion matrices comparing Pathologist 2 annotations to Pathologist 1 ground truth annotations are depicted, with (right) and without (left) the aid of virtual stain.

[0063] FIGs. 19A-D illustrate the result of model training to infer a virtual stain for CD3 through successive epochs.

[0064] FIGs. 20A-C illustrate the result of model training to infer a virtual stain for CD68 through successive epochs.

[0065] FIGs. 21A-B illustrate a sequential multiplexing workflow (FIG. 21A) and a virtual staining model (“VSM”) workflow (FIG. 21B), which are described further in Example 7.

[0066] FIGs. 22A-C illustrates cell-level annotations, represented by dots on top of squares marking sample cells, with CD68- and CD68+ classes colored blue and magenta, respectively. Annotations of (A) PD-Ll-only WSIs and (C) PD-L1 + VSM CD68, by P1-P3 (B) Groundtruth by the study pathologist, using PD-L1 + CD68 IHC sequentially stained on the same section. Note full-consensus false-negative due to model error on bottom-left cell. FIG. 22D shows an agreement analysis of consensus annotated CD68+ cells using PD-Ll-only WSIs. Black dots indicate cells with CD68+ ground truth annotations.

[0067] FIGs. 23A-F provide a set of graphs showing performance metrics compared to the study pathologist ground-truth across six evaluation regions (labeledROI) with and without assistive virtual staining multiplexing. PD-L1 staining only is shown by filled symbols, and assistive virtual stain multiplexing is shown by open symbols. Changes in pooled performance metrics are indicated by a dashed black line. FIGs. 23A-C provide receiver operating characteristic (“ROC”) plots for P1-P3, respectively. FIG. 23D is a ROC plot, FIG. 23E is a precision-recall plot and FIG. 23F shows Fl scores for majority -vote consensus annotations.

[0068] FIGs. 24-29 provide a set of images of the regions of interest (“ROIs”) used for macrophage detection evaluation described in Example 7. For each ROI, we give the PD-L1 only, PD-L1 + VSM CD68 and PD-L1 + sequential CD68 IHC.Detailed Description

[0069] The present disclosure relates, in general, to methods, programs, systems, and apparatuses for facilitating digital microscopy imaging (e.g., digital pathology or live cell imaging, etc.). More specifically, the present disclosure relates to implementing digital microscopy imaging using virtual staining.

[0070] Methods of virtual staining are known in the art. However, a majority of such methods are only able to infer a single stain, representing inference of a single biomarker, or a single combination of stains. In pathology and other scientific fields, tissue sections or cells are often stained for multiple biomarkers, for example with serial or multiplex IHC. As such, there is often a need to infer multiple virtual stains, for example when tissue sections are limited or to enable inference of more biomarkers on a single tissue section or image. For other methods of virtual staining that infer and display a stain of a single color, there will be ambiguity if different biomarkers were displayed with the same color of virtual staining.

[0071] In a traditional (single-plex) virtual staining model, the model is trained using a specific stain color and is only able to predict / virtually-multiplex with same color. In addition, in brightfield microscopy the spectrum of colors available for IHC staining is usually limited, which in turns limits ground-truth multiplexed slide generation. Moreover, for a traditional virtual staining model, the relative intensity of the virtual stain, as well as its contrast and opacity are fixed and cannot be optimized for a specific task or according to the preferences of the user (e.g. a pathologist). Traditional virtual staining models are described, e.g., in U.S. Patent Application No. 18 / 021,779, which published as Pre-grant Pub. No. 2024 / 0029409, the entire of content of which (e.g., describing virtual staining architectures and virtual stains) is herein incorporated by reference.

[0072] Among other benefits, the present systems and methods address the problem of predicting and visualizing one or more virtual stains on top of a single whole-slide imagethrough an Al model. Using a different architecture of the model, one can independently adjust stain control parameters for different virtual stains, enabling multiple different colors to be used for different virtual stains, or enabling different colors to be used for a single virtual stain. With the present methods, any color could be applied as virtual staining multiplexing regardless of the color of the ground-truth IHC. Also, the intensity of the virtual stain can be adjusted relative to the input image or relative to other virtual stains. These advantages enable several different virtual IHC stains to be multiplexed (visualized) on top of a single IHC image, H&E image, or other image used as input to the model. In some embodiments, the inferred virtual stain can be transformed by taking its negative to simulate fluorescent staining. Specifically, the decoupling of the antibody concentration from its color creates an 2D single layer intensity image (as opposed to an RGB image) which opens possibilities to use the image as a binary mask, or as an antibody heatmap, or to change its transparency while used in conjunction with other virtual stains, enabling virtual stain multiplexing.

[0073] The following detailed description illustrates a few exemplary embodiments in further detail to enable one of skill in the art to practice such embodiments. The examples described herein are provided for explanatory purposes and are not intended to limit the scope of the invention, which is defined solely by the claims.

[0074] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the described embodiments. It will be apparent to one skilled in the art, however, that other embodiments of the present disclosure may be practiced without some of these specific details. In other instances, certain structures and devices are shown in block diagram form. Several embodiments are described herein, and while various features are ascribed to different embodiments, it should be appreciated that the features described with respect to one embodiment may be incorporated with other embodiments as well. By the same token, however, no single feature or features of any described embodiment should be considered essential to every embodiment disclosed herein, as other embodiments may omit such features.

[0075] In a virtual staining approach, a machine learning (“ML”) capable Al model may be trained using a specific stain color, allowing the trained Al model to generate virtual stains with the same color. Here, color is only exemplary; in general, the models can generate virtual stains that correspond to the ground truth stains they have been trained with. For example, in brightfield microscopy, the spectrum of colors available for IHC staining is usually limited, which in turn limits ground-truth multiplexed slide generation. In addition, for a virtual staining model the relative intensity of the virtual stain, as well as its contrast and opacity are fixed andcannot be optimized for a specific task or according to the preferences of the user (e.g. pathologist).

[0076] The present disclosure relates to virtual stain generation which facilitates configurability of the virtual stain features and thus alleviates the above-mentioned limitations, in addition to providing other benefits that will become apparent in view of the following description and the drawings. In some aspects, the virtual stain generation may be based on multiplexing model architecture for changeable staining color and / or other stain characteristics. For example, FIG. 1 shows a block of virtual staining 100, which has an input 110 and an output 190. The input 110 is an image to be virtually stained. The output 190 is an image of a virtual stain. This may be an image of the virtual stain alone or it may be a combined image of the virtual stain and the image to be stained 110. In addition, there is a parameter 150 that relates to the virtual stain. By way of this parameter, the output of the virtual staining may be influenced.

[0077] As shown in FIG. 2, the virtual stain generation module 100 may further comprise an Al module 120 and a stain generation module 130. The Al module 120, in operation, obtains a virtual stain concentration image 125 (e.g., by applying an Al model to the image of a biologic sample). The stain generation module 130, in operation, enables setting of one or more staining control parameter(s) 150. Moreover, the stain generation module 130, in operation, generates a virtual stain image by processing the virtual stain concentration image 125 according to the staining control parameter 150. The processing of the virtual stain concentration image may be performed, e.g., by multiplexing the virtual stain concentration image with a selected color and / or filter and / or an Al model trained for stain processing. The virtual stain generation module 100 in FIG. 2 thus has a separated Al module and the stain generation module. This is an efficient architecture which enables training of virtual stain generation based on one staining type / color and still allows for further adapting the stain characteristics for visualization based on the input parameter. In this way, the training of the Al model may remain simple, while providing options to further design the stain appearance once it has been detected by the Al model. In this manner, stained tissue data with configurable stain visualization may be provided. Notwithstanding the above, the architecture shown in FIG. 2 is non-limiting and other configurations within the scope of the present disclosure are permissible.

[0078] FIG. 2 illustrates operation in which the Al module 120 works in an inference phase. In this phase, the Al module 120 is pre-trained. In general, the Al module 120, similarly to any Al module, undergoes two or three phases: training, possibly testing, and inference. During the training phase, parameters of the model within the Al module 120 are adjusted (set) based ontraining data. For example, if the Al model comprises an artificial neural network, the weights and / or biases may be adjusted based on the input data. For instance, in supervised learning, pairs of input data and ground truth are fed to the Al model and the parameters are adapted to output result possibly similar to ground truth when the corresponding input data is entered. The input data may be an input image (such as image 110) and the ground truth may be the virtual stain concentration image 125 that should be output when inputting the input image 110. In general, the Al module 120 may implement any known training methodology (supervised or unsupervised). Further detail regarding an exemplary model architecture and training loss structure, and training details for an exemplary Al model 120, are described below in FIGs. 12- 15. The test phase may be performed, but is not strictly required. In the test phase, the quality of inference may be tested. For example, a test data set is input to the Al module 120 and a difference between the output of the Al model (that has been input an input image from the test data set) and the corresponding ground truth is evaluated in order to assess whether the Al model has been sufficiently trained. The inference phase is the phase in which the Al model is used to generate the virtual stain concentration image 125 when entered an input image 110 as described above.

[0079] FIG. 3 is a block diagram illustrating an exemplary virtual staining system 300 in more detail. In this example, the staining control parameter 150 is a parameter 350 specifying color to be applied to the virtual stain concentration image. In FIG. 3, an input image 310 is input to the system 300.

[0080] In this example, the input image is a brightfield image (“BF”) of the biological sample. In bright-field microscopy, white light is transmitted through the biological sample (i.e., illuminated from below) and observed from above. The observation may include capturing such BF image. The contrast in the sample arises from the attenuation of transmitted light in dense areas of the specimen. In module “BF to OD” 315, the brightfield image is converted into an optical density (OD) image. The OD image is input to an artificial intelligence (Al) model 320. In this example, the Al model 320 is a U-Net. The output of the Al model is a virtual stain concentration image 325 which represents concertation of virtual stain at respective locations of the image.

[0081] The U-Net is a specific example of an Al model particularly suitable for the processing of images. In particular, U-Net is a type of convolutional neural network comprising a contracting and expanding path resulting in a u-shaped architecture. The contracting path comprises repeated application of convolutional layer followed by a rectified linear unit (“ReLU”) and max pooling. The expanding path, correspondingly, includes up-convolutionallayers with high-resolution features supplied from the contracting path. However, the present disclosure is not limited to the U-Net or to convolutional networks or to any specific architecture of neural networks. Rather, the Al model may be any model trained or trainable by machine learning. In the example of FIG. 3, the function of the Al model 320 is to detect, in the input image, locations at which a virtual stain should be present and a concentration of the stain in that location. In other words, the Al model 320 is trained to detect the virtual stain which may include location and intensity or concentration of the stain.

[0082] As already mentioned, the virtual stain concentration image 325 in the example of FIG. 3 is an image in an optical density space, in which an intensity of a pixel is correlated with stain concentration (for all pixels). However, it is noted that the present systems, methods, and other aspects of the disclosure, are not limited to BF images as an input and does not necessarily convert the images into OD space. For example, the input image may be a fluorescence image which already behaves similarly to OD space, in that the concentration in a location of each pixel is linearly correlated with pixel’s intensity.

[0083] In this example, the staining control parameter is a color control parameter 350. It may be, for instance a parameter defining RGB (red, green blue) components of the desired stain color. The color parameter is not necessarily specified by RGB components. It may be defined in a different way, such as by using a different color space (YUV, HSV, or any other color space) or by using an identifier associated with specific color values out of a pre-defined color palette or the like.

[0084] The architecture of FIG. 3 is guided by the physical staining process, since the U-Net model is trained with ground truth of the physically stained images. In other words, the U-Net is entered, during learning (training phase) pairs of unstained image and stained image. It is also possible to train the U-Net to generate a second stain type from a first stain type, e.g. by sequential staining. Correspondingly, in inference phase, the second stain type prediction is generated when an image with a first stain type is entered.

[0085] The virtual stain concentration of an unstained input image is then inferred using the trained U-Net. The virtual stain concentration is combined / de-coupled with either globally learned or predefined virtual stain absorption coefficients vector in optical density (OD) domain, in accordance with Beer-Lambert law. The Beer-Lambert law implies that a more concentrated solution absorbs more light than a more dilute solution does. When dealing with RGB images, absorption coefficients vector is three-dimensional (R, G, B). The virtual stain hue can be easily changed at inference time by leaving the Al model weights unchanged and switching only the absorption coefficients e.g. by way of the parameters 350.

[0086] As shown in FIG. 3, a first step 315 involves transforming an input image 310 or image patch (or a plurality of patches) stained with a first stain type to the optical density domain, followed by the U-Net neural network 320 for inferring a (here, single-channel) second type stain concentration map 325 for the image / patch. Subsequently, the concentration map 325 is multiplied by an absorption coefficient vector 350 to derive an optical density virtual stain 335. Absorption coefficient vector could be pre-defined (e.g. set by a user) or learnt directly from the ground-truth patches. For brightfield (BF) WSIs, the absorption coefficient is length three RGB vector 350. Thus, the staining model may be referred to as 3x1 architecture (3 absorption coefficients to one optical density component). The optical density virtual stain 335 is then added 340 to the optical density input (image / patch) 310, which yields an optical density virtually stain-multiplexed image / patch. This patch is subsequently inverse- transformed 360 to a virtually stain-multiplexed bright-field-like output. It is noted that although in the cases where the RGB absorption coefficient vector 350 may be learnt during training, it could be changed at inference time without effecting the inferred concentration map due to the separation of the U-Net model generating the concentration map and the parametrization 350 of the stain.

[0087] In general, the absorption coefficients here can be seen as a color parameter specifying the desired color of the virtual stain. Based on the color control parameter 350, in FIG. 3, an absorption image 355 is generated which corresponds to generating of an image in which all pixels have the value corresponding to the color specified by the color control parameter 350. In the present specific example, even though the color control parameter 350 is selected in the RGB space, it may be converted into the OD space so that it can be easily applied to (mixed with) the concentration image. In other words, the three R, G, B components may be converted into the respective components of the OD space.

[0088] It is noted that the step of conversion into the OD space may not be necessary and that a conversion into another space may be used. Moreover, it is noted that there are various ways of conversion. One of possible options is exemplified in the following. Starting from the Beer- Lambert law for absorption, an idealized model for the RGB intensity at (x,y) is given byy)E£, assuming constant bright field illumination - Ioand constant thickness - d, where c((x,y) are spatial concentrations of stains with corresponding vector absorption coefficients for the R, G, and B channels. The transformation to optical density (OD) can be defined as OD = — log (— ). However, please note that this is only an Uo / example; in practice there may be different ways e.g. to regularize the case where 1=0, where the usual log function would diverge to minus infinity.

[0089] In this way, the color parameter 350 is applied to the virtual stain concentration image 325 in the OD space in the stain configuration module 330. Thereby, a virtual stain image 335 is obtained. It is noted that for implementation efficiency reasons, the size (dimensions) of the input image to be virtually stained 310, the virtual stain concentration image 350, and the absorption image 355 may be the same. The image may be a WSI or merely one or more patches ofthe WSI.

[0090] FIG. 3 further comprises a multiplexing module 340 for combining the image of a biologic sample 310 (in this example after conversion into the OD space) with the virtual stain image 335. The combining corresponds to overlaying (superimposing) the two images. The resulting virtually stained image may be further converted in space conversion module 360 from the OD space into the desired format or space. As exemplified in FIG. 3 such desired space may be the BF format or another format. Thereby, an output image 390 is obtained.

[0091] It is noted that FIG. 3 represents only one specific exemplary implementation. The present disclosure is not limited to such specific example. For example, FIG. 4 schematically illustrates an example in which an artificial model 400 is trained to receive, as an input, an input image 410 and a virtual staining parameter 455 (or a plurality of such parameters) and to output a virtually stained image or a virtual stain image 490. It is noted that the input image 410 (or input images 110, 210, 310 of any examples in this disclosure) may be inputted to the Al model as a whole or in form of patches with a predetermined size. Such patches are obtained by dividing the input image. Patches may be square or circular or of different shapes or sizes. In some aspects, a patch may comprise a shape (e.g., a circle, square, rectangle, or ellipse) centered on or enclosing a feature (e.g., a cell or other feature identified in the input image using one or more feature detection algorithms). In some aspects, a patch may be sized based upon the feature (e.g., the patch may have a radius or diameter based on a detected feature). Input in the form of patches may enable faster processing, allowing for parallelization and potentially resulting in a reduction of complexity and increased processing speed.

[0092] As an option, instead of entering the virtual staining parameter 455, an image generated based on the virtual staining parameter may be inserted, such as an image with all pixels having a desired staining color as described above. In other words, the present disclosure is not limited to any specific format or way of inputting the virtual staining parameter - it does not have to be directly the color components of the desired stain color. Rather, it may be an index into a predefined color selection (e.g. out of a predefined color palette) or another representation of the desired virtual stain color.

[0093] It is noted that although the example in FIG. 3 referred to a color control parameter 350, the present disclosure is not limited to such staining control parameter. Alternatively or in addition, in an exemplary implementation, the staining control parameter is a parameter specifying a filter to be applied to the virtual stain concentration image. It is noted that the filter may include translation of the image back and forth to and from the OD space. In general, the filter may be linear or non-linear, may include contrast and / or brightness or any other image transformation.

[0094] Alternatively, or in addition to the above-mentioned aspects, the staining control parameter may be a parameter specifying at least one of level of quantization (binarization), contrast, brightness (amplification), or smoothness to be applied to the virtual stain concentration image. Thus, the control staining parameter may comprise filter strength for a smoothing filter (e.g. low-pass filter band) or a quantization step (binarization depth) or the like. In addition or alternatively, contrast stretching or morphological opening / closing filters may be used to clean up sporadic virtual staining spots. The use of the filters and / or their parametrization may be set by the staining control parameter.

[0095] The present disclosure and in particular the parametrization of the staining may be suitable for virtual staining applying multiple different stains as illustrated in FIG. 5.

[0096] For instance, the generating 500 of the virtual staining images 190 1 to 190_N includes repetition of (with A being an integer larger than 1):- the obtaining of the virtual stain concentration image 125 1 to 125 N. For that purpose, the input image 510 may be analyzed by a plurality of Al models 120 1 to 120 N that may work as the Al model 120 described with reference to FIG. 2. The models Al models 120 1 to 120_N may differ by being trained to respectively different kinds of virtual staining. Such kinds of virtual staining may include virtual staining representing various IHCs for immune cells, such as CD68, CD3, CD8 and CD20 or the like.- setting a staining control parameter 150 1 to 150_N for the respective stains 1 to N.- the generating 130 1 to 130_N a virtual stain image the plurality, N, of times, thereby obtaining the A virtual stain images 190 1 to 190_N.

[0097] Moreover, the input image 510 of the biologic sample may be combined with one or more of the N virtual stain images 190 1 to 190 N. For example, the input image 510 may be combined with all the N virtual stain images 190 1 to 190 N. For example, the staining control parameters 150 1 to 150 N may be different colors. In such case, the combined image may show superposition of multiple stains of different colors (e.g. colors highlighting respective different antibodies) with the input image 510. In another example, there may be means forselecting, out of the N images 190 1 to 190_N of virtual stain, one or more images that are combined with the input image 510. The means for selecting may be either a parameter that may be set by a human user or a computer, or a graphical user interface enabling a human user to select the number of images to be combined or the like.

[0098] The one or more staining control parameter(s) may be set by a human user or by a computer. For example, said setting a staining control parameter is obtained as an output of an artificial intelligence, Al, including a neural network to which one or more patches of said image of a biologic sample. For instance, said one or more patches are more than one patches. For each of the more than one patches, the Al obtains respective more than one patch staining control parameters. In said setting the staining control parameter, the staining control parameter is obtained as a statistic measure of the more than one patch staining parameters.

[0099] The statistic measure may be, for instance, mean or median over the stain parameters inferred from the patches and / or statistic parameters of higher order.

[0100] One particular advantage of the systems and methods described herein is the ability to independently vary stain control parameters, which can change the presentation of the intensity map (i.e., control what the virtual stain will look like in the combined image, or how the virtual stain appears in an image or layer consisting of the virtual stain.)

[0101] First, virtual stain color can be changed, to enable easier visualization by humans in the context of other colors, overlaid images, or other real or virtual stains. For example, an IHC image may display actual intensity data for PD-L1 in brown, and a virtual stain for CD68 in magenta. Alternatively, the same IHC image could display the virtual stain for CD68 in light blue. Furthermore, a second virtual stain for CD3 could be displayed in magenta with the CD68 virtual stain in light blue. Flexibility of display color can be important for visualization because different biomarkers are present at different levels (intensities) in different tissues, and also, different biomarkers have different distributions in cells, e.g., nuclear, cytoplasmic, or membrane localization, or localization to organelles or extracellular structures. Enabling the color of the virtual stain to be changed can ease visualization of complex structures or combinations of IHC stains, H&E stains, other cellular or tissue stains, or inferred biomarker data.

[0102] Within each stain, one or more stain control parameters can be adjusted, including the following set of examples. First, the virtual stain intensity can be changed, making the stain brighter or dimmer with respect to the rest of the image. This can be useful to adjust the appearance of the stain in regions of the tissue that have high or low levels of staining, to either highlight the background structures or the virtual stain. Second, the virtual stain can have athreshold applied to it, such that the virtual stain can be converted to a binary mask. This can be useful to mark cell identity (e.g., “CD4+ cells”) or to exclude certain cells or regions from further analysis. Third, the thresholding function can enable operations such as dilation or erosion of the binary mask. Erosion and dilation can be used to reduce noise in an image, by excluding regions of virtual stain that are small or fragmented. Fourth, control of the stain parameters can enable digitization, converting a continuous virtual stain distribution into a set of discrete values. For example, this could be useful to categorize individual cells or parts of images as no-expression, low expression, medium expression, and high expression of a certain biomarker represented by the virtual stain. This is similar to the situation when pathologists are working with real IHC images, where the images can be scored in a discrete manner, as “0, 1+, 2+, 3+.” Fifth, adjusting the stain control parameters enables taking the negative of the stain data, which can simulate a fluorescence image (light virtual stain on dark background) instead of a brightfield image of color or shade on a light background. For example, the virtual stain layer or channel can be displayed side-by-side with the original image, or multiple virtual stain images can be shown as grayscale or binary images side by side with the original IHC or H&E image, as well as being displayed as combined with the original image. Sixth, the stain control parameters can be adjusted to enable a nonlinear form of contrast enhancement. For example, a sigmoid function can be applied to the virtual stain image in optical density space, to expand or compress a part of the dynamic range. This could be useful in cases where the virtual stain represents expression of a biomarker (such as CD68), where the user wishes to highlight or suppress differences in the expression of the biomarker. For example, in some cases, it may be useful to have the virtual stain uniformly represent all cells expressing a biomarker (e.g., to mark cell identity in a discrete way as “CD68 + cells”). In this way, the stain control parameters can be adjusted and viewed by a user (e.g., pathologist) to easily and more uniformly mark cells expressing a biomarker that correlates to cells that are not of interest, enabling the user to focus on other cells or parts of the image.

[0103] In addition to the above parameters, other image processing operations may be applied to the virtual stain image, to ease visualization, enable multiplexing of multiple stains or image layers, or potentially, enable the virtual stain layer to be a more effective input into further machine learning models. The ability to manipulate the virtual stain layer independently of the image allows the user more flexibility of data analysis and display.

[0104] As mentioned above, the present disclosure provides, in some embodiments, virtual stain multiplexing and controlling virtual stain color. An Al-based virtual stain multiplexing model architecture with an inherently decoupled representation for the intensity map (e.g.antibody concentration obtained by the Al model 320) and the color 350 of the virtual stain in the Optical Density domain is shown in FIG. 3. The decoupled architecture is also shown in FIG. 2 and FIG. 5 and enables controlling the visual representation of virtual multiplexed staining at inference time (for instance while viewing / processing images), such as changing the virtual stain color, intensity, opacity and / or contrast, as well as taking the negative of the inferred virtual stain e.g. to simulate fluorescent staining. Such manipulations may allow multiplexing the outputs of multiple virtual stain models (as shown in FIG. 5) on top of the same input image 510, as well as optimizing virtual stain representation according to pathologist preferences or the needs of a specific task. It is noted that the present disclosure is not limited to digital pathology and may be applied in any biology / biochemistry domain, e.g. for experiments in biological or medical experiments.

[0105] To facilitate such representation according to pathologist preferences, a device or a method may be provided for generating a graphical user interface, GUI, the GUI comprising an input means for a user (such as the pathologist or the like) to set the staining control parameter; and for controlling a display to display the GUI. The display may be a part of the device or may be external to it. Such GUI 600 is illustrated in FIG. 6, including an area 610 enabling selection of one or more parameters. The area 610 may include one or more of:- a button which when acted upon (e.g. clicked by a mouse, a key or the like) causes displaying of a parameter selection area.- a parameter selection area with a name of a parameter and a GUI element enabling selection of a value for the parameter. Such selection may be out of displayed options or by entering the value using a keyboard or another means. For instance color, transparency, contrast, filtering or the like may be settable.

[0106] Alternatively or in addition, the GUI 600 may further comprise an area 620 for displaying the image of the biologic sample and an area 630 for displaying the virtual stain image in response to receiving an input via said input means. The virtual stain image may be displayed overlaid on the image of the biologic sample. It is noted that FIG. 6 is only a schematically represented example. It is not necessary to provide all three areas 610, 620, 630. Any one or more of these may be provided in the GUI. The GUI may further comprise elements controlling the resolution (zooming in / out) and / or scrolling of the virtual stain image and / or the image of the biologic sample. Further control elements may be provided.

[0107] In addition to input means 610 for a user to set the staining control parameter, in an exemplary implementation, the GUI further includes a widget that allows a user to select said N virtual stain concentration images among L virtual stain concentration image, L being equalto or larger than N. The GUI further includes a widget for each of the N virtual stain concentration images, respectively specify said control parameter. In general, there may be one or more parameters selectable by the widget and controlling the generation of the multiplexed virtual stain. Such parameter may include color, filter, quantization, concentration, transparency, or the like. After the selection, the GUI may be configured to display said N virtual stain concentration images superimposed on the image of the biologic sample. On the other hand, the present disclosure is not limited to the displaying the N virtual stain concentration images in superposition. The GUI may display the N virtual stain images respectively combined with the image of the biologic sample and displayed side-by-side horizontally and / or vertically. In general, the GUI may provide any representations of a selection of one or more virtual stains superimposed onto the image of the biologic sample. Further functionality may include user configurability of a displaying style, size of displaying areas of the displayed images, arrangement of the displaying areas, navigation tools such as zooming, shifting left / right / up / down of the image displayed in the displaying area, inserting marks, aligning the displayed images, or the like.

[0108] The present disclosure may be particularly suitable for predicting and visualizing multiple virtual stains on top of a single (e.g. whole-slide) image through an Al model. Accordingly, any color could be applied as virtual staining multiplexing regardless of the color of the ground-truth IHC. Moreover, the intensity of the virtual stain relative to the input image or other virtual stains multiplexed together could be changed. Several IHC could be multiplexed (visualized) on top of a single IHC. Inferred virtual stain can be transformed by taking its negative to simulate fluorescent staining. The de-coupling of the antibody concentration from its color creates a 2D single layer intensity image (as opposed to RGB image) which open possibilities to use the image as a binary mask, antibody heatmap or change its transparency while used as a virtual multiplexing.

[0109] In summary, the above-described examples enable changing color and / or other stain characteristics of virtual staining after the actual virtual staining model is trained. While for explanatory reasons, color (hue) has been described as a parameter for staining, the present disclosure allows for any other parameters. For example, the above-mentioned GUI may provide means for changing stain concentration (e.g. by a virtual slider or direct concentration setting parameter or the like in the GUI). This enables to amplify or reduce the visibility of the stain. In addition or alternatively, contrast adjustments may be enabled. A user may thus visualize a stain to fade-in or out within (superimposed on) the input image.

[0110] Further effects achievable by the parametrization and the corresponding visualization include generating a negative of the stain image (e.g. to imitate fluorescence) or binarizing (indicating for each pixel of the input image whether or not it belongs to stain). It is noted that when imitating fluorescence (or in other cases), the visualization of the image of the biologic sample may be also be adapted before or after combining with the virtual stain. The colors may be predefined specifically to imitate different staining approaches. For instance, a model to predict a CD68 macrophages antibody for a PD-L1 (another antibody) stained slides may be used. Thus may help in specific applications to distinguish between a tumor cell and immune cell, of which both can be stained with PD-L1 but only immune cells will be stained with magenta (CD68 positive). Alternatively or in addition, a virtual stain layer may target CD3 antibody (lymphocytes) or another marker. The corresponding virtual stain may then be added in a different color. The present systems and methods may be used to imitate any staining approach(es) applied in biology, biochemistry, medical applications, digital pathology or other fields involving biologic samples.

[0111] By applying multiple virtual staining models (Al models), multiple different stains may be generated and parametrized. Then, virtual multiplexing of different stainings may be performed and visualized. An example may be lymphocyte and macrophages staining or any other types of staining.

[0112] Although GUI is mentioned above as means for visualizing the results of the virtual staining, the present disclosure is not limited thereto. The results may be stored in a memory or any volatile or non-volatile storage (e.g. in a file) for later use by a human or a machine.

[0113] The present disclosure provides apparatuses such as those exemplified in FIGs. 1-5. In addition, the present disclosure also provides the corresponding methods. FIG. 7 shows an exemplary method 700 for generating virtual staining of an image of a biologic sample. The method 700 comprises a step 710 of obtaining a virtual stain concentration image including applying an Al model to the image of a biologic sample. The method 700 further comprises setting 720 a staining control parameter and generating 730 a virtual stain image by processing the virtual stain concentration image according to the staining control parameter. The method 700 is a computerized method, i.e. a method that may be performed on a generic or specialized computer architecture.

[0114] As already mentioned above, the staining control parameter may be a parameter specifying color to be applied to the virtual stain concentration image, a parameter specifying a filter to be applied to the virtual stain concentration image, or a parameter specifying at least one of level of quantization, contrast, brightness, or smoothness to be applied to the virtual stainconcentration image, or other kind of parameters. The virtual stain concentration image may be an image in an optical density space, in which an intensity of a pixel is correlated with stain concentration.

[0115] The method 700 may further comprise the combining 740 of the image of a biologic sample with the virtual stain image. The image of the biologic sample is an image input to the virtual staining process. It may be referred to also as original image and it may be an unstained image (although in some implementations it may be also an already stained image or the like). An arrow 750 indicates that the method may include repetition of the obtaining virtual stain concentration image; setting a staining control parameter; and the generating a virtual stain image a plurality, TV, of times, thereby obtaining N virtual stain images. The step 740 may correspondingly include combining the image of a biologic sample with the N virtual stain images. The 750 repetition of the obtaining 710 stain concentration image K times includes application of N different Al models trained to generate respective N different stain concentration images imitating staining respectively different antigens. However, the different staining types are not limited to different antigens and may include staining different biological or biochemical targets (e.g., H&E as mentioned above or other staining approaches). It is noted that in FIG. 7, the repetition in some exemplary embodiments includes N different independent virtual staining stages (that may be performed in series or in parallel), and combining of the respective obtained virtual stains by superimposing them onto the image of the biologic sample. This corresponds to the functional structure representation in FIG. 5. However, the present disclosure is not limited to such an approach, and it may imitate sequential staining. For example, the Al models may be trained respectively to add a stain on previously added one or more stains of a different kind.

[0116] The combining 740 may comprise color processing of the image of a biologic sample and setting the staining control parameter, so as to imitate result of a fluorescence-based staining. In some embodiments, said setting 720 of a staining control parameter is obtained as an output of an Al including a neural network to which one or more patches of said image of a biologic sample is inputted. Such Al may be trained, for instance, together with or as part of the model predicting the concentration map (parts of the model may be shared, with separate "heads" for predicting concentration and color. Alternatively, the control parameter may be trained using a dedicated ground truth. For example, the Al model may be trained to make virtual stain stronger in areas where other stains are stronger and weaker where other stains are weaker, to preserve relative contrast. In other words, the Al model may be trained to make virtual stain strength proportional to the stain obtained by a physical (not virtual) staining.

[0117] For example, said one or more patches are more than one patches. For each of the more than one patches, the Al obtains respective more than one patch staining control parameters. In said setting 720 the staining control parameter, the staining control parameter is obtained as a statistic measure of the more than one patch staining parameters.

[0118] The method 700 may further comprise (not shown in FIG. 7) generating the GUI as described above with reference to FIG. 6.

[0119] The methods may be implemented as a computer program or a part of a computer program stored on a non-transitory medium and including code instructions which, when executed on one or more processors, causes the one or more processors to perform steps of the method(s).

[0120] FIG. 8 illustrates an exemplary hardware structure of an apparatus 800 for generating virtual staining for an image of a biologic sample. The apparatus comprises processing circuitry 810 that is configured to obtain the virtual stain concentration image using the Al model on the biologic sample image, set the staining control parameter, and generate the virtual stain image according to the staining control parameter. The processing circuitry 810 may be configured by a program loaded from a memory 820 over a bus 890 interconnecting the processing and the memory and possible further parts of the apparatus 800. FIG. 9 shows the functional modules of such program stored in the memory 820. An Al model 910 configures the processing circuitry 810 to obtain the virtual stain concentration image. A parametrization module 920 configured the processing circuitry 810 to obtain the param eter(s) of the virtual stain to be generated. A virtual stain generation module configures the processing circuitry 810 to generate the virtual stain based on the parameter(s) and the virtual stain concentration image. A multiplexing module configures the processing circuitry 810 to multiplex the generated virtual stain with an input image.

[0121] Moreover, the apparatus 800 includes an output interface 880, configured to output the virtual stain image generated by the processing circuitry 810. The output may be transferred by means of the bus 890 for example to the memory 820 or to another storage part of the apparatus 800 or external thereto.

[0122] FIG. 10 illustrates a system including the apparatus 800. The apparatus 800 may further include a display control 830 that may control a display device 1060 to display the generated image output via the interface 800. The display device 1060 may be part of the apparatus 800 or be external to it. The display control 830 may be also configured to display a GUI (e.g. as described with reference to FIG. 6 or the like) that may be also generated by the processing circuitry 810. The apparatus may comprise a communication interface 840 whichmay be configured to transmit and / or receive signals to / from the apparatus 800. This may be for instance a wireless LAN, Bluetooth, cellular or other wireless or wired interface that connects the apparatus to a network 1095 or to another device or over a network to another device. The network may be a LAN or Internet or any kind of network. The communication interface 840 may be used to receive for example stained and / or unstained input images, parameters or the like, and / or to transmit the output images generated by the processing circuitry 810 to other devices. In FIG. 10, the network connects the apparatus 800 to a slide scanner 1070 that may provide e.g. stained and / or unstained images.

[0123] An operation input interface 850 may be configured to receive human user input over an input device 1080, such as input of the staining control parameters and / or selection of input / output images, commands or the like. The input device 1080 may be represented by a touch screen, keyboard, control buttons, mouse, or the like.

[0124] It is noted that the apparatus 850 is only exemplary. The apparatus may have a different architecture. The processing circuitry 810 may be any one or more hardware pieces such as one or more general-purpose or special-purpose processor, FPGAs, ASICs, or electronics. The memory 820 may be volatile and / or non-volatile. The display control 830 may be a hardware portion wired or programmed to control displaying on a display. The present disclosure is not limited to any specific platform or application style. For example, the above- mentioned GUI may be implemented as a browser based application or as a standalone program. It may be stored / operated locally (on a computer) or on a server or in a cloud or the like.

[0125] FIG. 11 shows an example of a virtually stained image 1110 in 40x magnification. It is a virtual stain (see also region pointed to by arrow 1101 highlighted by the coloring) of a P40 IHC, where the input was a Hematoxylin only stained image 1120.Examples

[0126] Example 1. Training an Exemplary Al Model for Virtual Staining.

[0127] Overview

[0128] FIG. 12 is a block diagram illustrating the model architecture and training loss structure for an exemplary implementation of the virtual staining module with separated stain generation and parameter-based stain processing with parameter controlling color of the virtual stain. Stain 1 was PD-L1 22C3 pharmDx and stain 2 was CD68. FIG. 12(a) is a 3x1 model architecture guided by physical staining process; virtual stain concentration is inferred using a U-Net is combined with globally learned virtual stain absorption coefficients vector in optical density (OD) domain, in accordance with Beer-Lambert law. FIG. 12(b) shows the virtually stained (output) and sequentially stained ground truth (GT) patches are used to compute aweighted combination of MSE and BCE losses. As shown by FIG. 12(c), with the 3x1 architecture, the virtual stain hue can be easily changed at inference time.

[0129] Background and Protocols

[0130] Sequential staining is a method for adding an additional immunohistochemical (IHC) stain (stain 2), to tissues which were already stained (stain 1) Moreover, according to the Beer- Lambert law for absorption of light passing through a medium, the optical density of stained tissue is linearly dependent on the local concentration and ab-sorption coefficient spectrum of chromogen.

[0131] A model architecture (FIG. 12(a)) which corresponds to these physical properties of sequentially stained tissues was designed. First, the model was optimized using optical density. Second, learning the virtual stain concentration map is separated from learning the virtual stain color, determined by the absorption coefficient vector.

[0132] The first step of the model involves transforming an input patch stained with stain 1 to the optical density domain, followed by a U-Net neural network for infer-ring a singlechannel stain 2 concentration map for the patch. Subsequently, the con-centration map is multiplied by a learned absorption coefficient vector to derive an optical density virtual stain. For brightfield (BF) WSIs, the absorption coefficient is length three RGB vector, hence this model is referred to as a 3x1 architecture. The optical density virtual stain is then added to the optical density input patch, which yields an optical density virtually stain-multiplexed patch. This patch is subsequently inverse-transformed to a virtually stain-multiplexed bright-field-like output. It is important to note that although the RGB vector is learnt during training, it is a global parameter vector and not dependent on the input patch.

[0133] The physical properties of sequential staining have also informed the development of the loss function for model optimization, as depicted in FIG. 12(b).

[0134] Given the additive nature of sequential / virtual staining, semantic masks, denoted as MaskGT / Mask0Ut, are constructed for the ground-truth and output patches, respectively. These masks are obtained by taking the difference between the optical density ground-truth / output patch and the input patch; any pixel with an optical density difference exceeding a threshold value is considered positive for sequential / virtual stain. The semantic masks are then used to compute a semantic pixel-wise binary cross entropy loss (BCE) between the corresponding ground-truth and output patches.

[0135] The semantic loss is combined with a mean squared error (MSE) loss between the optical density ground-truth and output patches. To focus the training on relevant differences,MaskGTis used as a mask for the MSE loss. The parameter a allows for tunable weighting of the relative strengths of these two loss functions, resulting in a combined loss function:Loss = MSE(output, GT; MaskGT) + aBCE(Mask0Ut, MaskGT) + 0(— v)v2

[0136] Where the last term is a regularization term, reflecting the additive nature of sequential staining, keeping the inferred virtual stain output positive, 0 is the Heaviside step-function and v denotes the virtual stain output.

[0137] Model performance is evaluated using the intersection over union (IOU) metric between MaskGTand Maskout, which is used for both hyperparameter tuning and computational evaluation.

[0138] FIG. 13 shows an exemplary workflow for dataset preparation. In short, 200 NSCLC slides were stained with PD-L1 IHC 22C3 PharmDx (GE006). 49 selected tissues were sequentially stained with CD68 PG-M1 (GA613) and visualized with Envision FLEX HRP Magenta chromogen (GV925) on top of the PD-L1 IHC 22C3 pharmDx. An additional section was prepared for each case, stained with hematoxylin and eosin (H&E). All stained slides were scanned using a high-resolution scanner at 40x magnification. The sequentially stained wholeslide images (WSIs) were aligned with their matching WSIs to near pixel-perfect alignment. The H&E WSI were also aligned with their matching WSI pair using rough global alignment.

[0139] FIG. 14 shows an example of tumor annotation using an Al model trained according to the methods described herein. Each case was annotated by the study pathologist, marking tumor and tumor-adjacent regions. Tumor identification was done by aligning the H&E and PD-L1 IHC 22C3 pharmDx stains. Regions were qualitatively classified according to common pathology practice: negative PD-L1 tumor (0 blue), weakly positive PD-L1 tumor (1+ green), and strongly positive PD-L1 tumor (2+, 3+ red). lOOOxlOOOpx regions from 26 WSIs, where macrophages were difficult to distinguish from tumor cells, or where macrophages were identified as infiltrating PD-L1 positive tumor areas, were annotated as validation-set regions for method evaluation and excluded from model training (brown).

[0140] FIG. 15 shows exemplary parameters of an optimized Al model. The model was optimized using Adam optimizer and a reduce-on-plateau learning-rate scheduler, with early stopping. All hyperparameters were manually tuned; the hyperparameter choice used for cellclassification evaluation was guided by visual inspection of the produced virtual stain by pathologist 1 and validation set loU. The additional initialization gain was required due to input transformation to optical density. Scheduler and early stopping were based on validation loU metric. Data sampling was balanced using the region annotations, ensuring each batch included the same number of patches from each region class.

[0141] The parameters shown in FIG. 15 are non-limiting. In other aspects of the systems and methods described herein, alternative parameters may be used. Indeed, any of these parameters may be manually tuned to create different virtual stain outputs, while also minimizing difference from the ground truth. The present systems and methods may be used to create one or more virtual stains that are inferred by the model, but the utility of the virtual stain or stains can depend on the goals of the user or the use case. For example, the particular parameters shown in FIG. 15 were manually tuned to generate a virtual stain for CD68 that was useful for a pathologist 1 to identify macrophages; however, the same set of hyperparameters was used to generate the virtual stain for CD3. A different virtual stain (e.g., for CD4, CD8, or any other cellular marker) may require different hyperparameter tuning according to the preferences of the user, in combination with monitoring the loU metric to ensure the model inference is close to the ground truth.

[0142] Example 2. Qualitative Evaluation of Virtual Stain Multiplexing

[0143] Overview

[0144] To evaluate the performance of the model trained in Example 1, a qualitative assessment was conducted using whole slide images (“WSIs”).

[0145] Background and Protocols

[0146] Several representative examples of model outputs generated on challenging validation set regions in FIG. 16 (left), alongside the corresponding input and sequentially stained ground truth patches. Notably, the virtual CD68 staining demonstrated a high degree of visual consistency with the actual CD68 stain, as evaluated by an expert pathologist. Notably, virtual staining using the present method effectively highlighted macrophages, as evidenced by the staining of most macrophages in the validation set regions.

[0147] FIG. 16 (right) presents several examples of macrophages correctly identified by the

[0148] model, despite the high variation in macrophage morphology and staining patterns. Not all these macrophages were correctly identified by Pathologist 2 without the model’s assistance.

[0149] Example 3. Ablation Tests.

[0150] Overview

[0151] Qualitative and quantitative ablation tests were conducted using the model trained in Example 1.

[0152] Background and Protocols

[0153] FIG. 17 shows a comparison of a ground truth patch and the corresponding output of a full model trained with combined MSE and BCE loss, to the output of an ablation model thatis solely trained using MSE loss. The comparison shows that the outputs of the ablation model possess two distinctive features in contrast to those of the full model. Specifically, the ablation model acquires faint and non-specific background staining artifacts that are present in the ground-truth patches but are not present in input patches. Nevertheless, these non-specific stains do not appear in the outputs of the full model. Moreover, the outputs of the ablation model exhibit fainter overall virtual staining in comparison to the outputs of the full model and the ground truth patches. We observed that for cells with explicit membranal staining, the full model tends to stain the entire cell cytoplasm, while the ablation model often displays partial staining patterns, similar to those found in the ground truth patches.

[0154] Several ablation tests were conducted to investigate the contributions of different parts of our proposed method.

[0155] Each test was repeated five times, with different random seeds. Validation loU values were averaged over last 10 epochs of each run, after convergence. To ensure stable evaluation, the learning schedule was fixed and early stopping disabled. The learning rate schedule was 0.0002 for 167 epochs and then linearly reduced by a factor of 100 over additional 84 epochs. Although removing semantic loss improved loU, the visual qualities of the resultant stain were difficult for pathologists to interpret. Removing the OD transformation has a strong effect, which is amplified by replacing the 3x1 architecture with a simple U-Net. The results of this study are summarized by Table 1 below.Architecture Validation loU mean (std)Baseline 0.617 (0.0015)No semantic loss (o = 0, unmasked MSE) 0.631 (0.0015)No OD transform 0.608 (0.002)No 3x1 architecture (simple U-Net) 0.618 (0.0012)No OD transform & No 3x1 architecture 0.58 (0.0023)Table 1. Summary of quantitative ablation test results.

[0156] Example 4. Validation Studies.

[0157] Overview

[0158] The aim of this evaluation was to directly test the potential of the suggested model as an assistive tool for pathologists in detecting macrophages. This evaluation scheme differs from a pixel-level evaluation, as it takes into account that the virtual stain may not always match the sequential ground truth stain pixel for pixel, but may still accurately stain the correct cells.

[0159] Background and Protocols

[0160] To carry out this evaluation, individual cells from 13 regions of interest, selected from the validation set, were annotated as macrophage / not-macrophage. Pathologist 1 generated cell-level ground-truth annotations based on matching pairs of PD-L1 & PD-L1 + sequentially stained CD68. Pathologist 2 first generated baseline annotations, based on PD-L1 WSI only. Then, Pathologist 2 generated annotations based on matching pairs of PD-L1 & PD-L1 + virtually stain multiplexed CD68, viewed side-by-side. These annotations were then compared to the ground truth annotations by Pathologist 1. The model performance was evaluated by measuring the change in Pathologist 2’s precision, sensitivity, and accuracy when assisted by the model compared to when not using the model.

[0161] Confusion matrices comparing Pathologist 2’s annotations are presented in FIG. 18. The addition of virtually stained multiplexed CD68 improved the performance of Pathologist 2’s annotations, increasing precision and sensitivity from 0.44 and 0.32 to 0.67 and 0.79, respectively, while specificity remained unchanged at 0.92. Paired McNemar’s test (McNemar, 1947) yielded a p-value of less than IO'30.

[0162] Example 5. Virtual Staining for CD3

[0163] Overview

[0164] FIGs. 19A-16D illustrate the result of model training to infer a virtual stain for CD3 through successive epochs.

[0165] Background and Protocols

[0166] Sixteen example regions of interest are shown from WSIs stained for PD-L1, illustrating different levels of staining and cellular distributions. Column 1 labeled PD-L1 shows the input data, comprising PD-L1 IHC staining with PD-L1 IHC 22C3 PharmDx assay (GE006). Column 2 labeled Seq / GT shows the sequentially stained ground truth (PD-L1 IHC 22C3 pharmDx) IHC sequentially stained with an antibody to CD3 and visualized with Envision FLEX HRP Magenta chromogen (GV925), where the magenta color indicates CD3+ cells such as lymphocytes. (The magenta color is represented by greyscale in the figures provided in the present application.) Column 3 labeled Output shows the output of the VS model with the VS overlaid (for epoch 0, pre training, this image is identical to the input). Column 4 labeled binary GT shows a binarized version of the CD3 stain, derived from the image data in columns 1 and 2. Column 5, VS only, shows the virtual stain output as a grayscale image. This grayscale image output can be arbitrarily colored and combined with other VS images in different colors to enable multiplex virtual staining. Note that as the training progresses, the VS output converges to be similar to the GT, as seen by the similarity of columns 2 and 3 and columns 4 and 5.

[0167] Example 6. Virtual Staining for CD68

[0168] Overview

[0169] FIGs. 20A-C illustrate the result of model training to infer a virtual stain for CD68 through successive epochs.

[0170] Background and Protocols

[0171] FIGs. 20A-C illustrate the result of model training to infer a virtual stain for CD68 through successive epochs. 16 example regions of interest are shown from NSCLC WSIs stained for PD-L1, illustrating different levels of staining and cellular distributions. Column 1 labeled PD-L1 shows the input data, comprising staining with the PD-L1 IHC 22C3 pharmDx assay. Column 2 labeled Seq / GT shows the sequentially stained ground truth (PD-L1 IHC 22C3 pharmDx) IHC sequentially stained with CD68 PG-M1 (GA613) and visualized with Envision FLEX HRP Magenta chromogen (GV925)), where labeling CD68+ cells is meant to assist in the identification of macrophages. (The magenta color is represented by greyscale in the figures provided the present application.) Column 3 labeled Output shows the output of the VS model with the VS overlaid (for epoch 0, pre training, this image is identical to the input). Column 4 labeled binary GT shows a binarized version of the CD68 stain derived from columns 1 and 2. Column 5, VS only, shows the virtual stain output as a grayscale image. This grayscale image output can be arbitrarily colored and combined with other VS images in different colors to enable virtual stain multiplexing.

[0172] Example 7. Additional Validation Studies

[0173] Overview

[0174] An in-depth evaluation of multiplexed virtual staining according to the methods described herein was performed, demonstrating its efficacy in creating virtual staining multiplexing (“VSM”) model using marker CD68 to assist pathologists in detecting macrophages in tumor microenvironment of PD-L1 IHC 22C3 pharmDx-stained NSCLC samples.

[0175] Background and Protocols

[0176] In the sequential multiplexing workflow, illustrated in FIG 21A, a PD-L1 IHC-stained tissue undergoes a sequential staining with an additional CD68 IHC marker applied to the same tissue section. The VSM workflow is illustrated in FIG. 21B. A U-Net convolutional neural network (“CNN”) forms the backbone of our virtual stain generation process. The input to the U-Net is a patch extracted from the PD-L1 stained brightfield (BF) RGB image, converted to optical density (OD) space to align with the Beer-Lambert law. The network predicts a concentration map of the CD68 stain, which is then multiplied by a learned RGB color vector, generating a virtual CD68 stain patch. To produce the final VSM image, the virtual CD68 patchis combined with the input PD-L1 patch in OD space and converted back to RGB space, resulting in a synthetically stained PD-L1 + virtual CD68 image pair.

[0177] The dataset used for training and evaluation is detailed in (Ben-David et al., 2024). The study pathologist reviewed 330 scanned NSCLC whole-slide images (WSIs) stained with the PD-L1 IHC 22C3 assay. From these, 49 cases were curated to cover the full TPS range and include cases with PD-L1 expression levels near clinically relevant thresholds. The selected sections were subsequently restained with the CD68 PG-M1 (GA613) marker and scanned again, to generate sequentially stained PD-L1 + CD68 WSI pairs. Both WSIs in each pair were globally aligned with RANS AC-optimized Euclidean transform, using ORB features (Rublee et al., 2011), followed by DIS optical flow (Kroeger et al., 2016) to enable pixel-accurate training, with the sequentially stained PD-L1 + CD68 WSI serving as a measurement-based ground truth. To focus the model on clinically relevant areas the study pathologist annotated tumor regions on each sequentially stained WSI pair for use during model training. In addition, six particularly challenging regions (ROIs, FIGs. 24-29) were selected for cell-level evaluation based on the difficulty of distinguishing macrophages from tumor cells or the presence of macrophages infiltrating PD-L1 -positive tumor areas, making them critical for assessing the efficacy of the proposed VSM method.

[0178] The main purpose of this study was to evaluate the efficacy of the VSM method rather than the performance of a specific model. A leave-tissue-out methodology was adopted, in view of the challenging nature of the regions chosen for evaluation. A separate model was trained for each evaluation region, excluding the entire WSI containing that region from the training data. This prevented any tissue-specific overfitting, while maximizing overall training set diversity. The training hyperparameters were kept consistent across all models, using the same settings reported in the ablation tests section of (Ben-David et al., 2024), ensuring comparability across results.

[0179] As previously reported (Ben-David et al., 2024), inspection of model prediction by the study pathologist revealed the VSM patterns follow closely true sequential staining. However, to quantitatively evaluate macrophage detection performance, the study pathologist provided cell-level annotations for each of the six evaluation regions, marking cells as CD68- positive (CD68+) or CD68-negative (CD68-) based on the paired and aligned PD-L1 and sequentially stained PD-L1 + CD68 WSIs (FIG. 22B). These annotations served as the groundtruth reference for model evaluation. With the aid of sequential CD68 staining, many cells could be confidently labeled by the study pathologist.

[0180] Three additional certified pathologists, P1-P3, independently performed two rounds of cell-level annotation for each region. To ensure the same cells were annotated by all pathologists, the cells in each region were pre-marked, and set to an “unclassified” class. In the first round, they classified cells using only the PD-L 1 -stained WSI (FIG. 22A). After a washout period of more than six months to minimize recall bias, they repeated the annotation task using pairs of PD-L 1 and VSM CD68 WSIs (FIG. 22C).

[0181] Since the task was defined as macrophage detection task, cells marked as “unknown” by P1-P3 were treated as CD68-, as the pathologists did not positively identify them as macrophages. To assess the robustness of this assumption, an alternative analysis was conducted where unknown cells were instead treated as CD68+, yielding qualitatively similar results. To ensure that the reference used for evaluation was reliable, cells annotated as “unknown” by the study pathologist were excluded from macrophage detection performance metrics.

[0182] Inter-pathologist agreement was measured using Fleiss’s Kappa for overall agreement across the three annotators and Cohen’s Kappa for pairwise agreement between each pathologist pair (Fleiss and Cohen, 1973). Performance metrics were computed for each of PIPS, comparing their annotations to the ground truth annotations by the study pathologist. Consensus labels were derived from majority -voting among P1-P3, full-consensus labels were obtained for cells where the annotations of P1-P3 were the same. Given the class imbalance present in the dataset, Fl scores were used as the primary performance measure, supplemented by precision, recall, and specificity.

[0183] The analysis included a total of 2,304 annotated cells, of which 457 were marked as CD68 positive in the ground truth annotations provided by the study pathologist. FIG. 22D summarizes the predictions of pathologists P1-P3 for a subset of 96 cells where the consensus PD-Ll-only annotation indicated CD68 positivity. Among these 96 cells, only 35 were confirmed as macrophages in the ground truth annotations by the study pathologist, as indicated by dots in the figure. Notably, full consensus among all three pathologists was reached for only six cells, and of these, just one was confirmed as a macrophage by the ground truth; four cells were annotated as CD68-, and one classified as “unknown.”F le is s ’ C ohe n5sMethodPl vs. P2 vs. P3 Pl vs. P2 Pl vs. P3 P2 vs. P3PD-L1 -0.11 0.07 -0.03 0.01VSM 0.62 0.79 0.51 0.58Table 2. Inter-pathologist agreement measured by Fleiss’ and Cohen’s kappa for PD-L1 and virtual staining multiplexing (VSM).

[0184] Table 2 presents Fleiss’ s and Cohen’s kappa measures of inter-rater agreement for PIPS, both with and without the assistance of VSM CD68. Agreement was significantly improved with the aid of VSM: Fleiss’ kappa increased from -0.1 (indicating extremely poor agreement) with PD-Ll-only annotations to 0.62 (substantial agreement) with VSM. The number of cells with full consensus annotations as CD68+ also increased markedly, rising to 463 with the use of VSM. Even the pair of Pl and P2, who demonstrated the highest agreement among the pairings, showed only slight agreement with PD-Ll-only annotations (Cohen’s kappa = 0.07). With VSM, all pairwise agreements improved to at least moderated levels (Cohen’s kappa > 0.5). The effectiveness of pathologists in detecting macrophages was further analyzed using VSM CD68. FIG. 23 provides detailed comparisons of performance with and without VSM. FIGs. 23A-C present receiver operating characteristic (ROC) plots for P1-P3, displaying recall and 1 -specificity across six evaluation regions.

[0185] Without the assistance of VSM, Pl and P2 exhibited low recall (0.06, 0.08) and high specificity (0.94, 0.97) across all evaluation regions, while P3 achieved higher recall (0.59) but at the expense of reduced specificity (0.66). The introduction of VSM substantially enhanced recall for Pl and P2 (0.58, 0.65), albeit with some decreases in specificity (0.9, 0.87). For P3, both recall (0.75) and specificity (0.76) improved across all evaluation regions with the aid of VSM.

[0186] FIGs. 23D-F focus on the performance of consensus annotations. FIG. 23D illustrates the ROC plot for consensus annotations, showing a similar trend to the individual performances of Pl and P2, with low recall (0.077) and high specificity (0.97) in the absence of VSM. VSM led to much higher recall (0.65), accompanied by a decrease in specificity (0.88). FIG. 23E depicts precision-recall curves, highlighting notably improved pooled recall (0.65) and precision (0.65), with precision increases of 2-3 times for certain regions. Finally, FIG. 23F summarizes Fl scores, which consistently improved with VSM. The pooled Fl score for consensus annotations showed a five-fold improvement with the aid of VSM, rising from 0.13 to 0.65.

[0187] Comparing majority-vote consensus annotations to full-consensus annotations revealed additional insights. Without VSM, full-consensus performance was notably low, with an Fl score of 0.012, recall of 0.006, and precision of 0.2. VSM significantly improved these metrics, raising them to 0.71, 0.72, and 0.69, respectively.

[0188] Unassisted macrophage detection showed agreement levels even lower than those observed in clinical scoring tasks such as HER2 and PD-L1, where inter-pathologist agreement is already below ideal thresholds (Robbins et al., 2023; Han et al., 2022). Unlike such scoring tasks, where pathologists’ annotations often serve as the only ground truth, our study leveraged sequentially stained CD68 IHC to provide an independent and more objective ground truth. Comparisons to this ground truth revealed consistently poor unassisted performance across the curated challenging regions.

[0189] Additional data and experimental details related to the examples described herein is available in Ben-David, Oded, et al., “Deep-Learning Based Virtual Stain Multiplexing Immunohistochemistry Slides-a Pilot Study” MICCAI Workshop on Computational Pathology with Multimodal Data (COMPAYL), 2024; as well as in Arbel, Elad, et al. “Evaluation of Virtual Stain Multiplexed CD68 for Macrophage Detection in NSCLC PD-L1 Slides,” Medical Imaging with Deep Learning. 2025, the entire respective contents of each of which (includng any appendices) are incorporated herein by reference.* * *

[0190] In closing, it is to be understood that although aspects of the present specification are highlighted by referring to specific embodiments, one skilled in the art will readily appreciate that these disclosed embodiments are only illustrative of the principles of the subject matter disclosed herein. Therefore, it should be understood that the disclosed subject matter is in no way limited to a particular compound, composition, article, apparatus, methodology, protocol, and / or reagent, etc., described herein, unless expressly stated as such. In addition, those of ordinary skill in the art will recognize that certain changes, modifications, permutations, alterations, additions, subtractions and sub-combinations thereof can be made in accordance with the teachings herein without departing from the spirit of the present specification.

[0191] Use of the terms “may” or “can” in reference to an embodiment or aspect of an embodiment also carries with it the alternative meaning of “may not” or “cannot.” As such, if the present specification discloses that an embodiment or an aspect of an embodiment may be or can be included as part of the inventive subject matter, then the negative limitation or exclusionary proviso is also explicitly meant, meaning that an embodiment or an aspect of anembodiment may not be or cannot be included as part of the inventive subject matter. In a similar manner, use of the term “optionally” in reference to an embodiment or aspect of an embodiment means that such embodiment or aspect of the embodiment may be included as part of the inventive subject matter or may not be included as part of the inventive subject matter. Whether such a negative limitation or exclusionary proviso applies will be based on whether the negative limitation or exclusionary proviso is recited in the claimed subject matter.

[0192] Notwithstanding that the numerical ranges and values setting forth the broad scope of embodiments of the disclosure are approximations, the numerical ranges and values set forth in the specific examples are reported as precisely as possible. Any numerical range or value, however, inherently contains certain errors necessarily resulting from the standard deviation found in their respective testing measurements. Recitation of numerical ranges of values herein is merely intended to serve as a shorthand method of referring individually to each separate numerical value falling within the range. Unless otherwise indicated herein, each individual value of a numerical range is incorporated into the present specification as if it were individually recited herein.

[0193] The terms “a,” “an,” “the” and similar references used in the context of describing aspects of the present disclosure (especially in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. Further, ordinal indicators — such as “first,” “second,” “third,” etc. — for identified elements are used to distinguish between the elements, and do not indicate or imply a required or limited number of such elements, and do not indicate a particular position or order of such elements unless otherwise specifically stated. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (for example, “such as”) provided herein is intended merely to better illuminate aspects of the present disclosure and does not limit the scope of the inventions otherwise claimed. No language in the present specification should be construed as indicating any non-claimed element essential to the practice of the various inventions defined by the claims.

[0194] When used in the claims, whether as filed or added per amendment, the open-ended transitional term “comprising” (and equivalent open-ended transitional phrases thereof like including, containing and having) encompasses all the expressly recited elements, limitations, steps and / or features alone or in combination with unrecited subject matter; the named elements, limitations and / or features are essential, but other unnamed elements, limitations and / or features may be added and still form a construct within the scope of the claim. Specific embodimentsdisclosed herein may be further limited in the claims using the closed-ended transitional phrases “consisting of’ or “consisting essentially of’ in lieu of or as an amended for “comprising.” When used in the claims, whether as filed or added per amendment, the closed-ended transitional phrase “consisting of’ excludes any element, limitation, step, or feature not expressly recited in the claims. The closed-ended transitional phrase “consisting essentially of’ limits the scope of a claim to the expressly recited elements, limitations, steps and / or features and any other elements, limitations, steps and / or features that do not materially affect the basic and novel characteristic(s) of the claimed subject matter. Thus, the meaning of the open-ended transitional phrase “comprising” is being defined as encompassing all the specifically recited elements, limitations, steps and / or features as well as any optional, additional unspecified ones. The meaning of the closed-ended transitional phrase “consisting of’ is being defined as only including those elements, limitations, steps and / or features specifically recited in the claim whereas the meaning of the closed-ended transitional phrase “consisting essentially of’ is being defined as only including those elements, limitations, steps and / or features specifically recited in the claim and those elements, limitations, steps and / or features that do not materially affect the basic and novel characteristic(s) of the claimed subject matter. Therefore, the open-ended transitional phrase “comprising” (and equivalent open-ended transitional phrases thereof) includes within its meaning, as a limiting case, claimed subject matter specified by the closed- ended transitional phrases “consisting of’ or “consisting essentially of.” As such embodiments described herein or so claimed with the phrase “comprising” are expressly or inherently unambiguously described, enabled and supported herein for the phrases “consisting essentially of’ and “consisting of.”

[0001] All patents, patent publications, and other publications referenced and identified in the present specification are individually and expressly incorporated herein by reference in their entirety for the purpose of describing and disclosing, for example, the compositions and methodologies described in such publications that might be used in connection with aspects of the present disclosure. These publications are provided solely for their disclosure prior to the filing date of the present application. Nothing in this regard should be construed as an admission that the inventors are not entitled to antedate such disclosure by virtue of prior invention or for any other reason. All statements as to the date or representation as to the contents of these documents are based on the information available to the applicants and does not constitute any admission as to the correctness of the dates or contents of these documents.

[0195] Lastly, the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to limit the scope of the present invention, which isdefined solely by the claims. Accordingly, the invention claimed herein is not limited to that precisely as shown and described.References

[0196] The entire disclosure of each of the following references is incorporated by reference.Agilent Technologies. PD-L1 IHC 22C3 pharmDx interpretation manual, NSCLC. URL: “www.agilent.com / cs / library / usermanuals / public / 29158_pd-ll-ihc-22C3- pharmdx-nsclc-interpretation-manual.pdf”, 2021.Frederik Aidt, Elad Arbel, Itay Remer, Oded Ben-David, Amir Ben-Dor, Daniela Rabkin, Kirsten Hoff, Karin Salomon, Sarit Aviel-Ronen, Gitte Nielsen, Jens Mollerup, Lars Jacobsen, and Anya Tsalenko. Quantification of her2-low and ultra-low expression in breast cancer specimens by quantitative ihc and artificial intelligence, 2025. URL: “www.biorxiv.org / content / early / 2025 / 01 / 25 / 2025.01.22. 634230”.B. Bai, H. Wang, Y. Li, K. de Haan, F. Colonnese, Y. Wan, J. Zuo, N.B. Doan, X. Zhang, Y. Zhang, and J. Li. Label-free virtual HER2 immunohistochemical staining of breast tissue using deep learning. BME Frontiers, 2022.Vipul Baxi, George Lee, Chunzhe Duan, Dimple Pandya, Daniel N Cohen, Robin Ed- wards, Han Chang, Jun Li, Hunter Elliott, Harsha Pokkalla, et al. Association of artifi- cial intelligence-powered and manual quantification of programmed deathligand 1 (pd- 11) expression with outcomes in patients treated with nivolumabiipilimumab. Modern Pathology, 35(11): 1529-1539, 2022.A. Beck, B. Glass, H. Elliott, J.K. Kerner, A. Khosla, A. Lahiri, H. Pokkalla, D. Wang, I. Wapinski, G. Lee, and V. Baxi. P730 an empirical framework for validating artificial intelligence-derived PD-L1 positivity predictions applied to urothelial carcinoma. In Cancer cells, volume 3728, pages 18-056, Unknown, 2019.Oded Ben-David, Elad Arbel, Sarit Aviel-Ronen, Frederik Heurlin Aidt, Kristopher Ker- sch, Tine Hagedorn Olsen, Daniela Rabkin, Itay Remer, Amir Ben-Dor, Lars Jacobsen, et al. Deep-learning based virtual stain multiplexing immunohistochemistry slides-a pi- lot study. In MICCAI Workshop on Computational Pathology with Multimodal Data (COMP AYL), 2024.K. de Haan, Y. Zhang, J.E. Zuckerman, T. Liu, A.E. Sisk, M.F. Diaz, K.Y. Jen, A. Nobori, S. Liou, S. Zhang, and R. Riahi. Deep learning-based transformation of h&e stained tissues into special stains. Nature communications, 12(1):4884, 2021.Joseph L Fleiss and Jacob Cohen. The equivalence of weighted kappa and the intraclass cor- relation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613-619, 1973.Astri Frafjord, Renate Skarshaug, Clara Hammarstrbm, Branislava Stankovic, Linda T. Dorg, Henrik Aamodt, Per Reidar Woldbsek, Aslaug Helland, Odd Terje Brustugun, Inger Oynebr ten, and Alexandre Corthay. Antibody combinations for optimized staining of macrophages in human lung tumours. Scandinavian Journal of Immunology, 92(1): el2889, 2020. doi: https: / / doi.org / 10.l l l l / sji.12889. URL URL “https: / / onlinelibrary.wiley.com / doi / abs / 10.l l l l / sji.12889”.Gang Han, Michael J Schell, Emily S Reisenbichler, Bohong Guo, and David L Rimm. Determination of the number of observers needed to evaluate a subjective test and its application in two pd-11 studies. Statistics in medicine, 41(8): 1361-1375, 2022.Fred R Hirsch, Murry W Wynes, David R Gandara, and Paul A Bunn Jr. The tissue is the issue: personalized medicine for non-small cell lung cancer. Clinical Cancer Research, 16 (20):4909-4911, 2010.Clifford C. Hoyt. Multiplex immunofluorescence and multispectral imaging: Forming the basis of a clinical test platform for immuno-oncology. Frontiers in Molecular Bio- sciences, 8, 2021. doi: 10.3389 / fmolb.2021.674747. URL: “https: / / www.frontiersin.org / joumals / molecular- biosciences / articles / 10.3389 / fmolb.2021.674747”.Muhammad Talha Imran, Imran Shafi, Jamil Ahmad, Muhammad Fasih Uddin Butt, San- tos Gracia Villar, Eduardo Garcia Villena, Tahir Khurshaid, and Imran Ashraf. Virtual histopathology methods in medical imaging-a systematic review. BMC medical imaging, 24(1):318, 2024.Janani S Iyer, Dinkar Juyal, Quang Le, Zahil Shanis, Harsha Pokkalla, Maryam Pouryahya, Aryan Pedawi, S Adam Stanford-Moore, Charles Biddle-Snead, Oscar Carrasco-Zevallos, et al. Ai -based automation of enrollment criteria and endpoint assessment in clinical trials in liver diseases. Nature Medicine, 30(10):2914-2923, 2024.Ansh Kapil, Armin Meier, Keith Steele, Marlon Rebelatto, Katharina Nekolla, Alexander Haragan, Abraham Silva, Aleksandra Zuraw, Craig Barker, Marietta L Scott, et al. Do- main adaptation-based deep learning for automated tumor cell (tc) scoring and survival analysis on pd-11 stained tissue images. IEEE Transactions on Medical Imaging, 40(9): 2513-2523, 2021.Leeat Keren, Marc Bosse, Steve Thompson, Tyler Risom, Kausalia Vijayaragavan, Erin McCaffrey, Diana Marquez, Roshan Angoshtari, Noah F Greenwald, Harris Fienberg, et al. Mibi-tof: A multiplexed imaging platform relates cellular phenotypes and tissue structure. Science advances, 5(10):eaax5851, 2019.Till Kroeger, Radu Timofte, Dengxin Dai, and Luc Van Gool. Fast optical flow using dense inverse search. ArXiv, abs / 1603.03590, 2016. URL “https: / / api.semanticscholar. org / CorpusID: 14824916”.Leena Latonen, Sonja Koivukoski, Umair Khan, and Pekka Ruusuvuori. Virtual staining for histology by deep learning. Trends in Biotechnology , 2024.Xiaoxian Li, Ji-Hoon Lee, Yuan Gao, Jilun Zhang, Katherine M Bates, David L Rimm, Huina Zhang, Geoffrey Hughes Smith, Diane Lawson, Jane Meisel, et al. Correlation of her2 protein level with mrna level quantified by rnascope in breast cancer. Modern Pathology, 37(2): 100408, 2024.Brittany McKelvey, Pedro A. Torres-Saavedra, Jessica Li, Glenn Broeckx, Frederik Deman, Siraj Ali, Hillary Andrews, Salim Arslan, Santhosh Balasubramanian, J. Carl Barrett, Pe- ter Caie, Ming Chen, Daniel Cohen, TathagataDasgupta, BrandonGallas, George Green, Mark Gustavson, Sarah Hersey, Ana Hidalgo-Sastre, Shahanawaz Jiwani, Wonkyung Jung, Kimary Kulig, Vladimir Kushnarev, Xiaoxian Li, Meredith Lodge, Joan Man- cuso, Mike Montalto, Satabhisa Mukhopadhyay, Matthew Oberley, Pahini Pandya, Oscar Puig, Edward Richardson, Alexander Sarachakov, Or Shaked, Mark Stewart, Lisa M. Me- Shane, Roberto Salgado, and Jeff Allen. Agreement across 10 artificial intelligence models in assessing her2 in breast cancer whole slide images:fmdings from the friends of cancer research digital path project. In San Antonio Breast Cancer Symposium, San Antonio, Texas, Dec. 10-13 2024. URL: “https: / / friendsofcancerresearch.org / wp-content / uploads / SABCS- Poster_Digital-PATH-Project-Findings.pdf”.Anais Mercier, Virginie Conan-Charlet, Isabelle Quintin-Roue, Laurent Doucet, Pascale Marcorelles, and Amaud Uguen. Reproducibility in pd-11 immunohistochemistry quan- tification through the tumor proportion score and the combined positive score: Could dual immunostaining help pathologists? Cancers, 15(10):2768, 2023.Owkin. Developing virtual staining at Owkin to enrich digital pathology. URL: “https: / / www.owkin.com / blogs-case-studies / developing- virtual-staining-at-owkin-to-enri ch-digital-pathology”, 2020.W. Paces, E. Ergon, E. Bueche, G.D. Young, V. Adisetiyo, C. Luengo, M. James, C. Cald- well, D. Miller, M. Wambaugh, and G. Metcalf. A digital assay for programmed death- ligand 1 (22C3) quantification combined with immune cell recognition algorithms in non- small cell lung cancer. Scientific Reports, 12(1):9745, 2022.Pushpak Pati, Sofia Karkampouna, Francesco Bonollo, Eva Comperat, Martina Radic, Mar- tin Spahn, Adriano Martinelli, Martin Wartenberg, Marianna Kruithof-de Julio, and Marianna Rapsomaniki. Accelerating histopathology workflows with generative ai-based virtually multiplexed tumour profiling. Nature Machine Intelligence, 6(9): 1077-1093, 2024.P. Pradhan, T. Meyer, M. Vieth, A. Stallmach, M. Waldner, M. Schmitt, J. Popp, and T. Bocklitz. Computational tissue staining of non-linear multimodal imaging using su- pervised and unsupervised deep learning. Biomedical Optics Express, 12(4):2280-2298, 2021.Hanna Pulaski, Stephen A Harrison, Shraddha S Mehta, Arun J Sanyal, Marlena C Vitali, Laryssa C Manigat, Hypatia Hou, Susan P Madasu Christudoss, Sara M Hoffman, Adam Stanford-Moore, et al. Clinical validation of an ai-based pathology tool for scoring of metabolic dysfunction-associated steatohepatitis. Nature Medicine, pages 1-8, 2024.Y. Rivenson, H. Wang, Z. Wei, K. de Haan, Y. Zhang, Y. Wu, H. Giinaydin, J.E. Zuckerman, T. Chong, A.E. Sisk, and L.M. Westbrook. Virtual histological staining of unlabeled tissue-autofluorescence images via deep learning. Nature Biomedical Engineering, 3(6): 466-477, 2019.Charles J Robbins, Aileen I Fernandez, Gang Han, Serena Wong, Malini Harigopal, Mima Podoil, Kamaljeet Singh, Amy Ly, M Gabriela Kuba, Hannah Wen, et al. Multi-institutional assessment of pathologist scoring her2 immunohistochemistry. Modem Pathology, 36(1): 100032, 2023.Marie E Robert, Josef Riischoff, Bharat Jasani, Rondell P Graham, Sunil S Badve, Manuel Rodriguez-Justo, Liudmila L Kodach, Amitabh Srivastava, Hanlin L Wang, Laura H Tang, et al. High interobserver variability among pathologists using combined positive score to evaluate pd-11 expression in gastric, gastroesophageal junction, and esophageal adenocarcinoma. Modern Pathology, 36(5): 100154, 2023.Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In 2011 International Conference on Computer Vision, pages 2564-2571, 2011. doi: 10.1109 / ICCV.2011.6126544.Ming Sound Tsao, Keith M Kerr, Mark Kockx, Mary-Beth Beasley, Alain C Borczuk, Johan Botling, Lukas Bubendorf, Lucian Chirieac, Gang Chen, Teh- Ying Chou, et al. Pd-11 immunohistochemistry comparability study in real-life clinical samples: results of blueprint phase 2 project. Journal of Thoracic Oncology, 13(9): 1302-1311, 2018.Y. Zhang, K. de Haan, Y. Rivenson, J. Li, A. Delis, and A. Ozcan. Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue. Light: Science & Applications, 9(1):78, 2020.

Claims

CLAIMS1. A computer-implemented method for generating virtual staining for an image of a biologic sample, the method comprising: obtaining a virtual stain concentration image by applying an artificial intelligence (“Al”) model to the image of a biologic sample; setting or receiving a staining control parameter; and generating a virtual stain image by processing the virtual stain concentration image according to the staining control parameter.

2. The computer-implemented method according to claim 1, wherein the staining control parameter is a parameter specifying at least one color to be applied to the virtual stain concentration image.

3. The computer-implemented method according to claim 1 or 2, wherein the staining control parameter is a parameter specifying at least one filter to be applied to the virtual stain concentration image.

4. The computer-implemented method according to claim 3, wherein the staining control parameter is a parameter specifying at least one of level of quantization, contrast, brightness, or smoothness to be applied to the virtual stain concentration image.

5. The computer-implemented method according to any of claims 2 to 4, wherein the virtual stain concentration image is an image in an optical density space, in which an intensity of a pixel is correlated with stain concentration.

6. The computer-implemented method according to any of claims 1 to 5, further comprising combining the image of a biologic sample with the virtual stain image.

7. The computer-implemented method according to any of claims 1 to 5, comprising: repeating the obtaining of the virtual stain concentration image, the setting or receiving the staining control parameter; and the generating the virtual stain image a plurality, N, of times, thereby obtaining N virtual stain images, and combining the image of a biologic sample with the N virtual stain images.

8. The computer-implemented method according to claim 7, wherein the repetition of the obtaining stain concentration image N times includes application of N different Al modelstrained to generate respective N different stain concentration images imitating staining respectively different antigens.

9. The computer-implemented method according to any of claims 6 to 8, wherein the combining comprises color processing of the image of a biologic sample and setting the staining control parameter, so as to imitate result of a fluorescence-based staining.

10. The computer-implemented method according to any of claims 1 to 9, wherein said setting a staining control parameter is obtained as an output of an Al model, optionally a neural network, processing one or more patches of said image of a biologic sample.

11. The computer-implemented method according to claim 10, wherein, said one or more patches comprise a plurality of patches; and for each of the plurality of patches, the Al model obtains more than one patch staining control parameters; and in said setting the staining control parameter, the staining control parameter is obtained as a statistic measure of the more than one patch staining parameters.

12. The computer-implemented method according to any of claims 1 to 11, further comprising: generating a graphical user interface (“GUI”), the GUI comprising an input means for a user to set the staining control parameter; and controlling a display to display the GUI.

13. The computer-implemented method according to claim 12, wherein the GUI further comprises: an area for displaying the image of the biologic sample and the virtual stain image in response to receiving an input via said input means.

14. The computer-implemented method according to any of claims 6 to 11, further comprising: generating a graphical user interface (“GUI”), the GUI comprising an input means for a user to set the staining control parameter; and controlling a display to display the GUI. wherein the GUI further includes a widget that allows a user to: select said N virtual stain concentration images among L virtual stain concentration image, L being equal to or larger than A,for each of the N virtual stain concentration images, respectively specify said control parameter, and displays said N virtual stain concentration images superimposed on the image of the biologic sample.

15. A computer-implemented method for generating virtual staining for an image of a biologic sample, the method comprising: repeating N times, N being an integer larger than 1, the following step z=l ..N obtaining an z-th virtual stain concentration image including applying an artificial intelligence (“Al”) model to the image of a biologic sample; setting an z-th staining control parameter; and generating an z-th virtual stain image by processing the i-the virtual stain concentration image according to the z-th staining control parameter, and combining the image of a biologic sample with the N virtual stain images.

16. A computer program stored on a non-transitory medium and including code instructions which, when executed on one or more processors, causes the one or more processors to perform steps of the method according any of claims 1 to 15.

17. An apparatus for generating virtual staining for an image of a biologic sample, the apparatus comprising: processing circuitry, which in operation: obtains a virtual stain concentration image by applying an artificial intelligence (“Al”) model to the image of a biologic sample; sets or receives a staining control parameter; and generates a virtual stain image by processing the virtual stain concentration image according to the staining control parameter, and an output interface, configured to output the generated virtual stain image.

18. The apparatus according to claim 17, wherein the staining control parameter is a parameter specifying at least one color to be applied to the virtual stain concentration image.

19. The apparatus according to claim 17 or 18, wherein the staining control parameter is a parameter specifying at least one filter to be applied to the virtual stain concentration image.

20. The apparatus according to claim 19, wherein the staining control parameter is a parameter specifying at least one of level of quantization, contrast, brightness, or smoothness to be applied to the virtual stain concentration image.

21. The apparatus according to any of claims 18 to 20, wherein the virtual stain concentration image is an image in an optical density space, in which an intensity of a pixel is correlated with a stain concentration.

22. The apparatus according to any of claims 17 to 21, wherein the processing circuitry, in operation further performs combining the image of a biologic sample with the virtual stain image.

23. The apparatus according to any of claims 17 to 21, wherein the processing circuitry, in operation further performs: repetition of the obtaining the virtual stain concentration image; setting or receiving the staining control parameter; and the generating a virtual stain image a plurality, TV, of times, thereby obtaining N virtual stain images, and combining the image of a biologic sample with the N virtual stain images.

24. The apparatus according to claim 23, wherein the repetition of the obtaining stain concentration image N times includes an application of N different Al models, each trained to generate a respective N different stain concentration images imitating the staining of respectively different antigens.

25. The apparatus according to any of claims 22 to 24, wherein the combining comprises color processing of the image of a biologic sample and setting the staining control parameter, so as to imitate a result of a fluorescence-based staining.

26. The apparatus according to any of claims 17 to 25, wherein said setting a staining control parameter is obtained as an output of an Al model, optionally a neural network, processing one or more patches of said image of a biologic sample.

27. The apparatus according to claim 26, wherein, said one or more patches comprises a plurality of patches; and for each of the plurality of patches, the Al model obtains a respective plurality of patchstaining control parameters; andin said setting the staining control parameter, the staining control parameter is obtained as a statistic measure of the plurality of patch-staining parameters.

28. The apparatus according to any of claims 17 to 27, wherein the processing circuitry, in operation further performs: generating a graphical user interface(“GUI”), the GUI comprising an input means for a user to set the staining control parameter; and controlling a display to display the GUI.

29. The apparatus according to claim 28, wherein the GUI further comprises: an area for displaying the image of the biologic sample and the virtual stain image in response to receiving an input via said input means.

30. The apparatus according to any of claims 22 to 27, wherein the processing circuitry, in operation further performs: generating a graphical user interface (“GUI”), the GUI comprising an input means for a user to set the staining control parameter; and controlling a display to display the GUI. wherein the GUI further includes a widget that allows a user to: select said N virtual stain concentration images among L virtual stain concentration image, L being equal to or larger than TV, for each of the N virtual stain concentration images, respectively specify said control parameter, and displays said N virtual stain concentration images superimposed on the image of the biologic sample.

31. An apparatus for generating virtual staining for an image of a biologic sample, comprising: processing circuitry that, in operation, performs: repeating N times, N being an integer larger than 1, the following step i=l ..N obtaining an z-th virtual stain concentration image by applying an artificial intelligence (“Al”) model to the image of a biologic sample; setting or receiving an z-th staining control parameter; and generating an z-th virtual stain image by processing the z-the virtual stain concentration image according to the z-th staining control parameter, and combining the image of a biologic sample with the N virtual stain images into a combined image; andan output interface, configured to output the combined image.

Images