Method for generating inference-based virtual staining image annotation

Abstract

A method for generating virtual staining image annotations. The method includes the step of providing a neural network executed by image processing software. The image processing software operates on a processor of a computing device. The method includes the step of training the neural network using a plurality of chemical staining patterns of endogenous signals to identify virtual staining patterns. The method includes the step of generating and annotating an image of a biological sample with respect to a biomarker. The biological sample includes endogenous signals. The method includes the step of using the trained neural network to identify virtual staining patterns within the image of the biological sample. Finally, the method includes the step of using spatial alignment to overlay the virtual staining patterns and annotations within the image of the biological sample to generate virtual staining image annotations.

Description

Technical Field

[0001] The present disclosure relates to virtual staining applications used for biomarker discovery, tissue-based research investigations, and diagnostic tests. More specifically, the present disclosure relates to a virtual staining methodology for the application of multiple IHC staining techniques to independent, identical tissue samples and the same tissue location without requiring washing of immunohistochemical (「IHC」) staining or modification of the tissue. The virtual staining methodology described herein can also be combined with other existing virtual staining and conventional assay readouts to create new multiplexed readouts.

Background Art

[0002] Immunohistochemical analysis is frequently used by clinicians and researchers for the evaluation and diagnosis of various diseases across many fields of medicine and biology. Within these fields, there is an increasing need for the use of biomarker information to evaluate in-situ protein expression within tumor tissue. As discussed in Immunohistochemical Staining Characteristics of Nephrogenic Adenoma Using the PIN-4 Cocktail (p63, AMACR, and CK903) and GATA-3, McDaniel, A., et al., Am. J. Surg. Pathol. December 2014; 38(12):1664-71 (Non-Patent Document 1), the immunohistochemical staining characteristics of nephrogenic adenoma using the PIN-4 cocktail (p63, AMACR, and CK903) and GATA-3 chromogenic IHC staining are the most widely used methodologies in the art, but this methodology is technically limited due to the number of tissue samples required per IHC staining panel. This limitation requires repeated sampling of rare tissues, which is often impossible due to the patient risk of tissue resection over a long period of time and can result in human error.

[0003] Furthermore, IHC staining techniques require physical staining of each individual tissue sample. These techniques use a "heat inactivation" step during the application of each stain or assay, whereby the high heat applied to the tissue completely denatures the previous antibody-enzyme complex and inactivates it for the application of the next IHC staining or assay. The use of the heat inactivation step also degrades the morphology of the tissue, rendering it unusable for additional IHC staining or assay readout. IHC staining techniques and assay readout are susceptible to human error, which can lead to the previous antibody-enzyme complex remaining active. Such situations can result in false positive or false negative results and can lead to incorrect staining patterns.

[0004] The risk of error increases with the number of specimens examined within a single tissue sample. As discussed in "The use of immunohistochemistry for biomarker assessment - can it compete with other technologies" in Toxicology Pathology, despite being semi-automated, multiplexing staining methods still have many drawbacks. Multiplexing staining methods are complex, labor-intensive, and time-consuming. For example, an increase in the number of automated fluid dispensing steps in a multiplexed staining procedure can lead to non-staining events where the reaction fails due to fluid escape during dispensing. In this case, false-negative results can occur without detection by the reviewing pathologist. Furthermore, due to steric hindrance from the application of antibodies and dyes, including non-specific background staining, it is not possible to create a generalizable method for multiplexed tissue staining (i.e., a method where the order of applied stains and the matching of antibodies and dye colors are not interchangeable). As discussed in "Multiplex Immunohistochemistry: The Importance of Staining Order When Producing a Validated Protocol" in Immunotherapy, markers such as CD3 and CD8, which are membrane stains using overlapping binding domains, interfere with the staining results of each individual marker. In this case, the study showed a 90% drop in the amount of CD8+ stained cells when combined with CD3 in multiplexed staining compared to individual staining. Additionally, it has been shown that the CD20 marker may not be combinable within a multiplex panel because the clinical protocol for this marker does not require heat-induced epitope retrieval, while these steps are required as part of a conventional multiplexed assay.

[0005] Different approaches have been developed to address some of the limitations described above and improve the overall workflow. In recent years, computational staining techniques have been developed using deep learning approaches to virtually stain tissue samples. Virtually staining tissue enables researchers to use virtual staining on digital images of tissue, thereby reducing the need for physical tissue samples and eliminating errors found in conventional staining methods and assay readouts. However, current methods of virtual staining only allow for one virtual staining per digital tissue image, thereby limiting the types of analysis that can be performed on digital tissue.

Prior Art Documents

Non-Patent Documents

[0006]

Non-Patent Document 1

Summary of the Invention

Means for Solving the Problems

[0007] In light of the foregoing, there is a need in the art for multiplex staining techniques that reduce time, increase efficiency and accuracy, and enable the reuse of biological tissue samples.

[0008] The object of the present disclosure is to provide a virtual staining technique that enables virtual staining multiplexing using virtual staining and conventional staining and assay readouts. Virtual staining based on certain embodiments of the present disclosure enables multiple stainings and assays to be applied independently and in combination to the same tissue sample without the need for washing of conventional staining or tissue modification. Thus, different stainings and assays may be applied multiple times at the same location. The present disclosure exploits the unique properties of virtual staining that enable customization and combination of multiple histological staining techniques.

[0009] In one embodiment, the method of the present disclosure includes the step of capturing an autofluorescence image of a physical tissue sample mounted on a slide. Multiple images of the physical tissue may be taken to obtain the most accurate representation of the tissue. Once the image of the slide is satisfactory, the digital image is automatically transferred to a standard image preprocessing pipeline in preparation for inference-based virtual staining. By using the autofluorescence image, one or more neural networks can be used to infer multiple virtual IHC stainings regarding issue classification. Since the digital image is created and stored, the image may undergo a second round of preprocessing prior to undergoing multiplexing.

[0010] After completion of the second round of preprocessing, the digital image of the stained tissue undergoes multiplexing. Here, the contrast staining and antibody-related staining of the image are converted to either a predefined staining vector or an eigenvector calculated using the image. In certain embodiments where more than one biomarker is used, the staining is separated into three or more images depicting each virtual antibody-related staining in a separate image. Each image containing the separated virtual antibody-related staining is then recolored to ensure that different biomarkers can be distinguished when recombined. The recolored virtual antibody-related staining and contrast staining are then recombined into a single image and converted from the optical density color space to the standard RGB color space where all three colors can be visualized.

[0011] In another embodiment, the tissue can be stained using conventional IHC staining techniques, and after autofluorescence imaging of the unlabeled tissue, specific biomarkers within the tissue can be identified. Once the tissue is stained using conventional antibody-related staining or assay techniques, including but not limited to 3,3'-diaminobenzidine ("DAB") staining, mass spectrometry, sequencing, and in situ hybridization ("ISH"), the IHC staining technique generates a stained tissue sample, which is then transferred onto a microscope slide that has a barcode label and is covered with a coverslip for imaging. The slide undergoes brightfield or whole-slide fluorescence imaging to acquire a digital image of the slide. The image undergoes quality control ("QC") to identify images of tissue that may not be suitable for processing. For example, cases include but are not limited to when the image is evaluated with respect to out-of-focus areas, missing tissue, cell count, necrosis, and other similar features. Once the digital image of the slide is satisfactory, the digital image is automatically transferred into the same image preprocessing pipeline as the virtually stained tissue, and the components of the stain are computationally separated. These separated stains can then be fused with the virtual stain in the same manner as described above.

[0012] The visualization of the staining combinations can be customized according to the needs of any particular user. The colors used for each of the combined stains can be individually changed, and the intensity and opacity of each stain can be customized. The process is implemented in real time using standard computers and may allow the user to immediately grasp the effect of color changes on the tissue.

[0013] In another embodiment, an alternative process can be applied that does not require any unrolling of the existing stain during inference. Instead, a virtual staining network can be trained to generate individual stains alone (i.e., contrast stain only, or antibody-related stain only) instead of generating conventional mixed stains. These individual stains can then be combined based on the user's preference. In a further embodiment, the virtual stain can also be generated using other modalities such as brightfield images of the stained tissue and used as an input to a neural network for inferring an image of virtual IHC. Additional imaging modalities that can be used to generate the virtual stain include, but are not limited to, phase imaging techniques and others used to image both labeled and unlabeled tissue, in addition to brightfield, darkfield, fluorescence lifetime, Raman, hyperspectral, and second harmonic generation microscopy.

[0014] In another embodiment, an alternative process can be used as a framework that directly utilizes virtual staining as an accurate means of providing annotations as an input for machine learning predictions. The quality of the annotation is directly related to the trained model prediction accuracy. A very accurate virtual stain can demarcate between sub-populations of cells, between cell states, and between signal transduction. Information from other channels such as additional staining, sequencing, or proteomics data can be used to provide accurate and rich annotations. This alternative workflow offers advantages over the current laborious and expensive annotation process.

[0015] The essential benefit of multiplexing using virtual staining from the same tissue section is the direct preparability for analytics. Using conventional approaches to multiplexing, individual biomarker channels must be unmixed, cell segmentation applied, and then the presence or absence of the biomarker on specific cells determined. This calculation is inefficient and can result in false positive or negative cells. In the virtual multiplexing approach, each biomarker stain is individually rendered and ready to call the biomarker status without using unmixing. Additionally, unlike conventional multiplexed immunohistochemistry (“mIHC”), where steric hindrance and other effects limit the ability to apply multiple IHC stains, there are no physical, chemical, or biological limits on the number of virtual stains and assays that can be rendered within the same cell or cell compartment.

[0016] The staining combinations generated by virtual stain multiplexing according to the present disclosure enable novel workflows and numerous combined readouts. The expected benefits of the virtual staining methodology include cellular architecture, protein targeting, diagnostic testing, and advanced insights into disease.

[0017] Furthermore, the present disclosure enables staining customization according to user needs, including specific configurations and intensities of multiple combined stains. Images of conventional IHC staining and virtual IHC staining can be separated into their individual components and then fused into any possible combination. In so doing, the present disclosure utilizes patterns from endogenous signals to digitally generate tissue staining patterns resulting from assays developed using antibodies, without requiring the use of antibodies or assay kits within the products being developed.

[0018] The present technology can be implemented and utilized in numerous ways, including, but not limited to, processes, apparatuses, systems, devices, and methods for computer-readable media and hardware devices, etc., that are currently known and later developed applications, which are specifically designed to perform the features and functions of the present technology. These and other unique features of the systems disclosed herein will become more readily apparent from the following description and the accompanying drawings.

Brief Description of the Drawings

[0019] Referring to this specification, the accompanying drawings, which form a part of this specification, illustrate, together with the description, the preferred embodiments of the present disclosure and serve to explain the principles of the present disclosure.

[0020]

Figure 1

[0021]

Figure 2

[0022]

Figure 3

[0023]

Figure 4

[0024]

Figure 5

[0025]

Figure 6

[0026]

Figure 7

[0027]

Figure 8

[0028]

Figure 9

[0029]

Figure 10

[0030]

Figure 11A

Figure 11B

[0031]

Figure 12A

Figure 12B

[0032]

Figure 13

[0033]

Figure 14

[0034] Detailed Description The advantages and other features of the methods disclosed herein will become readily apparent to those skilled in the art from the following detailed description of certain preferred embodiments, taken in conjunction with the drawings that depict representative embodiments of the present disclosure and in which like reference numerals identify similar structural elements. It should be understood that terms such as upper, lower, upward, downward, left, and right, as applied to the figures, are for the purpose of reference only and are not meant in a limiting sense.

[0035] Figure 1 illustrates a first embodiment 100 of the configuration of an inference-based virtual staining network. In operation step 110, the tissue section is de-waxed and a coverslip is applied. If there is a cytology or unused frozen section, etc., de-waxing is not required. In other situations, a fluorescence image can be obtained without the application of a coverslip. The fluorescence of the entire slide is then imaged. In step 120, the image is pre-processed and prepared for inference-based virtual staining. In step 130, inference-based virtual staining is performed for IHC marker 1 using nuclear counterstaining. This process is repeated n times as illustrated in steps 140-160. The image is collected and stored in step 170 with respect to the stained IHC marker and the associated nuclear counterstaining.

[0036] For each biomarker of interest within the virtual multiplexing panel, the network is designed and trained using a single biomarker stain, including nuclear counterstaining. The inference-based virtual staining network then generates an image virtually stained with each biomarker of interest in addition to the nuclear counterstaining. As seen in Figure 1, the virtually stained images must then be unmixed into individual images each showing an isolated biomarker of interest, which is then combined with the nuclear counterstaining for visualization in the individual images. The resulting individual image files may be used to detect cells and biomarker status for feeding into the calculation algorithm.

[0037] In the second embodiment 200 illustrated in FIG. 2, the coverslip is removed in step 210, and the tissue section on the microscope slide is dewaxed. The fluorescence of the entire slide is then imaged. In step 220, the resulting image is preprocessed and prepared for inference-based virtual staining. In step 230, inference-based virtual staining is performed for IHC marker 1 without using nuclear counterstaining. Inference-based nuclear counterstaining is performed independently of IHC marker 1 in step 240. This alternative inference-based virtual staining is performed for each IHC marker and nuclear counterstaining as illustrated in steps 250-255 and repeated n times. The images are then collected and stored in step 260 for the individually stained IHC markers and in step 270 for the associated separate nuclear counterstaining.

[0038] In this embodiment, the network for inference-based virtual staining generates two output image files, namely, one of pure biomarker staining and a second of corresponding nuclear counterstaining. Thus, the network for inference-based virtual staining unmixes the biomarker staining image from the nuclear counterstaining.

[0039] FIG. 3 illustrates a third embodiment 300 of the present disclosure in which each network for inference-based virtual staining generates a single output image file of pure biomarker staining. In operation step 310, the coverslip is removed and the tissue section on the microscope slide is dewaxed. The fluorescence of the entire slide is then imaged. Then, in step 320, the resulting image is preprocessed and prepared for inference-based virtual staining. In step 330, inference-based virtual staining is performed for IHC marker 1 without using nuclear counterstaining. The inference-based virtual staining is performed for each additional IHC marker as illustrated in steps 340-360 and repeated n times. In step 370, an inference-based harmonized nuclear counterstaining pattern is performed independently of the IHC markers. A fully unmixed virtual multiplexed ICH image file is created in step 380.

[0040] In the embodiment of FIG. 3, alongside each network for inference-based virtual staining is an additional network that generates a single nuclear counterstain image file. The single nuclear counterstain image file is applicable to any of the biomarker stains generated by the other networks of the inference-based virtual staining. Thus, the virtual staining unmixes the image files of the nuclear counterstain from the biomarker stains and seamlessly combines the recombination and combination of the nuclear counterstain image files to create a single reference nuclear counterstain.

[0041] In the fourth embodiment 400 of the present disclosure illustrated in FIG. 4, a single inference-based virtual staining network is trained to generate a fully mixed multiplexed image file containing all available biomarkers combined with the nuclear counterstain. Thus, the inference-based virtual staining network seamlessly combines the operation of virtual staining with the remixing and application of color vectors for biomarker stains required for visualization.

[0042] As shown in FIG. 4, in operation step 410, the coverslip is removed and the tissue section on the microscope slide is dewaxed. The fluorescence of the entire slide is then imaged. In step 420, the resulting image is preprocessed and prepared for inference-based virtual staining. Then, in step 430, inference-based virtual staining is performed for each IHC marker 1 using the nuclear counterstain. A fully mixed virtual multiplexed IHC image file is created in step 440.

[0043] In the fifth embodiment illustrated in FIG. 5, the method uses, alone, a single inference-based virtual staining network. Unlike other embodiments where all positive cells are stained for a biomarker by one single inference-based virtual staining network and subsequent individual subpopulations are stained by other single inference-based virtual staining networks, here the single network is trained for a single biomarker that generates two or more output image files where the staining patterns of specific cell subpopulations are partitioned across the images. The combinations of those stains will include the sum of the expected staining results from the conventional staining of the target biomarker. Thus, unmixed cell subpopulations that are positive for a biomarker may be analyzed and visualized separately.

[0044] In another embodiment illustrated in FIG. 6, in operation step 610, the coverslip is removed and the tissue section on the microscope slide is de-waxed. The fluorescence across the entire slide is then imaged. In step 620, the resulting image is preprocessed and prepared for inference-based virtual staining. In step 630, inference-based virtual staining is performed for an IHC marker using nuclear counterstaining associated with cell subpopulation 1. This combined inference-based virtual staining and counterstaining is performed for the IHC marker and associated counterstaining as illustrated in steps 640 - 660 and repeated n times for each cell subpopulation. Then, in step 670, associated individual cell subpopulation image files are created. In this embodiment, each network of inference-based virtual staining generates staining of the same biomarker on subpopulations of cells that are positive for the biomarker within the tissue section.

[0045] Figure 7 illustrates a unique way in which the resulting image can be constructed. The colors used for each of the combined stains can be individually changed according to a color palette or according to their staining vectors. The intensity of each component of the stain can be customized using a weighted slider. These intensities can further be changed using gamma correction to give the desired visualized intensity. In Figure 7, virtual IHC1 is the neural biomarker GFAP and virtual IHC2 is the neural biomarker IBA1, both of which are applied to tissues such as rat tissue. The separated components of the two stains can be seen below Figure 7, and the contrast stain separated from virtual IHC1 is shown on the side of the separated DAB stain for both IHC1 and 2. These separated stain components are then recombined according to defined parameters to generate a multiplexed IHC image. This process is implemented in real time using a standard computer and can enable the user to immediately grasp the effect of color changes.

[0046] The essential benefit of multiplexing using virtual stains from the same tissue section is the direct preparability for analytics. Using conventional approaches to multiplexing, the individual biomarker channels have to be unmixed, cell segmentation applied, and then the presence or absence of the biomarker on specific cells has to be determined. Conventional calculations are inefficient and can result in false positive or negative cells being called with respect to their biomarker status. In the virtual multiplexing approach, each biomarker stain is individually rendered and ready to call the biomarker status without using unmixing.

[0047] FIG. 8 illustrates a method by which the visualization of staining combinations can be customized according to the needs of any particular user. A plurality of IHC stains and counterstains are separated, the IHC stain image files are recolored to be distinguishable, and then the IHC stains and counterstains are recombined within a new image file. Here, a virtual staining network generates two separate neural biomarkers mIHC stains (GFAP and IBA1). mIHC is first performed by staining the biomarkers with DAB stain, and the background is counterstained. The DAB and counterstains are separated, but only the counterstain from stain 1 is used. Two images containing the separated DAB stain are recolored to be distinguishable.

[0048] Figure 9 demonstrates a method for creating a single combined nuclear counterstain by unmixing a set of virtual stains and based on the generated combinations of nuclear counterstains. In steps 901 and 902, an unmixed virtual IHC image and a virtual counterstain image are generated, respectively. The counterstain image is color corrected in step 903 by applying a linear function to the nuclear counterstain and then, if necessary, in step 904 by applying gamma correction to achieve the desired contrast. Nuclear options are determined in step 905 by segmenting the nuclear counterstain image. In step 906, eukaryotes are filtered using a combination of mappings to determine the probability that nuclei are present within a given area. The filtered combination of nuclei mapped in step 906 and the color-corrected combination of counterstains in step 903 are used and in step 907 are used to identify and estimate the true area and shape of the nuclei. This estimation can be the output for use in further analysis in step 908. In step 909, a harmonized single nuclear counterstain image is created using the contrast-adjusted counterstain in addition to the shape of the filtered area. This nuclear counterstain image and the unmixed virtual multiple IHC images can be combined into a single file in step 910.

[0049] Figure 10 illustrates the mapping of multiplexed virtual IHC to different colors for visualization. First, the image is unmixed in step 1010. Each component of the stain is then assigned a desired color within the RGB color space in step 1020. In step 1030, the stain is assigned to a specific viewing window such that all components are assigned to the same window if an overlay of all stains is desired. In step 1040, the relative intensity of the stain is adjusted to achieve the desired visualization. The image is then recombined in step 1060 and in step 1070 is converted from the optical density space to enable visualization by the user.

[0050] FIG. 11 illustrates an embodiment for a laboratory workflow used to generate hybrid assay results that combine virtual staining and a conventional assay in accordance with the present disclosure. The workflow begins at step 1101 where a tissue section mounted on a slide is deparaffinized (i.e., dewaxed), cover slip processed, and fluorescence imaged. At step 1102, image and tissue QC is performed to ensure that the image meets the appropriate criteria for virtual staining. If the image is rejected during QC, the sample may be re-imaged or re-processed at step 1103 to correct any defects, but is not passed on to proceed with conventional staining. If the image passes QC, the cover slip is removed at step 1104 to prepare the sample for the conventional assay. In parallel with step 1104, a "no problem staining" flag is set in the staining protocol database 1108 at step 1105, which enables the automated staining protocol to be implemented. After cover slip removal, the slide is loaded onto the automated staining platform at step 1106 to perform the conventional assay. The barcode on the slide is read at step 1107, which is referenced in the staining protocol database 1108 and the "no problem staining" flag is checked. If the "no problem staining" flag is not set, an error code is presented to the user at step 1110 and the slide is not allowed to be stained conventionally. If the "no problem staining" flag is set, at step 1111, the conventional staining protocol is downloaded to the automated staining platform, and the platform performs the conventional staining protocol at step 1112. Upon completion of the conventional staining assay, a cover slip is applied at step 1114, the assay results are imaged, and digitized. The result of this conventional assay process is the digitized image of the results at step 1115, which is suitable for combination with the virtual staining output at step 1116.

[0051] At the same time, the staining protocol database 1108 releases a panel of virtual stains, including the virtual multiplexed staining results. This input is combined with the preprocessing of the autofluorescence image performed in step 1117, and in step 1118, inference-based multiplexed virtual staining is performed. An exemplary method of step 1118 is disclosed in FIGS. 1-6. In step 1119, the multiplexed virtual staining results are returned from the system. This result is combined in step 1116 with the digitized version of the conventional staining result, an exemplary embodiment of which is disclosed in FIG. 12 below.

[0052] Figure 12 illustrates an embodiment of an unmixing and remixing process for digitally combining virtual staining results and conventional staining results for analysis or visualization in a hybrid multiplexing that is suitable for the present disclosure. In step 1201, an unmixed virtual multiplexed IHC biomarker stain is presented to the system. In step 1202, an unmixed conventional IHC biomarker stain is presented to the system. Step 1205 takes the outputs of steps 1201 and 1202 and performs an alignment between the two images. The purpose of the alignment step is to provide a common coordinate system for overlaying the images such that the pixels with virtual staining results are aligned with the same pixels containing the conventional staining results. In parallel, the virtual contrast stain image collected in step 1204 is prepared for alignment with the nuclear contrast stain image unmixed from the conventional assay in step 1203. In step 1206, the virtual nuclear contrast stain images are combined into a single image in a linear combination. Gamma correction is applied in step 1207 to rebalance the image, and the result is a harmonized single virtual nuclear contrast stain virtual image in step 1208. The result of step 1208 is combined with the output from step 1203 in step 1209, and an alignment is performed to align the pixels between the conventional and virtual nuclear contrast stains. A single image is created from the pair from step 1209 in step 1210, and a linear combination function is applied. The image is rebalanced again in step 1211 when using gamma correction, and a harmonized single nuclear contrast stain image is generated in step 1212.

[0053] In parallel with step 1212, the output of step 1205 is the stack of unmixed, virtual and conventional biomarker stains captured at step 1213, which may be combined as a stack of images including a single nuclear counterstain image at step 1214. This fully unmixed set of conventional and virtual biomarker stains and counterstains is suitable for further image analysis using conventional methods or linear remixing for visualization of the combined staining patterns.

[0054] FIG. 13 illustrates an embodiment of a computational workflow for utilizing virtual staining to generate annotations for use within a machine learning model or network, as referenced herein. At step 1301, an image of an unstained tissue section is presented to the machine learning model. At step 1302, a virtual staining algorithm is applied to convert the image of the unstained tissue section into a stained tissue. At step 1303, the stained tissue is annotated to extract additional information including, but not limited to, multiplexed immunofluorescence, FISH, proteomics, or sequencing data. The annotation can be performed at multiple levels depending on the annotation information extraction modality including, but not limited to, pixel level, tile level, or whole slide level. Where applicable, the tissue can also be augmented with virtual staining of specific biomarkers. At step 1304, the annotations are analyzed using a thresholding operator or another classifier. At step 1305, the annotations analyzed at step 1304 are overlaid on the virtually stained tissue from step 1302. This process utilizes the spatial matching between the annotations and data from steps 1302 and 1303 to extract additional information including, but not limited to, segmentation or classification.

[0055] FIG. 14 is a block diagram of a computer system 1400 arranged to perform operations associated with a neural network for inference-based virtual staining, as described herein. Exemplary computer system 1400 includes a central processing unit (CPU) 1402, a memory 1404, and an interconnect bus 1406. CPU 1402 may include a single microprocessor, multiple microprocessors, or special purpose processors for configuring computer system 1400 as a multiprocessor system. Memory 1404 illustratively includes main memory and read-only memory. Computer 1400 also includes a mass storage device 1408, such as, for example, various disk drives, tape drives, etc. Memory 1404 also includes dynamic random access memory (DRAM) and high-speed cache memory. In operation, memory 1404 stores at least some of the instructions and data for execution by CPU 1402. Memory 1404 may also contain computational elements, such as deep in-memory architecture (DIMA), where data is sent to the memory and functions of the data (e.g., matrix vector multiplication) are read by CPU 1402.

[0056] The mass storage device 1408 may include one or more magnetic disks, optical disk drives, and / or solid state memories for storing data and instructions for use by the CPU 1402. Preferably, at least one component of the mass storage system 1408, in the form of a non-volatile disk drive, solid state, or tape drive, stores a database that processes neural network data for inference-based virtual staining and is used to control the functionality. The mass storage system 1408 may also include one or more drives for various portable media such as floppy (registered trademark) disks, flash drives, compact disk read only memories (CD-ROM, DVD, CD-RW, and variants), memory sticks, or integrated circuit non-volatile memory adapters (i.e., PC-MCIA adapters) for inputting data and code into and outputting from the computer system 200.

[0057] As an example, computer system 1400 may also include one or more input / output interfaces for communication, shown as interface 1410 and / or transceiver for data communication via network 1412. Data interface 1410 may be a modem, an Ethernet® card, or any other suitable data communication device. To provide the functionality of a processor for operating a neural network for inference-based virtual staining, data interface 1410 may provide a relatively high-speed link to network 1412, such as an intranet, the Internet, or the Internet, either directly or through another external interface. The communication link to network 1412 may be, for example, optical, wired, or wireless (e.g., via a satellite or cellular network). Computer system 1400 may also be connected to at least one other computer system via data interface 1410 and network 1412 and may perform remote or distributed multisensor processing, for example, related to a COP (common operational picture). Alternatively, computer system 1400 may include a mainframe or other type of host computer system capable of Web-based communication via network 1412. Computer system 1400 may include software for operating network applications such as a web server and / or web client.

[0058] Computer system 1400 may also include suitable input / output ports that use interconnect bus 1406 for connection to a local display 1416 and keyboard 1414 or equivalents, which interface with a portable data storage device or serve as a local user interface for programming and / or data retrieval purposes. The display 1416 may include touch screen capabilities to enable a user to interface with the system 1400 by touching a portion of the surface of the display 1416. Server operators may interact with the system 1400 to control and / or program the system from a remote terminal device via network 1412.

[0059] Computer system 1400 may run various application programs and store data associated within the database of mass storage system 1408. One or more such applications may include neural networks for inference-based virtual staining, such as those described with respect to FIGS. 1-13.

[0060] The components contained within computer system 1400 may enable the computer system to be used as a server, workstation, personal computer, network terminal, mobile computing device, cellular phone, system on chip (SoC), and equivalents. System 1400 may include software and / or hardware that implements a web server application. The web server application may include software such as HTML, XML, WML, SGML, PHP (Hypertext Preprocessor), CGI, and similar languages.

[0061] The foregoing features of the present disclosure may be implemented as software components operating within system 1400, which may include a Unix® workstation, a Windows® workstation, a LINUX® workstation, or other types of workstations. Other operating systems such as, but not limited to, Windows®, MAC OS®, and LINUX® may be employed. In some aspects, the software may optionally be implemented as a C language computer program, or as a computer program written in any high-level language including, but not limited to, Javascript®, Java®, CSS, Python, Keras, TensorFlow, PHP, Ruby, C++, C, Shell, C#, Objective-C, Go, R, TeX, VimL, Perl, Scala, CoffeeScript, Emacs Lisp, Swift, Fortran, or Visual BASIC. Some script-based programs such as XML, WML, PHP, etc. may be employed. System 200 may use a digital signal processor (DSP).

[0062] As described above, mass storage device 1408 may include a database. The database may be any suitable database system including, but not limited to, a commercially available Microsoft Access database, and may be a local or distributed database system. The database system may implement Sybase and / or SQL Server. The database may be supported by any suitable persistent data memory such as a hard disk drive, a RAID system, a tape drive system, a floppy® disk, or any other suitable system. System 1400 may include a database integrated with a neural network for inference-based virtual staining, however, it should be understood that in other implementations, the database and mass storage device 1408 may be external elements.

[0063] In one implementation, system 1400 may include an Internet browser program and / or be configured to operate as a web server. In some configurations, the client and / or web server may be configured to recognize and interpret various network protocols that may be used by client or server programs. Commonly used protocols include, for example, the Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Telnet, and Secure Sockets Layer (SSL), and Transport Layer Security (TLS). However, new protocols and revisions to existing protocols may be frequently introduced. Thus, new revisions of server and / or client applications may be continuously developed and released to support new or revised protocols.

[0064] In one implementation, the neural network may be configured and operate on a network-based, e.g., Internet-based application, on any combination of system 1400 and / or other components of the neural network for inference-based virtual staining. The computer system 1400 may include a web server that operates a Web 2.0 application or equivalent. Web applications operating on the neural network may use server-side dynamic content generation mechanisms such as, but not limited to, Java servlets, CGI, PHP, or ASP. In one implementation, mashed-up content may be generated by a web browser that operates client-side scripting such as, but not limited to, JavaScript® and / or applets on a wireless device.

[0065] In one implementation, the neural network or computer system 1400 for inference-based virtual staining may include applications that use asynchronous loading and content presentation techniques, and employ asynchronous JavaScript (TM) + XML (Ajax) and similar technologies. These techniques may include, but are not limited to, XHTML and CSS for style presentation, the Document Object Model (DOM) API exposed by web browsers, asynchronous data exchange of XML data, and client-side scripting, such as JavaScript (TM). Some web-based applications and services may utilize web protocols, including, but not limited to, the Service-Oriented Access Protocol (SOAP) and Representational State Transfer (REST). REST may utilize HTTP with XML.

[0066] The neural network, computer system 1400, or another component of the neural network for inference-based virtual staining may also provide enhanced security and data encryption. Enhanced security may include access control, biometric authentication, decryption authentication, message integrity checks, encryption, digital rights management services, and / or other similar security services. Security may include protocols such as IPSEC and IKE. Encryption may include, but is not limited to, DES, 3DES, AES, RSA, ECC, and any similar public or private key-based scheme.

[0067] Those skilled in the art will understand that the functions of some elements may be performed by fewer elements or a single element in alternative embodiments. Similarly, in some embodiments, any functional element may perform fewer or different operations than those described with respect to the illustrated embodiments. Also, for purposes of illustration, functional elements shown as distinct may be incorporated within other functional elements in a particular implementation (e.g., modules, databases, interfaces, computers, servers, and the like may perform any combination of functional elements).

[0068] This technology has been described with respect to preferred embodiments, but those skilled in the art will readily understand that various changes and / or modifications can be made to this technology without departing from the spirit or scope of the present disclosure. The appended claims are illustrative and can be combined and arranged in any manner, including multiple dependencies and equivalents.

Claims

1. A method for generating inference-based virtual staining image annotation, comprising: providing one or more neural networks executed by image processing software operating on one or more processors of a computing device; training the one or more neural networks using a plurality of images of chemical staining of one or more endogenous signals to identify one or more virtual staining patterns; acquiring data corresponding to a biological sample; acquiring and generating an image of the biological sample including one or more endogenous signals identified by an annotation technique; detecting the one or more virtual staining patterns in the image of the biological sample using the one or more neural networks; overlaying the virtual staining pattern detected in the image of the biological sample using a spatial matching technique to create an inference-based virtual staining image annotation and including a method.

2. The method according to claim 1, wherein the annotation includes features used by the one or more neural networks to perform semantic segmentation.

3. The method according to claim 1, wherein the acquisition and generation of the image of the biological sample incorporates sequencing or imaging mass spectrometry.

4. The method according to claim 1, wherein the acquisition and generation of the image of the biological sample incorporates immunohistochemistry or immunofluorescence techniques.

5. The method according to claim 4, wherein the immunohistochemistry or immunofluorescence technique includes directly comparing virtual staining patterns for two or more antibody clones on the same tissue section.

6. The immunohistochemistry or immunofluorescence technique The method according to claim 4, including creating unique virtual monoclonal antibody staining from a plurality of monoclonal clones and adjusting the concentration to a desired expression level.

7. The method according to claim 1, wherein the opacity level of the virtual staining pattern can be adjusted to enable focusing on a specific antibody clone.

8. The method according to claim 1, wherein the virtual staining pattern renders pattern differences that enable quantitative analysis.

9. The method according to claim 1, further comprising individually manipulating the virtual staining patterns of the overlaid virtual staining patterns.

10. The method according to claim 9, wherein the manipulation includes adjusting the intensity of each virtual staining pattern within a real-time process.

11. The method according to claim 1, further comprising multiplexing the overlaid virtual staining patterns with existing virtual staining or conventional assay readouts.

12. A method for generating virtual staining image annotations, comprising: acquiring an image of a biological sample; virtually staining the biological sample using a machine learning algorithm executed via a computer program operating on a processor, wherein the machine learning algorithm detects a virtual staining pattern of an endogenous signal in the biological sample; annotating the virtually stained biological sample with respect to a biomarker; analyzing the annotation of the virtually stained biological sample; overlaying the analyzed annotation on the virtually stained biological sample and including.

13. The method according to claim 12, further comprising semantically segmenting the virtually stained biological sample.

14. The method according to claim 12, further comprising training the machine learning algorithm using a plurality of virtual staining patterns of endogenous signals.

15. A method for generating an inference-based virtual staining image, comprising: providing a neural network executed by image processing software operating on one or more processors of a computing device; training the neural network using a plurality of images of chemical stains of one or more endogenous signals and identifying one or more virtual staining patterns; acquiring and generating an image of a biological sample; using the neural network to detect the virtual staining pattern in the image of the biological sample using the trained neural network; using the neural network to analyze the endogenous signal of the detected virtual staining pattern; outputting two separate virtual images of each corresponding endogenous signal and including, A method in which the two separate virtual images depicting the analyzed endogenous signals can be combined and recombined to selectively generate one or more new multiplexed virtual images.

Images