Method of generating inference-based virtual stained image annotations

Abstract

A method of generating virtual stained image annotations. The method includes providing a neural network executed by image processing software. The image processing software runs on a processor of the computing device. The method includes training a neural network with a plurality of chemical stain patterns of an endogenous signal to identify a virtual stain pattern. The method includes generating and annotating an image of a biological sample for a biomarker. The biological sample includes an endogenous signal. The method includes identifying a virtual staining pattern in an image of a biological sample using a trained neural network. Finally, the method includes superimposing the virtual stain pattern in the image of the biological sample with the annotation using spatial matching to produce a virtual stain image annotation.

Description

Technical Field

[0001] The present disclosure relates to a virtual staining application for biomarker discovery, tissue-based research and diagnostic testing. More specifically, the present disclosure relates to a virtual staining method for independently applying multiple immunohistochemistry ("IHC") staining techniques to the same tissue sample and the same tissue location without washing the IHC stain or modifying the tissue. The virtual staining method described herein can also be combined with other existing virtual staining and traditional assay readouts to create new multiple readouts. Background Art

[0002] Immunohistochemical analysis is often used by clinicians and researchers in many medical and biological fields to evaluate and diagnose various diseases. In these fields, the demand for the in situ protein expression in tumor tissues is growing using biomarker information. As discussed in Immunohistochemical Staining Characteristics of Nephrogenic Adenoma Using the PIN-4 Cocktail (p63, AMACR, and CK903) and GATA-3, McDaniel, A., et al., Am. J. Surg. Pathol. 2014 Dec; 38 (12): 1664-71, using PIN-4 mixture (p63, AMACR and CK903) and GATA-3 color development IHC staining to nephrogenic adenoma is the most widely used method in the field, but due to the number of tissue samples required for each group of IHC staining, the method is technically limited. This limitation requires repeated sampling of scarce tissue, which is often time-consuming, impractical due to the risk of tissue removal for the patient, and can lead to human error.

[0003] Additionally, IHC staining techniques require physical staining of each individual tissue sample. These techniques use a "heat inactivation" step between each staining or assay application, where the high heat applied to the tissue completely denatures the previous antibody-enzyme complex, rendering it inactive for the next IHC staining or assay application. The use of a heat inactivation step also degrades the morphology of the tissue, rendering it unusable for additional IHC staining or assay readouts. IHC staining techniques and assay readouts are susceptible to human error, which can cause previous antibody-enzyme complexes to remain active. Such situations can result in erroneous staining patterns, leading to false-positive or false-negative results.

[0004] The greater the number of analytes tested in a single tissue sample, the greater the risk of error. Despite being semi-automated, multiplex staining methods still have many disadvantages, as described in "The use of immunohistochemistry for biomarker assessment–can it compete with other technologies" in Toxicological Pathology. Multiplex staining methods are complex, labor-intensive, and time-consuming. For example, the increased number of automated fluid dispensing steps in the multiplex staining process may lead to non-staining events, that is, reaction failures due to missed fluid dispensing. In this case, false-negative results may occur without the reviewing pathologist being able to detect them. In addition, due to steric hindrances caused by the application of antibodies and dyes, including nonspecific background staining, it is impossible to create a universal multiplex tissue staining method (i.e., a method in which the order of application of stains and the matching of antibody and dye colors are not interchangeable). As discussed in "Multiplex Immunohistochemistry: The Importance of Staining Order When Producing a Validated Protocol" in Immunotherapy, markers such as CD3 and CD8 (membrane stains with overlapping binding domains) will interfere with the staining results of each individual marker. In this case, the study showed that the amount of CD8+ stained cells decreased by 90% when combined with CD3 in a multiplex staining compared to staining alone. In addition, the study showed that the CD20 marker could not be combined with the multiplex panel because the clinical protocol for this marker does not require heat-induced epitope retrieval, which is required for traditional multiplex detection.

[0005] Different approaches have been developed to address some of the above limitations and improve the overall workflow. Recently, computational staining techniques have been developed that use deep learning methods to virtually stain tissue samples. Virtually stained tissue enables researchers to use virtual staining on digital images of tissue, thereby reducing the need for physical tissue samples and eliminating errors found in traditional staining methods and assay readouts. However, current virtual staining methods only allow for one virtual stain per digital tissue image, limiting the types of analyses that can be run on digital tissue. Summary of the invention

[0006] In view of the above, there is a need in the art for a multiplex staining technique that reduces time, improves efficiency and accuracy, and allows for the reuse of biological tissue samples.

[0007] The purpose of the present disclosure is to provide a virtual staining technology, which allows the use of virtual staining and traditional staining and determination readings to perform virtual staining multiplexing (virtual stain multiplexing). Virtual staining based on the embodiments of the present disclosure allows multiple staining and determination agents to be applied to the same tissue sample independently and in combination without cleaning traditional staining or modifying tissue. In this way, different staining and determination agents can be applied to the same position many times. The present disclosure utilizes the unique properties of virtual staining, which enables customization and combination of multiple histological staining techniques.

[0008] In one embodiment, the method of the present disclosure includes capturing an autofluorescence image of a physical tissue sample mounted on a slide. Multiple images of the physical tissue can be taken to obtain the most accurate representation of the tissue. Once the image of the slide is satisfactory, the digital image is forwarded to a standard image preprocessing pipeline to prepare for inference-based virtual staining. Using the autofluorescence image, one or more neural networks can be used to infer multiple virtual IHC stainings of the tissue section. Now that the digital image has been created and stored, the image can undergo a second round of preprocessing before multiplexing.

[0009] After completing the second round of preprocessing, the digital image of the stained tissue is multiplexed. Here, the counterstaining and antibody-associated staining of the image are converted into a pre-specified staining vector or feature vector calculated using the image. In embodiments using more than one biomarker, the staining is divided into three or more images, depicting each virtual antibody-associated staining in a separate image. Each image containing the separated virtual antibody-associated staining is then recolored to ensure that different biomarkers can be distinguished when recombined. The recolored virtual antibody-associated staining and counterstaining are then recombined into a single image and converted from the optical density color space to the standard RGB color space, where all three colors can be virtualized.

[0010] In another embodiment, if desired, the tissue can be stained using traditional IHC staining techniques to identify specific biomarkers within the tissue after autofluorescence imaging of the unlabeled tissue. Once the tissue is stained using traditional antibody-associated staining or assay techniques, including but not limited to 3,3'-diaminobenzidine ("DAB") staining, mass spectrometry, sequencing, and in situ hybridization ("ISH"). The IHC staining technique produces a stained tissue sample, which is then transferred to a microscope slide with a barcode label and covered with a coverslip for imaging. The slide is imaged by bright field or fluorescent whole slide to obtain a digital image of the slide. The image is quality controlled ("QC") to identify tissue images that are not suitable for processing. For example, examples include but are not limited to evaluating out-of-focus areas, missing tissue, cell counts, necrosis, and other similar features of the image. Once the digital image of the slide is satisfactory, the digital image is forwarded to the same image preprocessing pipeline as the virtual stained tissue and the components of the staining are separated by calculation. These separated stains can then be merged with the virtual staining in the same manner as described above.

[0011] Visualization of stain combinations can be customized to the needs of any particular user. The color of each stain used in the combination can be changed individually, and the intensity and opacity of each stain can be customized. The process can be performed in real time using a standard computer, allowing the user to immediately see the effect of the color change on the tissue.

[0012] In another embodiment, an alternative process can be applied that does not require any deconvolution of existing staining during reasoning. Instead, the virtual staining network can be trained to generate only a single staining (i.e., only counterstaining, or only antibody-associated staining), rather than generating traditional mixed staining. These single stainings can then be combined based on the user's preferences. In another embodiment, other modalities can also be used to generate virtual staining, such as bright field images of stained tissues, and used as inputs to neural networks to infer images of virtual IHC. Additional imaging modalities that can be used to generate virtual staining include, but are not limited to, bright field, dark field, fluorescence lifetime, Raman, hyperspectral, harmonic wave generation microscopy, and phase imaging techniques and other techniques for imaging both labeled and unlabeled tissues.

[0013] In another embodiment, an alternative process can be used as a framework that directly utilizes virtual staining as an accurate method for providing annotations as input to a machine learning project. The quality of the annotations is directly related to the training model prediction accuracy. Highly accurate virtual staining can distinguish cell subpopulations, cell states, and signal transduction. Information from other channels, such as additional staining, sequencing, or proteomics data, can be used to provide accurate and rich annotations. This alternative workflow has advantages over the current cumbersome and expensive annotation process.

[0014] The inherent benefit of multiplexing with virtual staining from the same tissue section is that analysis can be performed directly. With traditional multiplexing methods, each biomarker channel must be unmixed, cell segmentation applied, and then whether a biomarker is present on a specific cell. This calculation is inefficient and may result in false positive or false negative cells. In the virtual multiplexing method, each biomarker stain is presented separately, and the biomarker state can be called without unmixing. In addition, unlike traditional multiple immunohistochemistry ("mIHC") (where steric hindrance and other effects limit the ability to apply multiple IHC staining), there is no physical, chemical or biological limitation on the number of virtual staining and assays presented in the same cell or cell compartment.

[0015] According to the present disclosure, the staining combinations generated by virtual staining multiplexing allow new workflows and a large number of combined readouts. The expected benefits of virtual staining methods include deep insights into cell structure, protein targeting, diagnostic testing, and disease.

[0016] In addition, the present disclosure is able to customize staining according to user needs, including specific configurations and intensities of multiple combined stainings. Images of traditional IHC staining and virtual IHC staining can be separated into their individual components and then merged in any possible combination. In doing so, the present disclosure utilizes patterns from endogenous signals to digitally generate tissue staining patterns produced by assays developed using antibodies, but does not require the use of antibodies or test kits in deployed products.

[0017] It should be appreciated that the subject technology can be implemented and utilized in a variety of ways, including but not limited to as a process, apparatus, system, device, method for applications now known and later developed, such as computer-readable media and hardware devices specifically designed to implement the features and functions of the subject technology. These and other unique features of the system disclosed herein will become more apparent from the following description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated herein and constitute a part of this document, illustrate preferred embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0019] Figure 1 A standard configuration of a network design for a parallel inference network using inference-based virtual coloring according to the present disclosure is shown.

[0020] Figure 2 A second embodiment of a network design according to the present disclosure is shown, where each inference-based virtual stainer produces a pure biomarker staining and a corresponding nuclear staining.

[0021] Figure 3A third embodiment of a network design according to the present disclosure is shown, where each inference-based virtual stainer produces a single output image of pure biomarker staining.

[0022] Figure 4 A fourth embodiment of a network design according to the present disclosure is shown, in which a single inference-based virtual stainer is trained to produce fully mixed multiplexed images of all biomarkers combined with a nuclear counterstain.

[0023] Figure 5 A fifth embodiment of a network design according to the present disclosure is shown, in which a single inference-based stainer for a single biomarker produces two or more output images, in which the staining patterns of specific cell subpopulations are divided between the images.

[0024] Figure 6 A sixth embodiment of a network design according to the present disclosure is shown, in which each inference-based virtual stainer produces staining of the same biomarker on a subpopulation of cells that are positive for the biomarker in a tissue section.

[0025] Figure 7 Shown is a virtual staining network that generates two separate representative IHC stainings using the neural biomarkers GFAP and IBA1 according to the present disclosure.

[0026] Figure 8 Show Figure 7 Customized visualization of the staining combinations used in .

[0027] Fig. 9 A method for creating a single nuclear counterstain layer given a set of virtual stains according to the present disclosure is shown.

[0028] Fig.10 A method for mapping colors from unmixed multiplexed IHCs together for visualization according to the present disclosure is shown.

[0029] 11 illustrates an embodiment of a laboratory workflow for generating hybrid assay results combining virtual staining and traditional assays according to the present disclosure.

[0030] 12 illustrates an embodiment of a computational workflow for digitally combining virtual staining results and traditional assay results into a hybrid multiplex suitable for analysis or visualization according to the present disclosure.

[0031] Fig.13 An embodiment of a computational workflow for generating annotations for a machine learning model using virtual coloring is shown.

[0032] Fig.14 is a block diagram of a computer system arranged to perform processing associated with the inference-based virtual coloring neural network described herein. DETAILED DESCRIPTION

[0033] Those skilled in the art will more easily understand the advantages and other features of the methods disclosed herein from the following detailed description of certain preferred embodiments in conjunction with the accompanying drawings, which illustrate representative embodiments of the present disclosure and in which the same reference numerals identify similar structural elements. It should be understood that references to the accompanying drawings, such as top, bottom, upward, downward, left, and right, are relative to the accompanying drawings and are not restrictive.

[0034] Figure 1 A first embodiment 100 of a configuration of an inference-based virtual staining network is shown. At operation step 110, a tissue section is dewaxed and a coverslip is applied. In some cases, such as cytology or fresh frozen sections, dewaxing is not required. In other cases, a fluorescence image can be obtained without applying a coverslip. The fluorescence of the entire slide is then imaged. At step 120, the image is preprocessed and prepared for inference-based virtual staining. At step 130, an inference-based virtual staining is performed on IHC marker 1 with nuclear counterstain. The process is repeated n times, as shown in steps 140-160. In step 170, images of the stained IHC markers and associated nuclear counterstains are collected and stored.

[0035] For each biomarker of interest in the virtual multiplex panel, the network is designed and trained using a single biomarker stain and includes a nuclear counterstain. The inference-based virtual staining network then generates a virtual staining image with each biomarker of interest as well as a nuclear counterstain. Figure 1 As shown in , the virtual stained image must then be unmixed into separate images, each showing the biomarker of interest in isolation, which are then combined with a nuclear counterstain to be visualized in a single image. The resulting single image file can be used to detect cell and biomarker status for input into computational algorithms.

[0036] exist Figure 2 In the second embodiment 200 shown in , the coverslip is removed in step 210 and the tissue section on the microscope slide is dewaxed. The fluorescence of the entire slide is then imaged. At step 220, the resulting image is preprocessed and prepared for inference-based virtual staining. At step 230, inference-based virtual staining is performed on IHC marker 1 without nuclear counterstaining. In step 240, inference-based nuclear counterstaining is performed independently of IHC marker 1. This alternating inference-based virtual staining is performed for each IHC marker and nuclear counterstaining and repeated n times, as shown in steps 250-255. Images of the individually stained IHC markers are then collected and stored in step 260, and images of the associated individually stained nuclear counterstains are collected and stored in step 270.

[0037] In this embodiment, the inference-based virtual staining network generates two output image files, one of pure biomarker staining and the other of corresponding nuclear counterstaining. In this way, the inference-based virtual staining network unmixes the biomarker stained image from the nuclear counterstaining image.

[0038] Figure 3 A third embodiment 300 of the present disclosure is shown, wherein each inference-based virtual staining network produces a single output image file of pure biomarker staining. At operation step 310, the coverslip is removed and the tissue section on the microscope slide is dewaxed. The fluorescence of the entire slide is then imaged. Then at step 320, the resulting image is preprocessed and prepared for inference-based virtual staining. At step 330, inference-based virtual staining is performed on IHC marker 1 without nuclear counterstaining. Inference-based virtual staining is performed for each additional IHC marker and repeated n times, as shown in steps 340-360. At step 370, an inference-based coordinated nuclear counterstaining pattern is performed independently of the IHC marker. A fully unmixed virtual multiplexed ICH image file is created at step 380.

[0039] exist Figure 3 In an embodiment of the invention, next to each inference-based virtual staining network is an additional network that produces a single nuclear counterstain image file. The single nuclear counterstain image file is applicable to any biomarker staining produced by another inference-based virtual staining network. In this way, the virtual staining is seamlessly combined with the image file of the biomarker staining unmixed from the nuclear counterstain and the recombination of the nuclear counterstain image file to create a single reference nuclear counterstain.

[0040] exist Figure 4 In the fourth embodiment 400 of the present disclosure shown in , a single inference-based virtual staining network is trained to generate a fully mixed multiplexed image file containing all available combinations of biomarkers and nuclear counterstains. In this way, the inference-based virtual staining network seamlessly combines the operation of virtual staining with the remixing of biomarker stains and the application of color vectors required for visualization.

[0041] like Figure 4 As shown in , in operation step 410, the cover slip is removed and the tissue section on the microscope slide is dewaxed. Then the fluorescence of the entire slide is imaged. In step 420, the resulting image is pre-processed and prepared for inference-based virtual staining. Then in step 430, inference-based virtual staining is performed for each IHC marker 1 with nuclear counterstain. A fully mixed virtual multiplexed IHC image file is created at step 440.

[0042] exist Figure 5In the fifth embodiment shown in, the method uses only a single virtual staining network based on reasoning. Different from other embodiments in which all positive cells are stained with a single virtual staining network based on reasoning and subsequent individual populations are stained with other single virtual staining networks based on reasoning, here, a single network is trained for a single biomarker, and the single biomarker produces two or more output image files, wherein the staining pattern of a specific cell subpopulation is divided between images. The combination of these stainings will constitute the total expected staining result of the traditional staining of the target biomarker. In this way, the unmixed cell populations that are positive for biomarkers can be analyzed and visualized separately.

[0043] exist Figure 6 In another embodiment shown in , at operation step 610, the coverslip is removed and the tissue section on the microscope slide is dewaxed. The fluorescence of the entire slide is then imaged. At step 620, the resulting image is preprocessed and prepared for inference-based virtual staining. At step 630, inference-based virtual staining is performed on the IHC markers and associated nuclear counterstaining of cell population 1. The combined inference-based virtual staining and counterstaining is performed on the IHC markers and associated counterstaining, and repeated n times for each cell population, as shown in steps 640-660. Then at step 670, an associated single cell population image file is created. In this embodiment, each inference-based virtual staining network produces staining of the same biomarker on a subpopulation of cells that are positive for the biomarker in the tissue section.

[0044] Figure 7 shows the unique ways in which the resulting image can be configured. The color used for each combined tint in the combination can be changed individually, either according to the palette or according to its tint vector. The intensity of each component of the tint can be customized using the weight sliders. These intensities can be further varied using gamma correction to provide the desired visualization intensity. Figure 7 In , virtual IHC 1 is the neural biomarker GFAP, and virtual IHC 2 is the neural biomarker IBA1, both of which are applied to tissue, such as rat tissue. Figure 7 The separated components of the two stainings are seen at the bottom of the figure, where the separated counterstain of virtual IHC 1 is displayed next to the separated DAB staining of both IHC 1 and 2. These separated staining components are then recombined according to the specified parameters to generate a multiplexed IHC image. This process can be performed in real time using a standard computer, allowing the user to immediately see the effect of the color change.

[0045] An inherent benefit of multiplexing using virtual staining from the same tissue section is that analysis can be performed directly. With traditional multiplexing methods, individual biomarker channels must be unmixed, cell segmentation applied, and then a determination of whether a biomarker is present on a specific cell is made. Traditional computational inefficiencies can result in false positive or negative cells being called for their biomarker status. In a virtual multiplexing approach, each biomarker stain is presented individually, and no unmixing is required to call the biomarker status.

[0046] Figure 8 Visualization showing how to customize stain combinations to the needs of any particular user. Separate multiple IHC stains and counterstains, recolor the IHC stain image file for differentiation, and then recombine the IHC stains and counterstains into a new image file. Here, the virtual staining network generates two separate neural biomarker mIHC stains (GFAP and IBA1). mIHC is initially performed by staining the biomarkers with DAB stain and counterstaining the background. Separate the DAB and counterstains, but only use the counterstain from stain 1. Recolor the two images containing the separated DAB stains for differentiation.

[0047] Fig. 9 Demonstrates a method for unmixing a set of virtual stains and creating a single combined nuclear counterstain based on a generated nuclear counterstain combination. In steps 901 and 902, an unmixed virtual IHC image and a virtual counterstain image are generated, respectively. The counterstain image is color corrected by applying a linear function to the nuclear counterstain in step 903, and then gamma correction is applied as needed in step 904 to achieve the desired contrast. The selection of the nucleus is determined by segmenting the nuclear counterstain image in step 905. In step 906, to determine the probability that a nucleus exists in a given region, the true nucleus is down-selected using the mapped combination. The combination of the down-selected nuclei mapped in step 906 and the color-corrected counterstain combination in step 903 are used to identify and estimate the true area and shape of the nucleus in step 907. This estimate can be output in step 908 for further analysis. In step 909, a coordinated single nuclear counterstain image is created using the contrast-adjusted counterstain and the shape of the down-selected region. In step 910, the nuclear counterstain image and the unmixed virtual multiple IHC image can be combined into a single file.

[0048] Fig.10Multiplexed virtual IHCs are shown mapped to different colors for visualization. First, the image is unmixed in step 1010. Each component of the stain is then assigned to a desired color in the RGB color space in step 1020. The stain is assigned to a specific viewing window at step 1030, and if all stains need to be covered, all components are assigned to the same window. The relative intensity of the stain is adjusted in step 1040 to achieve the desired visualization. The image is then reassembled in step 1060 and converted from optical density space in step 1070 to allow visualization by the user.

[0049] Figure 11 illustrates an embodiment of a laboratory workflow for producing a mixed assay result of combined virtual staining and traditional assay according to the present disclosure. The workflow begins at step 1101, wherein the tissue section mounted on the slide is dewaxed (dewaxed), covered with a coverslip and subjected to fluorescence imaging. In step 1102, QC is performed on the image and the tissue to ensure that the image meets the appropriate standards for virtual staining. If the image is rejected during QC, the sample can be re-imaged or reprocessed in step 1103 to correct any defects, but it will not be passed on for traditional staining. If the image passes QC, the coverslip is removed in step 1104 to prepare the sample for traditional assay. In parallel with step 1104, a "can be stained" flag is set in the staining protocol database 1108 in step 1105, which allows the execution of an automatic staining protocol. After removing the coverslip, the slide is loaded onto the automatic staining platform to perform traditional assays in step 1106. The barcode on the slide is read in step 1107, which is referenced in the staining protocol database 1108, and the "OK TO STAIN" flag is checked. If the "OK TO STAIN" flag is not set, an error code is displayed to the user in step 1110, and conventional staining of the slide is not allowed. If the "OK TO STAIN" flag is set, the conventional staining protocol is downloaded to the automated staining platform in step 1111, and the platform executes the conventional staining protocol in step 1112. At the end of the conventional staining assay, a coverslip is applied, and the assay results are imaged and digitized in step 1114. The result of this conventional assay process is a digitized image of the result in step 1115, suitable for combination with the virtual staining output in step 1116.

[0050] At the same time, the staining protocol database 1108 publishes a set of virtual stainings, which include the virtual multiplexed staining results. This input is combined with the pre-processing of the autofluorescence image in step 1117, and the inference-based multiple virtual staining is performed at step 1118. An example method of step 1118 is Figure 1-6In step 1119, the multiplexed virtual staining result is returned from the system. This result is combined with a digitized version of the traditional staining result in step 1116, an example embodiment of which is disclosed in FIG. 12 below.

[0051] FIG. 12 illustrates an embodiment of an unmixing and remixing process for digitally combining virtual staining results and traditional staining results into a hybrid multiplex suitable for analysis or visualization according to the present disclosure. In step 1201, the unmixed virtual multiple IHC biomarker staining is presented to the system. In step 1202, the unmixed traditional IHC biomarker staining is presented to the system. Step 1205 uses the outputs of steps 1201 and 1202 to perform registration between the two images. The purpose of the registration step is to provide a common reference frame for superimposing images so that pixels with virtual staining results are aligned with the same pixels containing traditional staining results. At the same time, the virtual counterstaining image collected in step 1204 is prepared for registration with the unmixed nuclear counterstaining image of the traditional determination in step 1203. In step 1206, the virtual nuclear counterstaining images are combined into a single image in a linear combination. Gamma correction is applied in step 1207 to rebalance the image, and the result is a coordinated single virtual nuclear counterstaining virtual image in step 1208. The results of step 1208 are combined with the output of step 1203 in step 1209 where registration is performed to align pixels between the conventional and virtual nuclear counterstains. A single image is created from the pair in step 1209 in step 1210 where a linear combination function is applied. The images are balanced again using gamma correction in step 1211 and a coordinated single nuclear counterstain image is produced in step 1212.

[0052] In parallel with step 1212, the output of step 1205 is a stack of unmixed virtual and traditional biomarker stains seen in step 1213, which can be combined into an image stack, including a single nuclear counterstain image in step 1214. This fully unmixed set of traditional and virtual biomarker stains and counterstains is suitable for further image analysis using traditional methods or linear remixing to visualize the combined staining pattern.

[0053] Fig.13An embodiment of a computational workflow for a machine learning model or network described herein using virtual staining to generate annotations is shown. In step 1301, an image of an unlabeled tissue slice is presented to a machine learning model. In step 1302, a virtual staining algorithm is applied to convert the image of the unlabeled tissue slice into a stained tissue. In step 1303, the stained tissue is annotated to extract additional information, including but not limited to multiplexed immunofluorescence, FISH, proteomics or sequencing data. Annotation can be performed at multiple levels, depending on the information extraction modality of the annotation, including but not limited to pixel level, tile level or entire slide level. If applicable, virtual staining of specific biomarkers can also be used to enhance the tissue. In step 1304, the annotation is parsed using a threshold operator or another classifier. In step 1305, the virtual stained tissue from step 1302 is superimposed with the annotation parsed in step 1304. The process utilizes spatial matching between annotations and data from steps 1302 and 1303 to extract additional information, including but not limited to segmentation or classification.

[0054] Fig.14 1400 is a block diagram of a computer system 1400 that is arranged to perform processing associated with a neural network based on inference virtual coloring as described herein. The exemplary computer system 1400 includes a central processing unit (CPU) 1402, a memory 1404, and an interconnect bus 1406. The CPU 1402 may include a single microprocessor or multiple microprocessors or a dedicated processor for configuring the computer system 1400 as a multiprocessor system. The memory 1404 illustratively includes a main memory and a read-only memory. The computer 1400 also includes a mass storage device 1408, such as a variety of disk drives, tape drives, etc. The memory 1404 also includes a dynamic random access memory (DRAM) and a cache memory. In operation, the memory 1404 stores at least a portion of the instructions and data executed by the CPU 1402. The memory 1404 may also include a computing element, such as a deep memory architecture (DIMA), where data is sent to the memory and the function of the data (e.g., matrix vector multiplication) is read out by the CPU 1402.

[0055] The mass storage system 1408 may include one or more disks, optical drives, and / or solid-state memory for storing data and instructions for use by the CPU 1402. At least one component of the mass storage system 1408, preferably in the form of a non-volatile disk drive, solid-state drive, or tape drive, stores a database for processing data and controlling the neural network functions for inference-based virtual coloring. The mass storage system 1408 may also include one or more drives for various portable media, such as floppy disks, flash drives, optical disk read-only memories (CD-ROM, DVD, CD-RW, and variants thereof), memory sticks, or integrated circuit non-volatile memory adapters (i.e., PC-MCIA adapters), for inputting and outputting data and code to and from the computer system 200.

[0056] The computer system 1400 may also include one or more input / output interfaces for communication, for example, as an interface 1410 and / or a transceiver for data communication via a network 1412. The data interface 1410 may be a modem, an Ethernet card, or any other suitable data communication device. In order to provide the functionality of a processor running a neural network based on inference virtual coloring, the data interface 1410 may provide a relatively high-speed link to a network 1412 (such as an intranet, the Internet, or the Internet) directly or through another external interface. The communication link to the network 1412 may be, for example, optical fiber, wired, or wireless (e.g., via a satellite or cellular network). The computer system 1400 may also be connected to at least one other computer system via the data interface 1410 and the network 1412 to perform remote or distributed multi-sensor processing associated with, for example, a common operational picture (COP). Alternatively, the computer system 1400 may include a mainframe or other type of host computer system capable of Web-based communication via the network 1412. Computer system 1400 may include software for operating network applications, such as a Web server and / or a Web client.

[0057] The computer system 1400 may also include appropriate input / output ports that may interface with a portable data storage device or interconnect with a local display 1416 and keyboard 1414, etc. using the interconnect bus 1406, for use as a local user interface for programming and / or data retrieval purposes. The display 1416 may include touch screen functionality to enable a user to interface with the system 1400 by touching certain portions of the surface of the display 1416. A server operator may interact with the system 1400 from a remote terminal device via the network 1412 to control and / or program the system.

[0058] Computer system 1400 may run various applications and store associated data in a database on mass storage system 1408. One or more such applications may include a neural network for inference-based virtual coloring, such as Figure 1-13 described.

[0059] The components included in computer system 1400 may enable the computer system to function as a server, workstation, personal computer, network terminal, mobile computing device, mobile phone, system on chip (SoC), etc. System 1400 may include software and / or hardware to implement a Web server application. The Web server application may include software such as HTML, XML, WML, SGML, PHP (Hypertext Preprocessor), CGI, and similar languages.

[0060] The above features of the present disclosure can be implemented as software components running in system 1400, wherein system 1400 includes Unix workstations, Windows workstations, LINUX workstations or other types of workstations. Other operating systems can be used, such as but not limited to Windows, MAC OS and LINUX. In some aspects, the software can be optionally implemented as a C language computer program, or a computer program written in any high-level language including but not limited to Javascript, Java, CSS, Python, Keras, TensorFlow, PHP, Ruby, C++, C, Shell, C#, Objective-C, Go, R, TeX, VimL, ​​Perl, Scala, CoffeeScript, Emacs Lisp, Swift, Fortran or Visual BASIC. Certain script-based programs such as XML, WML, PHP, etc. can be used. System 200 can use a digital signal processor (DSP).

[0061] As previously mentioned, mass storage 1408 may include a database. The database may be any suitable database system, including the commercially available Microsoft Access database, and may be a local or distributed database system. The database system may implement Sybase and / or SQL Server.

[0062] The database may be backed by any suitable persistent data storage, such as a hard drive, a RAID system, a tape drive system, a floppy disk, or any other suitable system. The system 1400 may include a database integrated with a neural network for inference-based virtual coloring, however, it is understood that in other implementations, the database and mass storage 1408 may be external components.

[0063] In some implementations, the system 1400 may include an Internet browser program and / or be configured to operate as a network server. In some configurations, the client and / or web server may be configured to identify and interpret various network protocols that the client or server program may use. Commonly used protocols include, for example, Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Telnet, Secure Sockets Layer (SSL), and Transport Layer Security (TLS). However, new protocols and revisions of existing protocols may be introduced frequently. Therefore, in order to support new or revised protocols, new revisions of server and / or client applications may be continuously developed and released.

[0064] In one implementation, the neural network includes a network-based (e.g., Internet-based) application (which may be configured and run on the system 1400), and / or any combination of other components of the neural network, for inference-based virtual coloring. The computer system 1400 may include a web server running a Web 2.0 application or the like. The web application running on the neural network may use a server-side dynamic content generation mechanism such as, but not limited to, Java servlets, CGI, PHP, or ASP. In some implementations, the mixed content may be generated by, for example, a web browser running a client-side script (including, but not limited to, JavaScript and / or an applet on a wireless device).

[0065] In some implementations, a neural network or computer system 1400 for inference-based virtual coloring may include applications that employ asynchronous JavaScript + XML (Ajax) and similar technologies, which use asynchronous loading and content rendering techniques. These technologies may include, but are not limited to, XHTML and CSS for style rendering, Document Object Model (DOM) APIs exposed by Web browsers, asynchronous data exchange of XML data, and Web browser-side scripts, such as JavaScript. Certain Web-based applications and services may utilize Web protocols, including, but not limited to, Service-Oriented Access Protocol (SOAP) and Representational State Transfer (REST). REST may utilize HTTP with XML.

[0066] The neural network, computer system 1400, or another component of the neural network for inference-based virtual coloring can also provide enhanced security and data encryption. The enhanced security can include access control, biometric authentication, password authentication, message integrity checking, encryption, digital rights management services, and / or other similar security services. Security can include protocols such as IPSEC and IKE. Encryption can include, but is not limited to, DES, 3DES, AES, RSA, ECC, and any similar public or private key based schemes.

[0067] It will be appreciated by those skilled in the art that, in alternative embodiments, the functions of several elements may be implemented by fewer elements or a single element. Similarly, in some embodiments, any functional element may perform fewer or different operations than those described for the illustrated embodiments. In addition, functional elements shown as different for illustrative purposes may be incorporated into other functional elements in a particular implementation (e.g., any combination of executable functional elements such as modules, databases, interfaces, computers, servers, etc.).

[0068] Although the subject technology has been described with respect to preferred embodiments, it will be readily appreciated by those skilled in the art that various changes and / or modifications may be made to the subject technology without departing from the spirit or scope of the subject disclosure. The appended claims are exemplary and may be combined and arranged in any manner, including with multiple dependent items, etc.

Claims

1. A method for generating inference-based virtually stained image annotations, comprising: providing one or more neural networks executed by image processing software running on one or more processors of a computing device; training the one or more neural networks to recognize one or more virtual staining patterns using a plurality of images of chemical staining of one or more endogenous signals; obtaining data corresponding to the biological sample; obtaining and generating an image of the biological sample including one or more endogenous signals identified by the annotation technique; detecting the one or more virtual staining patterns in the image of the biological sample using the one or more neural networks; as well as The virtual staining patterns detected in the image of the biological sample are superimposed using a spatial matching technique to create an inference-based virtual staining image annotation.

2. The method according to claim 1, wherein: The annotations include features used by the one or more neural networks to perform semantic segmentation.

3. The method according to claim 1, wherein: The obtaining and generating the image of the biological sample comprises sequencing or imaging mass spectrometry techniques.

4. The method according to claim 1, wherein: The obtaining and generating the image of the biological sample may include immunohistochemistry or immunofluorescence techniques.

5. The method according to claim 4, wherein: The immunohistochemical or immunofluorescence technique involves direct comparison of virtual staining patterns of two or more antibody clones on the same tissue section.

6. The method according to claim 4, wherein: The immunohistochemistry or immunofluorescence techniques include: Create unique virtual polyclonal antibody stainings from multiple monoclonal antibodies and titrate to desired expression levels.

7. The method according to claim 1, wherein: The opacity level of the virtual staining pattern can be adjusted to enable focusing on specific antibody clones.

8. The method according to claim 1, wherein: The virtual staining pattern presentation allows for quantitative analysis of pattern differences.

9. The method of claim 1, further comprising individually manipulating the virtual dye patterns of the superimposed virtual dye patterns.

10. The method according to claim 9, wherein: The manipulation involves adjusting the intensity of each virtual staining pattern in a real-time process.

11. The method of claim 1, further comprising multiplexing the superimposed virtual staining pattern with existing virtual staining or traditional assay readouts.

12. A method for generating virtual stained image annotations, comprising: obtaining images of biological samples; virtually staining the biological sample using a machine learning algorithm executed via a computer program running on a processor, the machine learning algorithm detecting a virtual staining pattern of an endogenous signal in the biological sample; annotating the virtually stained biological sample for biomarkers; parsing the annotation of the virtually stained biological sample; and The virtually stained biological sample is overlaid with the parsed annotations.

13. The method according to claim 12 further comprises performing semantic segmentation on the virtually stained biological sample.

14. The method of claim 12, further comprising training the machine learning algorithm using a plurality of virtual staining patterns of endogenous signals.

15. A method for generating an inference-based virtual stained image, comprising: providing a neural network executed by image processing software running on one or more processors of a computing device; training the neural network to recognize one or more virtual staining patterns using a plurality of images of chemical staining of one or more endogenous signals; obtaining and generating images of biological specimens; detecting, using the neural network, the virtual staining pattern in the image of the biological sample using a trained neural network; parsing the endogenous signal of the detected virtual staining pattern using the neural network; as well as Outputs two separate virtual images for each corresponding intrinsic signal, Therein, the two separate virtual images describing the resolved endogenous signal may be combined and recombined to selectively generate one or more new multiplexed virtual images.

Images