System and method for generating stained images

Abstract

A method for generating a staining image comprises the following steps: obtaining a set of first images of a key sample slice; and processing the first images with a multimodal staining learning engine to generate at least one staining image. The staining image represents the key sample slice stained with at least one staining agent. Advantageously, the tissue can be observed using a virtual tissue staining method by simply imaging the sample tissue with a digital microscope, without staining the tissue, or with only a fast and low-cost staining agent, and then these images are input into a computer program to generate images of the same tissue under other staining agents.

Description

Technical Field

[0001] The present invention relates to a system and method for generating a staining image, and in particular (but not exclusively) to a system and method for generating a virtual staining image of a tissue sample slice. Background Art

[0002] The practice of histology is an important area of ​​medical science that aids in medical research, diagnosis, and treatment. Part of this practice involves preparing slides containing sections of tissue. These slides, in turn, can be placed under a microscope for further study or analysis.

[0003] Staining tissues for contrast when viewed under a microscope is an integral part of pathology. Tissue sections can be stained using stains or colorants in different situations, such as by soaking the tissue in a chemical or antibody solution before microscopic examination. These stains allow certain cells, features, or structures to become more apparent when the slide is viewed or considered under magnification. However, physical staining methods can be time consuming and expensive. In addition, they can cause workplace and environmental contamination. Summary of the invention

[0004] According to a first aspect of the present invention, there is provided a method for generating a dyed image, comprising the following steps:

[0005] - obtaining a set of first images of a critical sample section; and

[0006] - Processing the first image with a multimodal stain learning engine to generate at least one stain image, wherein the stain image represents the key sample section stained with at least one stain.

[0007] In one embodiment of the above first aspect, the key sample sections are unstained or stained with an input stain.

[0008] In one embodiment of the first aspect above, the method further comprises the step of obtaining an image of each of at least three adjacent sample slices, the three adjacent sample slices including the key sample slice and an adjacent sample slice obtained near the key sample slice.

[0009] In one embodiment of the first aspect above, one of the at least three adjacent sample slices is stained with an output stain, and the remaining adjacent sample slices are stained with an input stain.

[0010] In one embodiment of the first aspect above, the stained image represents the key sample slice stained with the output stain.

[0011] In one embodiment of the first aspect above, the remaining adjacent sample sections are stained with multiple types of input stains in an alternating manner.

[0012] In one embodiment of the first aspect above, the multimodal staining learning engine includes a machine learning network configured to generate the at least one staining image.

[0013] In an implementation method of the first aspect above, the machine learning network includes a generator network, which is configured to generate the at least one stained image within a plurality of cycles; and a discriminator network, which is configured to analyze the at least one stained image to provide feedback to the generator network at each of the plurality of cycles.

[0014] In one embodiment of the first aspect above, the machine learning network is a generative adversarial network.

[0015] In one embodiment of the first aspect above, the generative adversarial network is trained using images of the at least three adjacent sample slices stained with the input stain and the output stain.

[0016] In one embodiment of the first aspect above, the generative adversarial network is further trained using images of the at least three adjacent sample slices imaged under different lighting conditions.

[0017] In one embodiment of the first aspect above, the lighting conditions include bright field or autofluorescence.

[0018] In one embodiment of the first aspect above, each of the input stain and the output stain comprises at least one of hematoxylin-eosin (H&E) and a special stain.

[0019] In one embodiment of the first aspect above, the special stain comprises an immunostain.

[0020] In one embodiment of the above first aspect, the at least three adjacent sample slices are frozen.

[0021] In one embodiment of the above first aspect, the at least three adjacent sample sections are formalin-fixed and embedded in paraffin.

[0022] In one embodiment of the first aspect above, the method further includes a step of using an estimation process to correct a spatial mismatch between a staining image representing the key sample slice stained with the output stain and a staining image representing the adjacent sample slice stained with the input stain.

[0023] In one embodiment of the first aspect above, the step of correcting the spatial mismatch includes aligning all staining images of the adjacent sample slices stained with the input stain with the staining image representing the key sample slice stained with the output stain using two-dimensional image registration.

[0024] In an embodiment of the above first aspect, the estimation process is an interpolation method.

[0025] In one embodiment of the first aspect above, the estimation process is an extrapolation method, which further includes the step of applying an image extrapolation transformation to generate at least one estimated input image based on a stained image of the adjacent sample slice stained with the input stain.

[0026] According to a second aspect of the present invention, there is provided a system for generating a stained image, comprising:

[0027] - an image gateway configured to obtain a set of first images of a key sample slice; and

[0028] - an image generator configured to process the set of first images with a multimodal stain learning engine to generate at least one stain image, wherein the at least one stain image represents the key sample section stained with at least one stain.

[0029] In one embodiment of the above second aspect, the key sample sections are unstained or stained with an input stain.

[0030] In one embodiment of the second aspect above, the image gateway is further configured to obtain an image of each of at least three adjacent sample slices, the three adjacent sample slices including the key sample slice and an adjacent sample slice obtained near the key sample slice.

[0031] In one embodiment of the second aspect, one of the at least three adjacent sample sections is stained with an output stain, and the remaining adjacent sample sections are stained with an input stain.

[0032] In one embodiment of the second aspect above, the at least one stained image represents the key sample section stained with the output stain.

[0033] In one embodiment of the second aspect above, the remaining adjacent sample sections are stained with multiple types of input stains in an alternating manner.

[0034] In one embodiment of the second aspect above, the multimodal staining learning engine includes a machine learning network configured to generate the at least one staining image.

[0035] In one embodiment of the second aspect above, the machine learning network includes a generator network, which is configured to generate the at least one stained image within a plurality of cycles; and a discriminator network, which is configured to analyze the at least one stained image to provide feedback to the generator network at each of the plurality of cycles.

[0036] In one embodiment of the second aspect above, the machine learning network is a generative adversarial network.

[0037] In one embodiment of the second aspect above, the generative adversarial network is trained using images of the at least three adjacent sample slices stained with the input stain and the output stain.

[0038] In one embodiment of the second aspect above, the generative adversarial network is further trained using images of the at least three adjacent sample slices imaged under different lighting conditions.

[0039] In one embodiment of the second aspect above, the lighting conditions include bright field or autofluorescence.

[0040] In one embodiment of the second aspect above, each of the input stain and the output stain comprises at least one of hematoxylin-eosin (H&E) and a special stain.

[0041] In one embodiment of the second aspect above, the special stain comprises an immunostain.

[0042] In one embodiment of the above second aspect, the at least three adjacent sample slices are frozen.

[0043] In one embodiment of the above second aspect, the at least three adjacent sample sections are formalin-fixed and embedded in paraffin.

[0044] In one embodiment of the above-mentioned second aspect, the method further includes a step of using an estimation process to correct the spatial mismatch between the staining image representing the key sample section stained with the output stain and the staining image of the adjacent sample section stained with the input stain.

[0045] In one embodiment of the second aspect above, the step of correcting the spatial mismatch includes aligning all staining images of the adjacent sample slices stained with the input stain with the staining image representing the key sample slice stained with the output stain using three-degree-of-freedom rigid body registration.

[0046] In an embodiment of the second aspect, the multimodal staining learning engine is further configured to apply an image transformation to generate at least one estimated input image based on all staining images of the adjacent sample slices stained with the input stain. Therefore, embodiments of the present invention provide many benefits. Advantageously, tissue can be observed using virtual tissue staining, simply by imaging the sample tissue with a digital microscope, without staining the tissue, or staining it only with a fast and low-cost stain, and then these images are input into a computer program to generate images of the same tissue under other stains. Virtual staining may be particularly advantageous for time-consuming or expensive staining using traditional methods. A good example is immunostaining. Virtual staining can also reduce contaminants, promote digital medicine, and reduce hospital congestion.

[0047] Furthermore, multimodal image synthesis is a great improvement over single-input, single-output methods because it allows multiple input images to generate an output image. In addition, it can also improve the accuracy of image generation.

[0048] Furthermore, multimodal image synthesis may exploit its robustness better than existing pathology methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0050] Figure 1 is a schematic block diagram of a computer system that operates as a system for generating a staining image according to one embodiment of the present invention.

[0051] Figure 2 It is a display Figure 1 Block diagram of the operating process of the system for generating staining images.

[0052] Figure 3 It is a display Figure 2 Block diagram of the operation of an example machine learning network of the system for generating staining images.

[0053] Figure 4 It is a schematic diagram of a correction process performed by interpolation or extrapolation when input and output images are obtained at different positions of a plurality of sample slices being prepared.

[0054] Figure 5 It is used for display Figure 1 A processing flow chart of an example collection of method steps for generating a stained image using a system for generating a stained image.

[0055] Figure 6The estimated image set input to the neural network (bright field, GFP and RFP images of H&E stained sections) and the comparison between the virtually stained images and the physically stained images are shown. DETAILED DESCRIPTION

[0056] Reference Figure 1 , which shows an embodiment according to the present invention. The embodiment is configured to provide a system for generating a stained image, comprising:

[0057] - an image gateway configured to acquire a first image of a critical sample slice; and

[0058] - an image generator configured to process the first image using a multimodal stain learning engine, the multimodal stain learning engine configured to generate at least one stain image, wherein the at least one stain image represents the key sample section stained with at least one stain.

[0059] In this exemplary embodiment, the image gateway and image generator are implemented by a computer. The computer has an appropriate user interface, communication port, and processor. The computer can be implemented by any computing architecture, including a stand-alone personal computer, a client / server architecture, a "dumb" terminal / host architecture, a portable computing device, a tablet computer, etc. A computer, a wearable device, a smart phone, or any other appropriate architecture. The computing device can be appropriately programmed to implement the present invention.

[0060] Please refer to Figure 1 , which shows a schematic diagram of a computer system, which in this embodiment includes a server or computing device 100. The server 100 includes suitable components for receiving, storing and executing appropriate computer instructions. These components may include a processing unit 102, a read-only memory (ROM) 104, a random access memory (RAM) 106, and input / output devices such as a disk drive 108, an input device 110 such as an Ethernet port, a USB port, etc. A display 112, such as a liquid crystal display, a light-emitting display, or any other suitable display and a communication link 114. The server 100 contains instructions that may be included in the ROM 104, the RAM 106, or multiple disk drives 108, and these instructions can be executed by the processing unit 102. A plurality of communication links 114 may be provided, which may be connected to one or more computing devices such as servers, personal computers, terminals, and wireless or handheld computing devices such as tablet computers, smart phones, or wearable devices. At least one of the plurality of communication links may be connected to an external computing network via a telephone line or other type of communication link.

[0061] The server may include a storage device, such as a disk drive 108. The disk drive 108 may include a solid-state drive, a hard disk drive, an optical drive, or a tape drive. The server 100 may use a single disk drive or multiple disk drives. The server 100 may be able to access one or more remote data storage devices, databases, or cloud-based data center systems 120 to access information, retrieve data, or obtain or provide commands or instructions. The server 100 may also have a suitable operating system 116, which resides on the disk drive of the server 100 or in ROM.

[0062] In the present exemplary embodiment, the server or computing device 100 may be implemented as a system for generating a stained image, which can be used to generate a virtual or simulated image of a tissue sample section that has been stained with any one or more of a variety of available stains (or colorants) to enable medical personnel, scientists, or researchers to identify specific features or markers in the tissue sample section.

[0063] In some examples of arrangements, the above system is configured to be used with hospital or laboratory equipment to process tissue samples. The system can be connected to or configured to communicate with an image source, such as an optical or electron microscope or a tissue image database, to obtain an image of a tissue sample. Once the image of the tissue sample is obtained, the system is arranged to process the image to generate a new image of the tissue sample, but the new image is simulated as being stained by one or more stains desired by the user. These different stains include standard stains or stains, such as hematoxylin and eosin (H&E), or non-standard stains or stains such as special stains and / or immunostains. Preferably, the system operates to generate a virtual or generated stain image of a tissue sample slice, which is different from the image of the first stain of the tissue sample. Therefore, as an example, if the first image is an unstained tissue sample, the generated stain image can be a tissue sample stained by a desired stain such as H&E. In an alternative example, if the first image is a tissue sample stained with a first stain such as H&E, the generated stain image can be a tissue sample stained with another different stain such as an immunostain.

[0064] These simulated or emulated images of tissue sample slices stained with one or more stains or colorants can be provided to the user in turn to allow the user to make a diagnosis or provide a medical or biological assessment. The generation of these simulated or emulated images may be advantageous because the staining process may change the physical properties of each tissue sample slice. This in turn means that if a stain is used on an example tissue sample slice, another stain may not be used, thereby limiting the purpose of the tissue sample slice for research or diagnosis. Therefore, the ability to generate these simulated or emulated images will enable users to obtain images of tissue sample slices stained with various stains. By generating images of simulated and emulated stained tissue sample slices, the images of tissue sample slices can be simulated as being stained by various stains, thereby providing greater room for research or diagnosis purposes. This will in turn improve research and development in biological and medical sciences and medical or veterinary treatment. In addition, the example of the present system being used to generate stained images may also be advantageous because the speed of generating stained images is much faster than physical staining. This provides significant advantages for pathology practices that require time-sensitive analysis of tissue samples.

[0065] Reference Figure 2 , a block diagram illustrating the flow of an exemplary embodiment of a system for generating a stained image 200 is shown. In this exemplary embodiment, the system 200 first acquires images of frozen tissue sections 202. These tissue sections, which may also be referred to as key tissue sections or key sample sections, may be unstained or stained with one or more stains, such as hematoxylin and eosin (H&E) or any other stain.

[0066] It can be seen that these images 202 are input to the system through the image gateway 204. The image gateway 204 can be a suitable input device and can be configured to receive image data from various sources, including from an optical or electron microscope, a database, a computer network, a computer cloud server, a portable storage device or other electronic, processing or storage device. The input image 202 can be one or more images of a tissue sample slice, with or without a stain. Images of adjacent samples, with or without various stains, can also be optionally input into the image gateway 204. This may be necessary or optional, depending on the user's intentions and the level of pre-training of the image generator 206, which will refer to Figure 2 and 3 This is described in further detail below.

[0067] As shown in this embodiment, the image generator 206 is configured to receive the image 202 of the tissue sample slice and regenerate a new image of the tissue sample slice 208 that has been stained with one or more stains. Preferably, this is performed by a machine learning processor (e.g., Figure 3 306, such as a neural network, a convolutional neural network (CNN), or a generative adversarial network (GAN) in an exemplary embodiment as described below.

[0068] The machine learning network 306 is configured to be trained with a plurality of training images of tissue sample slices and stained tissue sample slices. In turn, the trained machine learning network will be able to receive the tissue sample slice image as input and generate a simulated image of the tissue sample slice shown in the tissue sample slice image, but stained with various stains. This generated simulated image of the tissue sample slice can then be output to the user for further processing, or to retrain the machine learning network 306 if necessary.

[0069] Reference Figure 3 , in which a block diagram of an example machine learning processor having a machine learning network 306 is shown. Figure 3 The block diagram in is used to show an example operation of a machine learning network 306 to generate one or more simulated images 308. The machine learning network can first be trained with a suitable training data set, which can include, for example, a number of images of tissue sample slices and adjacent tissue sample slices stained with various stains. Once the network 306 has been properly trained to provide acceptable accuracy, the image 304 of the tissue sample slice 302 can be fed into the network to generate an image of the tissue sample slice, simulating staining with one or more stains.

[0070] For example, tissue samples come from clinical situations, such as biopsies and surgical resections. Tissue can come from any organ and be normal or diseased. The disease may be different types of cancer, but other diseases that require histopathological diagnosis may also be suitable. Tissue sections can be prepared using formalin fixation / paraffin embedding or freezing methods.

[0071] A separate neural network can be trained for each virtually stained disease, since different diseases involve different tissue types and are diagnosed using different preparation methods and stains. For example, similar tissue samples (for the same disease) can be collected from different patients and hospitals to reflect the diversity of populations and preparation methods. However, it is preferred that different hospitals use similar diagnostic methods (i.e., the same staining).

[0072] like Figure 3As shown, the input image 304 here includes an image of a key tissue sample slice (K1), which can be one of various key tissue sample slices from one or more tissue sample sets 302. A1 and A2 are images of adjacent tissue sample slices, which are tissue sample slices adjacent to the key tissue sample slice. It should be noted that adjacent tissue slices are not necessarily physically adjacent, but they can be in a close position, or any other position. Therefore and preferably, some structural or other similarities can be established with the key tissue sample slice, so that the morphology between the two sample slices can share some similarities.

[0073] Without wishing to be bound by theory, tissue sections should be compatible with the multimodal neural network during training. Once training is complete, the sectioning step should follow standard procedures for the disease to avoid disrupting hospital procedures.

[0074] Preferably, the training process can start with obtaining an image of each of at least three adjacent sample slices. The at least three adjacent sample slices include a key sample slice and an adjacent sample slice obtained near the key sample slice. For example, the three adjacent samples include the key sample slice and the above two adjacent sample slices, including K1, A1 and A2 assigned in the order shown in the figure, or the adjacent three sample slices are arranged in other possible orders as long as they are close to each other.

[0075] During training, at least three directly adjacent slices should be cut to standard thickness (approximately 10 microns)

[0076] More slices may improve the final network performance, and these slices are preferably adjacent, because adjacent slices are very similar in structure, which will allow the network to generate new coloring without obvious structural confusion.

[0077] Preferably, one of at least three adjacent sample slices can be stained with the output stain, and the remaining adjacent sample slices can all be stained with the input stain. Accordingly, at least one stained image, or output stained image, represents a key sample slice stained with the output stain.

[0078] For example, all slices except one are stained with hematoxylin and eosin (H&E) because H&E is routinely used in almost all histopathology scenarios. H&E is the input stain, and the trained network only needs one H&E-stained slice for future applications. The slices to be stained are stained with the target output stain, such as an immunostain. Since a slice cannot usually be stained more than once, it is generally necessary to stain individual slices with either the input or output stain.

[0079] Alternatively, all but one of the sample sections may be stained with multiple types of input stains in an alternating fashion. For example, if stains other than H&E are commonly used for diagnosis of a disease and these stains can be performed quickly, then sections may be stained alternately with H&E and the other stains.

[0080] Reference Figure 3 , these adjacent tissue sample slices A1, A2 can also be stained, such as with the desired staining strategy described above for simulating the key tissue sample slice being stained with the same stain. In turn, the images of these adjacent tissue sample slices A1, A2 will therefore present a structure similar to the key tissue, but stained with one or more different stains at the same time. Therefore, on its basis, these adjacent images A1, A2 will provide information or guidance on how K1 should be displayed when the key tissue sample slice is also stained with one or more stains, and this information can be used as knowledge in the machine learning network to generate simulated images of the key tissue sample slice after being stained with alternative stains. When the key tissue sample slice K1 is stained, for example, with a stain different from that used to stain the adjacent tissue samples A1 and A2, then the images of these key tissue sample slices K1 or adjacent tissue sample slices A1, A2 can be obtained in various ways, including directly obtaining the output of an optical / electron microscope, or obtaining these images as a collection from a storage or computing device or a telecommunications device. In an example usage scenario, adjacent tissue slices A1, A2 may be tissue slices adjacent to the key sample slice K1, and therefore may be obtained from a position close to the key sample slice K1. As an example of these adjacent tissue slices, the adjacent tissue slices may include the previous tissue slice or the next tissue slice. A unique property of these adjacent tissue slices is that they may represent tissue structures similar to the key sample because they are close to or directly adjacent to the key sample slice. Therefore, the similarity of these tissue structures can be used to generate virtual staining of the key sample slices.

[0081] In this exemplary procedure, these adjacent tissue slices A1, A2 may also be stained with other stains, such as special stains and / or immunostains. In turn, the images of these adjacent tissue slices, also referred to as adjacent images, may thus show an image of a tissue sample slice K1, whose tissue structure is similar to the key sample, but stained with a different stain. This will in turn provide a basis for the similarity or relationship between the key tissue sample K1, which is unstained or stained with a first stain, and the adjacent tissue sample A1 or A2 stained with another stain.

[0082] These image sets 304 from multiple samples 302 can in turn be used to train a stain learning engine, which can include a machine learning network 306 or processor. After training such an engine, the machine learning network 306 or processor can take unstained images as input and output images of the same tissue section stained with a desired stain 308.

[0083] Additional or stained (e.g., H&E) images may also be used as input, and the system may output images of the same tissue section stained with different stains. This, in turn, allows a medical professional, scientist, or researcher to obtain a tissue sample stained with one of a variety of available stains, such as H&E, as well as other non-standard or alternative stains (e.g., special stains or immunostains). These virtual images, in turn, may allow the scientist or medical professional to diagnose a medical condition, detect various results or further research, or may be presented to an end user for other scientific, analytical, storage, processing, or diagnostic purposes.

[0084] Additionally, sample tissues can optionally be imaged under different illumination conditions, including but not limited to bright field and autofluorescence. Advantageously, by providing different illumination conditions and / or different input stains, a multimodal staining learning engine can be established to facilitate more input / output requirements in different applications.

[0085] The stained sections can be imaged using a standard digital light microscope found in a pathology laboratory. For example, a Nikon Eclipse Upright microscope or an Olympus IX81 microscope can be used. H&E sections are imaged using a bright field filter. Optionally or alternatively, sample sections can be imaged using green fluorescent protein (GFP) and / or red fluorescent protein (RFP) filters to obtain autofluorescence images to enable the multimodal capabilities of the trained network. Other input stained sections (non-H&E) can be imaged using their specified filters. Similarly, output stained sections can also be imaged using their specified filters. For network training, large sections can be imaged using a calibrated scanning stage microscope to obtain more data. For network applications, obtaining a live view is sufficient to reduce image acquisition time. Finally, it may be necessary to save the acquired images in the original image format of the microscope rather than in a compressed format such as .jpeg. The inventors of the present invention have found that it is more desirable not to make any image browsing adjustments, such as adjustments to contrast and brightness before saving the image file.

[0086] In order to further improve the accuracy or performance of the multimodal staining learning engine, the multimodal staining learning engine is preferably further configured to use an estimation process to correct the spatial mismatch between the staining image representing the key sample slice stained with the output stain and the staining image of the adjacent sample slice stained with the input stain.

[0087] The inventors of the present invention have conceived that in virtual tissue staining, especially when using multimodal methods, the input and output images used for training can be obtained from different tissue slices, although these different tissue slices are very close and very similar to each other. Therefore, there may be spatial mismatches in the images of each pair of adjacent sample slices. If these mismatches are not corrected, it may result in blurred images, which may make the output images unsuitable for clinical use.

[0088] Preferably, an estimation process can be used to estimate what the output slice would look like if stained with the input stain. The estimated input image is then used for network training along with the obtained output image.

[0089] Reference Figure 4 , if the output slice is between adjacent input slices, an estimate can be made using interpolation. Interpolation is preferred because the estimate is likely to be more accurate. However, some clinical situations allow output slices to be made only on one side of all input slices. In this case, extrapolation is used (see below). It should be noted that the estimate is made for each input staining image (e.g., bright field, GFP, RFP) separately.

[0090] Preferably, interpolation is performed by registering all input images to the output image using two-dimensional image registration, or three degrees of freedom rigid body registration (two translations, one rotation). This takes into account the fact that any slice can be placed anywhere on the slide at any angle, but the shape is fixed. The objective function is a function suitable for inter-modality registration, such as mutual information. After registration, slice interpolation is applied to the aligned input images to estimate the image of the output slice stained with the input stain. The resulting image is then referenced as the estimated input image.

[0091] Note that interpolation may require knowledge of the thickness and order of the slices, which may be known if the images were acquired specifically for network training. Alternatively, cutting directly adjacent slices, as might be done in histopathology, may lead to better estimates.

[0092] In addition, the estimation process may be an extrapolation method. Although extrapolation may result in worse estimates than interpolation, sometimes the training data comes from clinical scenarios where extrapolation is needed. For example, surgical resections are often cut into blocks and processed in different ways, with one block stained with H&E and another block stained with an immunostain. For extrapolation, similar to interpolation, all input images can be aligned to the output image using a rigid body registration method. After registration, the extrapolation method further includes the step of applying an image extrapolation transform to generate at least one estimated input image based on a stained image of an adjacent sample slice stained with the input stain. Examples of image extrapolation transforms are discrete orthogonal transforms or learning-based transforms.

[0093] Note that the extrapolation method also requires the system to know the thickness and ordering of the sections, and directly cutting adjacent sections will lead to better estimates.

[0094] After all estimations are complete, each input colorant should have an estimated image that spatially matches the output image.

[0095] exist Figure 5 In an exemplary embodiment shown in FIG. 1 , the inventors of the present invention conducted an experimental operation on an example of the above system, that is, generating a stained image by performing five exemplary steps (500, 502, 504A / 504B, 506, and 508). The results show that the system for generating a staining network can cooperate with existing hospital or laboratory procedures when obtaining actual tissue samples, and can produce accurate results of simulated tissue sample section staining that are considered by experts to be sufficient for diagnosis or research purposes. The five steps taken by the inventors of the present invention refer to Figure 5 Description is as follows.

[0096] Phase I - Tissue Collection and Preparation 500

[0097] The method for generating a staining image preferably requires at least one image of a tissue sample slice to be processed in order to generate a virtual staining image. Therefore, in this exemplary experiment, the inventor or user can first obtain a biopsy-sized tissue specimen (millimeter scale), such as those discarded from surgery, in order to prepare frozen sections from the specimen. Images of these specimens can in turn be used to train the machine learning network used in this example method for generating staining images. According to tests and experiments conducted by the inventors of the present invention, N=100 is the estimated minimum sample size for training an example machine learning network. Additional (N>=10) samples may be further required for prospective evaluation.

[0098] To obtain these samples, the procedure described below is similar to the standard hospital frozen specimen sectioning procedure. This will facilitate translation of this experimental example of a system for generating stained images into a clinical setting. Punch biopsies can be used to collect at least one sample from each tissue specimen.

[0099] As a test sample, the inventors of the present invention obtained a tissue sample from a rat, which was first placed on a mold, and an optimal cutting temperature (OCT) compound was used to cover the sample. The tissue was quickly frozen in isopentane and placed on a shelf in a cryostat awaiting sectioning.

[0100] After this procedure, the fixative can then be placed over the tissue and the entire assembly is placed in a freezing chamber at -22 degrees Celsius. Insert the razor blade and trim away any excess OCT from the tissue. Cut a 5 micron thick section, followed by a series of at least n adjacent sections. n is two sections for each non-H&E stain (see next stage). Spread the tissue sections on a non-roll board using a cooled brush. Press a room temperature glass slide (for microscopy) onto the tissue; it will stick due to the temperature difference. Rapidly fix the tissue by soaking it in 95% ethanol for a few seconds. The sections are now ready for staining.

[0101] Stage 2 - Dyeing 502

[0102] In the staining stage after the above-mentioned stage 1, all sample slices of each tissue sample prepared in the previous stage can be stained (stage 1 500). For example, the inventors of the present invention stained all sample slices except one sample slice with H&E. The remaining slice is then stained with a special stain or an immunostain. Common examples that can be used include cytokeratin markers, SOX10, and P40. Afterwards, in the image acquisition stage (504) and machine learning network training (the third and fourth stages later), the bright field and autofluorescence images of the H&E slices are converted into bright field or fluorescence images of adjacent slices, and the stains of the adjacent slices are different.

[0103] Since the distance between sections is within micrometers, the microstructure of the tissue is very similar. An example of an H&E staining protocol for frozen sections is given below.

[0104] H&E:

[0105] Rinse the slides in tap water

[0106] Place in hematoxylin for 1 minute

[0107] Rinse in tap water for 5 seconds

[0108] Rinse in Scott's tap water for 5 seconds to get a blue color

[0109] Soak in eosin for 20 seconds

[0110] Rinse quickly in tap water

[0111] 95% ethanol, 10 seconds

[0112] 100% ethanol, 10 seconds

[0113] 100% ethanol, 10 seconds

[0114] Soak in xylene for 20 seconds

[0115] Install with DPX

[0116] For special and immunostaining, users can follow the protocols determined by the manufacturer.

[0117] As mentioned above, multiple types of input stains may be necessary in order for a multimodal stain learning engine to be trained, and thus all but one of the prepared samples may be stained with a selected one of the multiple types of input stains, preferably in an alternating manner, and the remaining sample

[0118] Sections are instead stained with the output stain.

[0119] The examples of the above-described preparation procedures suitable for use with the example methods and systems for generating staining images are advantageous because the ability of the systems and methods to produce staining images is able to process frozen sections, rather than the more common formalin-fixed, paraffin-embedded sections, both of which are not necessary or desired when operating using an exemplary embodiment of the system for generating staining images. As shown in the experiments of the inventors of the present invention, the method for generating staining images does not require the common formalin-fixed, paraffin-embedded sections of tissue samples. This in turn is advantageous because it makes the preparation of tissue sample sections faster, more efficient, and cheaper. In addition, the advantages provided may result in faster tissue processing times, which is particularly advantageous in surgical operations that require rapid diagnosis and treatment.

[0120] Phase III - Image Acquisition 504A

[0121] This third stage 504A can obtain microscope images for training the machine learning network. Bright field and autofluorescence images can be obtained from all H&E sections. Bright field and autofluorescence images can also be obtained from special / immunostained sections. The choice of bright field or fluorescence, including which filter set to choose, depends on the specific stain used. In the next stage, the machine learning network will be trained to convert two H&E images of one section into a staining of an adjacent section. In one example operation, the input image can be obtained after H&E staining. Because it is less likely to disrupt hospital protocols, the above-mentioned example of a method and system for generating stained images may be advantageous because it can operate with existing hospital settings because H&E staining is always performed on frozen sections. In turn, it allows the example of a system and method for generating stained images to operate with existing hospital protocols to minimize interference.

[0122] In this example, microscopic image acquisition can be performed with a high-resolution microscope (e.g., a Nikon Eclipse Upright microscope) that is capable of both fluorescence and bright field microscopy. A slide with a tissue section will be placed on the stage. Images can then be captured with multiple (e.g., 4 and 40X) objective lenses and a 10X eyepiece. The sensor preferably has a high pixel count (e.g., 3376x2704).

[0123] Additionally, autofluorescence images can be acquired with the FITC (465-495 nm excitation, 515-555 nm emission) filter set. The slide will automatically translate with the stage to obtain multiple FOVs until the entire slide is imaged. This is advantageous because the translation stage will allow larger samples to be imaged and provide more training data. Fluorescence images of stains can also be acquired using the filter set that the specific stain is designed for.

[0124] Preferably, key image acquisition settings, such as camera exposure time and gain, will be set manually. This is to ensure that the acquisition settings remain constant across different samples.

[0125] Stage 4 - Image Correction 504B and Training Machine Learning Network 506

[0126] After all necessary sample images have been acquired with different input / output stains and different lighting conditions, image correction 504B may be performed to correct for spatial mismatches between all captured images.

[0127] Since images of the tissue samples have been obtained and corrected using interpolation and / or extrapolation, these images can then be used to train the machine learning network using the microscope images obtained above. In an exemplary embodiment, the machine learning network can be trained to take inputs, such as bright field and autofluorescence images of tissue sections stained with H&E and / or other stains, to produce an output, i.e., adjacent tissue sections stained with different stains. Since adjacent tissue sections are very similar in morphology, the machine learning network can be trained to virtually apply such stains when generating the stained image 507 as output.

[0128] Preferably, a machine learning network can be trained for each non-H&E stain. After training, the network can receive an input image, such as a frozen section of a surgical patient, and generate multiple images of the section under different staining conditions.

[0129] During trials, experiments and research, the inventors of the present invention have found that various machine learning networks can achieve similar results. However, in a preferred embodiment, the inventors of the present invention use a conditional generative adversarial network (cGAN) architecture designed for lung histology images.

[0130] Since cGAN is an extension of the original GAN, it has competing generator and discriminator networks. The generator is trained to map H&E images to images stained with other stains, while the discriminator is simultaneously trained to evaluate whether the generator output is real or fake. In cGAN, the network is conditioned on additional (image) information.

[0131] In one example, the inventors of the present invention have used a network with a cGANpix2pix model with a U-Net architecture. The generator has eight convolutional layers, each consisting of convolution, batch normalization, and activation function LeakyRelu. Each layer has a specific function that helps to handle transformations when multiple image sets are input during training. The convolution uses a 4x 4 spatial filter with a stride and a downsample factor of 2. The number of output channels will increase in a doubling manner (3, 64, 128, 256, 512, 1024). This ensures that a more detailed feature map can be extracted in the deeper layers of the convolution. Batch normalization is also used to normalize the data for each image file, and finally a rectification function Leaky-ReLu with a slope of 0.2 is used to increase the nonlinearity of the data during training. There are five convolutional layers in the discriminator, each consisting of convolution, batch normalization, and activation function LeakyRelu. The structure and function of each layer is similar to that of the convolutional layers in the generator, except that the discriminator places a sigmoid function in the last convolutional layer. This function outputs a probability matrix showing the likelihood that the output image is similar to the real image.

[0132] The entire training process is initially run on 200 hyperparameters (epochs), and during further training, the epoch number will be increased to improve the accuracy of the training. Also, some special techniques such as skip connections and image augmentation are used to prevent overfitting and improve the accuracy during the training process.

[0133] Phase V - Prospective Evaluation 508

[0134] This final stage is optional and was completed by the inventors as part of their trials and experiments to evaluate the accuracy of images generated by the methods and systems for generating stained images. In this example, the inventors evaluated training the machine learning network on additional H&E stained tissue sections (N>=10 for each tissue type that was not part of the training set).

[0135] For example, a trained network can be prospectively evaluated using at least 10 additional image sets that were acquired and processed for training. Note that evaluation differs from actual application of the network in that the output images need to be compared to physically stained images for accuracy. Therefore, only one image of the input slice is required for application.

[0136] During training, the estimated images are fed into the network to obtain output images. The computer output images are compared qualitatively and quantitatively to the physically stained versions. Qualitative comparisons are performed by experienced pathologists, looking for similarities in key features used for disease diagnosis, such as nuclear morphology and epithelial architecture. Quantitative comparisons are performed using standard image metrics, such as pixel-level intensity differences, peak signal-to-noise ratio, and structural similarity index measures.

[0137] Reference Figure 6 , shows an estimated set of images (bright field, GFP, and RFP images of H&E stained sections) fed into a neural network trained with only 20 sets. The generated output images are qualitatively and quantitatively close to the physically stained images (bright field images of pan-keratin stained sections). This performance is significantly better than single modality approaches.

[0138] An exemplary embodiment of a system for generating stained images is advantageous because it can provide a service for virtually staining frozen tissue sections for histopathological evaluation. The system and method are configured to receive an input image of a section stained with a standard stain, such as H&E, and an output image of the same section stained with a different stain. In turn, the system can be adapted for intraoperative use in tumor resection surgery.

[0139] Advantageously, tissue can be viewed using virtual tissue staining, whereby sample tissue is imaged using a digital microscope, either unstained or stained with only rapid and low-cost stains, and these images are then fed into a computer program to generate images of the same tissue under other stains.

[0140] Virtual staining may be particularly advantageous for staining that is time-consuming or expensive using traditional methods. A good example is immunostaining. Virtual staining also reduces contaminants, promotes digital medicine, and reduces crowding in hospitals.

[0141] Furthermore, multimodal image synthesis is a great improvement over single-input, single-output methods because it allows multiple input images to generate an output image. In addition, it can also improve the accuracy of image generation.

[0142] Furthermore, multimodal image synthesis may exploit its robustness better than existing pathology methods.

[0143] Existing single-modality image generation methods. For example, a single-modality method may be limited to the generation of a single input image resulting in an output image, but the input may be an unstained tissue section imaged with a bright field microscope, and the output is a hematoxylin and eosin (H&E) stained section imaged with a bright field. Advantageously, in a multimodal synthesis, as input, bright field and autofluorescence images of an unstained section can be used to generate a bright field image stained with H&E. The input image can also be an unstained and H&E stained tissue section for generating a special staining image of the tissue. Although not required, the embodiments described with reference to the accompanying drawings can be implemented as an application programming interface (API) or a series of libraries for use by developers, or can also be included in another software application, such as a terminal or personal computer operating system or a portable computing system. In general, since program modules include routines, programs, objects, components, and data files that assist in performing specific functions, those skilled in the art will understand that the functionality of the software application can be distributed among several routines, objects, or components to achieve the same functionality as desired herein.

[0144] It is also understood that if the invention and system of the present invention are implemented in whole or in part by a computing system, any suitable computing system architecture may be used. This will include stand-alone computers, network computers, and dedicated hardware devices. When "computer system" and "computer device" are used herein, these terms are intended to cover any suitable arrangement of computer hardware capable of implementing the described functions.

[0145] It will be appreciated by those skilled in the art that many changes and / or modifications may be made to the invention shown in the specific embodiments without departing from the spirit or scope of the invention as generally described. Therefore, the above-mentioned embodiments are to be considered illustrative in all aspects, rather than restrictive.

[0146] Unless otherwise stated, any reference to prior art in this document should not be taken as an admission that the information is common general knowledge.

Claims

1. A method for generating a dyeing image, comprising the following steps: obtaining a first image of a tissue sample section; and processing the first image with a multimodal stain learning engine to virtually stain the first image with at least one stain, thereby generating at least one generated stained image; Wherein, the multimodal staining learning engine includes a machine learning network, which is used to generate the at least one generated staining image; the machine learning network is trained by using a training image of each of at least three adjacent sample slices, and the three adjacent sample slices include a key sample slice and an adjacent sample slice obtained near the key sample slice; the multimodal staining learning engine is further configured to use an extrapolation method to correct the spatial mismatch between the training image representing the key sample slice and the training image of the adjacent sample slice; the extrapolation method is based on a predetermined slice thickness and a predetermined ordering of the key sample slice and the adjacent sample slice.

2. The method for generating a staining image according to claim 1, wherein the tissue sample section is not physically stained, or is physically stained with an input stain.

3. The method for generating a stained image according to claim 1, wherein one of the at least three adjacent sample slices is stained with an output stain, and wherein the remaining adjacent sample slices are stained with an input stain.

4. The method for generating a staining image according to claim 3, wherein the at least one generated staining image represents the key sample slice stained with the output stain.

5. The method for generating a staining image according to claim 3, wherein the remaining adjacent sample sections are stained with a plurality of types of the input stains in an alternating manner.

6. The method for generating a stained image according to claim 1, wherein the machine learning network comprises a generator network and a discriminator network; the generator network is configured to generate at least one generated stained image within a plurality of cycles; and the discriminator network is configured to analyze the at least one generated stained image to provide feedback to the generator network at each of the plurality of cycles.

7. The method for generating a stained image according to claim 6, wherein the machine learning network is a generative adversarial network.

8. The method for generating a stained image according to claim 7, wherein the generative adversarial network is trained using the training images of the at least three adjacent sample slices.

9. The method for generating a stained image according to claim 8, wherein the generative adversarial network is further trained using the training images of the at least three adjacent sample slices imaged under different lighting conditions.

10. The method for generating a staining image according to claim 9, wherein the illumination conditions include bright field and autofluorescence.

11. The method for generating a staining image according to claim 3, wherein each of the input stain and the output stain comprises at least one of hematoxylin-eosin (H&E) and a special stain.

12. The method for generating a staining image according to claim 11, wherein the special stain comprises an immunostain.

13. The method for generating a staining image according to claim 12, wherein the at least three adjacent sample sections are frozen.

14. The method for generating a staining image according to claim 13, wherein the at least three adjacent sample sections are fixed with formalin and embedded in paraffin.

15. The method for generating a stained image according to claim 1, wherein the training image representing the key sample slice is stained with an output stain; and the training image of the adjacent sample slice is stained with an input stain.

16. The method for generating a stained image according to claim 15, wherein the multimodal staining learning engine is further configured to align all the training images of the adjacent sample slices stained with the input stain with the training image representing the key sample slice stained with the output stain using an extrapolation method.

17. A method for generating a stained image according to claim 1, wherein the extrapolation method further includes the step of applying an image extrapolation transformation to produce at least one estimated input image based on the training image of the adjacent sample slice stained with the input stain.

18. A system for generating a stained image, comprising: An image gateway comprising an input device configured to obtain a first image of a tissue sample section; as well as an image generator, comprising a multimodal stain learning engine, the multimodal stain learning engine further comprising a machine learning network; the image generator is configured to process the first image, and the multimodal stain learning engine is configured to virtually stain the first image using at least one stain, thereby generating at least one generated stained image; Wherein, the multimodal staining learning engine includes a machine learning network, which is used to generate the at least one generated staining image; the machine learning network is trained by using a training image of each of at least three adjacent sample slices, and the three adjacent sample slices include a key sample slice and an adjacent sample slice obtained near the key sample slice; the multimodal staining learning engine is further configured to use an extrapolation method to correct the spatial mismatch between the training image representing the key sample slice and the training image of the adjacent sample slice; the extrapolation method is based on a predetermined slice thickness and a predetermined ordering of the key sample slice and the adjacent sample slice.

19. The system for generating a staining image according to claim 18, wherein the tissue sample section is not physically stained or is physically stained with an input stain.

20. The system for generating a staining image according to claim 18, wherein one of the at least three adjacent sample slices is stained with an output stain, and wherein the remaining adjacent sample slices are stained with an input stain.

21. The system for generating a staining image according to claim 20, wherein the at least one generated staining image represents the key sample section stained with the output stain.

22. The system for generating a staining image according to claim 20, wherein the remaining adjacent sample sections are stained with multiple types of the input stains in an alternating manner.

23. A system for generating a stained image according to claim 18, wherein the machine learning network includes a generator network and a discriminator network; the generator network is configured to generate at least one generated stained image within a plurality of cycles; and the discriminator network is configured to analyze the at least one generated stained image to provide feedback to the generator network at each of the plurality of cycles.

24. The system for generating a stained image according to claim 23, wherein the machine learning network is a generative adversarial network.

25. The system for generating a staining image according to claim 24, wherein the generative adversarial network is trained using the training images of the at least three adjacent sample slices.

26. The system for generating a stained image according to claim 25, wherein the generative adversarial network is further trained using the training images of the at least three adjacent sample slices imaged under different lighting conditions.

27. The system for generating a staining image according to claim 26, wherein the lighting conditions include bright field and autofluorescence.

28. The system for generating a staining image according to claim 20, wherein each of the input stain and the output stain comprises at least one of hematoxylin-eosin (H&E) and a special stain.

29. A system for generating a staining image according to claim 28, wherein the special stain comprises an immunostain.

30. The system for generating a staining image according to claim 29, wherein the at least three adjacent sample sections are frozen.

31. A system for generating a staining image according to claim 30, wherein the at least three adjacent sample sections are fixed in formalin and embedded in paraffin.

32. The system for generating a stained image according to claim 18, wherein the training image representing the key sample slice is stained with an output stain; and the training image of the adjacent sample slice is stained with an input stain.

33. A system for generating a stained image according to claim 32, wherein the multimodal staining learning engine is configured to align all of the training images of the adjacent sample slices stained using the input stain with the training image representing the key sample slice stained with the output stain using an extrapolation method.

34. A system for generating a stained image according to claim 33, wherein the multimodal staining learning engine is further configured to apply an image transformation to generate at least one estimated input image based on the training image of the adjacent sample slice stained using the input stain.

Images