Imaging method and system for generating a digitally stained image, training method for training an artificial intelligence system, and non-transitory storage medium

Abstract

In one aspect, the invention relates to an imaging method for generating a digitally stained image (51') of a biological tissue probe (50) from a physical image (50') of the biological tissue probe (50), wherein the physical image (50') of the biological tissue probe (50) is obtained by simultaneous multi-modality microscopy (53). In another aspect, the invention relates to a training method for training an artificial intelligence system to be used in such a method. Furthermore, the invention relates to a system for generating a digitally stained image (51') of a biological tissue probe (50) and / or for training an artificial intelligence system. The system comprises a processing unit for performing at least one of the methods. Furthermore, the invention relates to a non-transitory storage medium containing instructions which, when executed by a computer, cause the computer to perform the methods.

Description

Technical Field

[0001] This application relates to digitally stained images of biological tissue probes. More specifically, it relates to an imaging method for generating digitally stained images of biological tissue probes from unstained biological tissue probes, a training method for training an artificial intelligence system used in this method, a system for generating digitally stained images of biological tissue probes and / or for training an artificial intelligence system, and a non-transitory storage medium containing instructions. Background Technology

[0002] The concept of digital staining images of biological tissue probes is known in documents such as WO 2017 / 146813 A1. The method disclosed in that document involves acquiring data containing input images of biological cells illuminated using optical microscopy techniques and processing the data using a neural network. The image processing system consists of a staining cell neural network for generating various types of virtual staining images and a cell feature neural network for processing the staining cell image data. More precisely, it involves extracting or generating cell features characterizing the cells.

[0003] WO 2019 / 172901 A1 discloses a machine learning predictor model that is trained to generate predictions of the appearance of tissue samples stained with special staining agents (e.g., IHC (immunohistochemistry) staining agents) from input images that are unstained or stained with H&E (hematoxylin and eosin). The model can be trained to predict special stained images of various tissue types and special staining types.

[0004] The methods disclosed in the two documents above use paired images that must be precisely aligned with each other. This requires considerable effort and is practically impossible. This is because imaging the same tissue section requires a great deal of effort, and even slight morphological variations are likely to occur. In many cases, overlapping sections are also used, one stained and one unstained; these sections can have considerable morphological differences.

[0005] In addition, WO 2019 / 191697 A1 discloses a deep learning-based digital staining method that can obtain digital / virtual stained microscopic images from labeled or unstained samples based on autofluorescent images acquired using a fluorescence microscope.

[0006] In some scientific literature, the general concept of digital staining is discussed, for example,

[0007] - Rivenson, Y., Wang, H., Wei, Z. et al. Virtual histological staining of unlabelled tissue - autofluorescence images via deep learning.

[0008] Nat Biomed Eng 3, 466–477 (2019).

[0009] https: / / doi.org / 10.1038 / s41551 - 019 - 0362 - y;

[0010] - Rana A, Lowe A, Lithgow M, et al. Use of Deep Learning to Develop and Analyze Computational Hematoxylin and Eosin Staining of Prostate Core Biopsy Images for Tumor Diagnosis. JAMA Netw Open.

[0011] 2020; 3(5):e205111.doi:10.1001 / jamanetworkopen.2020.5111;以及

[0012] - Navid Borhani, Andrew J. Bower, Stephen A. Boppart, and Demetri Psaltis, “Digital staining through the application of deep neural networks to multi - modal multi - photon microscopy,” Biomed. Opt.

[0013] Express 10, 1339 - 1350(2019).

[0014] Most existing references use only a single imaging modality in basic optical microscopy. WO 2019 / 191697 A1 mentions that this method can benefit from various imaging modalities, such as fluorescence microscopy and nonlinear microscopy. However, the systems used in existing technologies require sequential imaging. Therefore, the processing time of different modalities is added together, resulting in a rather low frame rate at which images can be displayed or further evaluated. Furthermore, spatial registration of different imaging modalities is a major technical challenge, as mentioned in the aforementioned article by Borhani et al. Summary of the Invention

[0015] Therefore, one object of this application is to provide an improved method and system for generating digitally stained images, which overcomes or at least reduces the disadvantages of the prior art. In particular, the method and system allow for shorter processing times when using different modalities, and the frame rate of the images can be displayed or further evaluated at a reduced rate.

[0016] Furthermore, this process preferably does not require precisely aligned image pairs to train the tissue staining neural network. This also improves image quality. Additionally, it preferably avoids subtle differences in image pairs that could lead to artifacts or morphological changes during the coloring process, which could otherwise mislead subsequent diagnosis.

[0017] In a first aspect, the present invention relates to an imaging method for generating a digital stained image of a biological tissue probe from a physical image of the probe. The imaging method includes the following steps:

[0018] G1) Obtain physical images of unstained biological tissue probes using optical microscopy.

[0019] G2) generates digital staining images from physical images by using an artificial intelligence system, which is trained to predict digital staining images that can be obtained by staining probes using physical staining methods.

[0020] In the list of steps above, the letter "G" stands for "generate".

[0021] According to a first aspect of the invention, step G1) comprises obtaining a physical image of the unstained probe by simultaneous multimodal microscopy. In other words, two or more microscopic modalities are used simultaneously to obtain a physical image of the unstained probe.

[0022] Simultaneous multimodal microscopy can shorten the processing time for different modalities. Furthermore, it allows for a reduction in the frame rate for displaying or further evaluating images. Additionally, it enables the rapid acquisition of images covering different length scales by several orders of magnitude. Moreover, spatial registration between different imaging modalities is unnecessary. Furthermore, in some embodiments of the invention, the process does not require precisely aligned image pairs. This can offer significant advantages in test image generation and preprocessing, as it avoids highly complex and computationally intensive image registration algorithms. Furthermore, image quality is improved because images obtained through multimodal microscopy contain more information than those obtained through conventional microscopy. Additionally, in some embodiments, misleading artifacts or morphological changes during physical coloring can be avoided.

[0023] In some existing research, it has been established that H&E staining using autofluorescence images is possible. However, it is currently expected that staining quality will improve with increased image information. This increase in image information is achieved by using the multimodal imaging method according to the present invention. It is already apparent to the human eye that certain modalities do not contain important information about H&E staining. Regarding immunohistochemistry based on antigen-antibody reactions, using additional imaging modalities will result in better staining quality—especially considering the ongoing development of numerous different marker types. By obtaining simultaneous multimodal images of the tissue under study, rapid intracellular and cellular-level metabolic processes can be better visualized and quantified.

[0024] Step G1) can be performed in vitro, i.e., outside the patient; for this purpose, the tissue probe may have already been removed in previous steps, and may or may not be part of the method according to the invention. Alternatively, the physical image of the tissue probe can be obtained in vivo, i.e., inside the patient.

[0025] European patent application EP 20188187.7 and any possible subsequent patent applications that derive priority from that application disclose multimodal microscopy systems and related methods that can be used for simultaneous multimodal microscopy. The disclosures of these applications regarding multimodal microscopy systems are incorporated herein by reference. Specific details will be clearly disclosed below, but are included by reference only as not being limited to these specific details.

[0026] Specifically, when performing step G1) in vivo, this can be accomplished using a biopsy needle, which can form the scanning unit mentioned in the aforementioned application, as will be explained further below. The biopsy needle enables the user to apply the modality in vivo in a backscattering manner, i.e., signals generated from the tissue are collected and transmitted back to the detection unit via the bio-optical fiber needle.

[0027] Preferably, the method includes a further step G3 of displaying a digital staining image on a display device. This digital staining image is similar to a probe image that has been physically stained using one of the aforementioned staining methods. A trained person (especially a pathologist) can therefore deduce the characteristics of the tissue based on their experience in interpreting physically stained tissue. For example, a trained person can detect the progression or regression of a tumor or disease from the displayed image. Furthermore, by combining the aforementioned imaging methods with spectroscopy, the patient's response to treatment can be detected early. Additionally, the dynamic response to a specific treatment modality can be visualized.

[0028] The display device can be positioned near the multimodal microscopy system used in the imaging method, or it can be positioned away from it.

[0029] In addition to or instead of displaying digitally stained images on a display device, the interpretation of digitally stained images can be performed by the same or additional artificial intelligence system. This artificial intelligence system can also be trained in a known manner (but not in the context of digitally stained images).

[0030] In a second aspect, the present invention also relates to a training method for training an artificial intelligence system for use in the imaging method described above. The training method includes the following steps:

[0031] T1) Obtain a large number of image pairs, each pair including

[0032] - Physical images of unstained biological tissue probes obtained through optical microscopy, and

[0033] - A stained image of the probe obtained by a physical staining method.

[0034] T2) Based on the image pair, train an artificial intelligence system to predict a digitally stained image from the physical image of the unstained probe, the digitally stained image being obtained by staining the probe using the staining method.

[0035] In the above list of steps, the letter "T" stands for "training".

[0036] According to a second aspect of the invention, step T1) includes obtaining a physical image of the unstained probe by simultaneous multimodal microscopy.

[0037] When an artificial intelligence system is trained in this way, it can be used to generate digital stained images of biological tissue probes that have never been stained in the imaging methods described above.

[0038] Physical staining methods can be known in themselves and can be methods used in pathological disciplines selected from the following groups: histology, especially immunohistochemistry, especially immunohistochemistry; cytology; serology; microbiology; molecular pathology; clonal analysis; PARR (PCR for antigen receptor rearrangement); and molecular genetics. Histology includes the aforementioned H&E (hematoxylin and eosin) methods. The discipline of “molecular pathology” is understood in this paper as the application of the principles, techniques, and tools of molecular biology, biochemistry, proteomics, and genetics to diagnostic medicine. This includes immunofluorescence (IF), in situ hybridization (ISH), microRNA (miRNA) analysis, digital pathological imaging, toxicological genomic assessment, quantitative polymerase chain reaction (qPCR), multiplex PCR, DNA microarrays, in situ RNA sequencing, DNA sequencing, antibody-based immunofluorescence histography, molecular profiling of pathogens, and analysis of bacterial antimicrobial resistance.

[0039] Simultaneous multimodal microscopy may include at least two different modes selected from the group consisting of: two-photon excited fluorescence, two-photon autofluorescence, fluorescence lifetime imaging, autofluorescence lifetime imaging, second harmonic generation, third harmonic generation, incoherent / spontaneous Raman scattering, coherent anti-Stokes Raman scattering (CARS), broadband or multiple CARS, stimulated Raman scattering, coherent Raman scattering, stimulated emission loss (STED), nonlinear absorption, confocal Raman microscopy, optical coherence tomography (OCT), single-photon / linear fluorescence imaging, bright-field imaging, dark-field imaging, three-photon, four-photon, second harmonic generation, third harmonic generation, fourth harmonic generation, phase contrast microscopy, photoacoustic (or synonymous photoacoustic) techniques such as single-frequency and multi-frequency spectral photoacoustic imaging, photoacoustic tomography, photoacoustic microscopy, photoacoustic remote sensing and its variants.

[0040] All of the above imaging techniques can be applied in vivo, ex vivo, living tissue, or excised tissue, including any suitable endoscopic technique.

[0041] The cell nucleus and the DNA it contains exhibit low autofluorescence, but absorb primarily in the ultraviolet range. However, the cell nucleus and its DNA are important for many diagnostic purposes. To overcome this drawback, at least one modality can be phase contrast microscopy. In the context of this application, phase contrast microscopy (known per se) should be understood as an optical microscopy technique that converts a phase shift of light passing through a transparent sample into a brightness change in an image. The phase shift itself is invisible, but becomes visible when displayed as a brightness change. The advantage of phase contrast microscopy is that the cell nucleus and its DNA can be displayed more clearly compared to several other imaging modalities. This greatly facilitates the application of AI technology in this application. The aforementioned photoacoustic techniques are also particularly suitable because the light applied in these techniques is highly absorbed by the cell nucleus and / or the molecules within it.

[0042] Artificial intelligence systems can include at least one neural network, particularly convolutional neural networks (CNNs) and / or generative adversarial networks (GANs), such as Cycle-GAN, which take a physical image of an unstained biological tissue probe as input to provide a corresponding digitally stained image as output. The neural network can transform images in image space and images obtained through multimodal microscopy into corresponding images in feature space, specifically into vectors or matrices in feature space. Preferably, the vectors or matrices in feature space have a lower dimension than the images in image space. From the images in feature space, the digitally stained image can be obtained.

[0043] For example, neural networks similar to those disclosed in Ronneberger et al.'s U-Net: Convolutional Networks for Biomedical Image Segmentation (available at https: / / arxiv.org / pdf / 1505.04597.pdf) can be used. Specifically, the neural network can apply one or more of the following known principles: data augmentation, particularly when only a small amount of data is available; downsampling and / or upsampling, particularly as disclosed in Ronneberger et al.'s paper; and weighted loss.

[0044] As an alternative to neural networks, it is also conceivable to perform pixel-by-pixel or region-by-region colorization on images, for example, through known classification algorithms such as the Random Forest model. However, even though this invention includes such alternatives, such algorithms are currently considered to be more complex and produce lower quality images.

[0045] In contrast to methods known in the prior art, in this invention, images that are not precisely aligned can be trained. This can be achieved through a training architecture, which is preferably different from the imaging architecture used in the imaging method and may contain at least one neural network component used solely for training.

[0046] This method can be an adaptation of the approach proposed by Zhu et al. in Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (available at https: / / arxiv.org / pdf / 1703.10593.pdf), which discloses a generator-discriminator network: In a preferred embodiment of the invention, during training, the transformed image, in this case a digitally stained image, is re-transformed into the input image of the physical stained probe. Finally, the re-transformed image and the original image should be as consistent as possible.

[0047] More specifically, the training in step T2) may include the steps of the first sequence:

[0048] T2.U.1) transforms the primary unstained training image in the image space, i.e. the unstained primary training image obtained by multimodal microscopy using an unstained probe, into the first vector or matrix in the feature space through the first image-to-feature transformation.

[0049] T2.U.2) transforms a first vector or matrix from the feature space into a digitally colored image in the image space through a first feature-to-image transformation;

[0050] T2.U.3) transforms a digitally colored image from the image space into a second vector or matrix in the feature space through a second image-to-feature transformation;

[0051] T2.U.4) transforms the second vector or matrix in the feature space into a secondary uncolored image through the second feature-to-image transformation;

[0052] T2.U.5) compares the secondary uncolored image with the primary uncolored training image;

[0053] When the comparison produces differences outside the predefined range, the transformation can be modified and re-transformed, and steps T2.U.1) to T2.U.4) and optionally T2.U.5) can be repeated.

[0054] In addition, training may include steps of a second sequence.

[0055] T2.S.1) The primary staining training image in the image space, i.e. the primary staining training image obtained from the physical staining probe by multimodal microscopy, is transformed into the first vector or matrix in the feature space through the second image-to-feature transformation.

[0056] T2.S.2) transforms the first vector or matrix from the feature space into a digital uncolored image in the image space through the second feature-to-image transformation;

[0057] T2.U.3) transforms a digital uncolored image from the image space into a second vector or matrix in the feature space through a first image-to-feature transformation;

[0058] T2.U.4) transforms the second vector or matrix in the feature space into a secondary colored image through the first feature-to-image transformation;

[0059] T2.S.5) compares the secondary staining image with the primary staining training image.

[0060] Similarly, when the comparison produces differences outside the predefined range, the transformation can be modified and re-transformed, and steps T2.S.1) to T2.S.4) and optionally T2.S.5) can be repeated.

[0061] In the above list of steps, the letter "U" indicates "unstained" and "S" indicates "stained".

[0062] However, the neural network used for transformation and re-transformation is the opposite of the neural network disclosed by Zhu et al. The discriminator described above takes over the evaluation of image plausibility by comparing and evaluating the labeled data (ground truth) with the digitally stained image. Unlike the architectures known from Zhu et al. or from *GAN-based Virtual Re-Staining: A Promising Solution for Whole Slide Image Analysis* (available from the document by Zhaoyang et al., https: / / arxiv.org / pdf / 1901.04059.pdf), the input unstained image and output digitally stained image in this invention have different numbers of modalities. Since this poses a problem for identity loss, modifications are needed for use in this invention.

[0063] Depending on the quality of the modal image, preprocessing in the form of denoising and deconvolution can be performed to ensure that the color image ultimately meets the desired requirements. This preprocessing can be accomplished using a self-supervised image denoising neural network, such as the neural network described in Kobayashi et al.'s paper Image Deconvolution via Noise-Tolerant Self-SupervisedInversion (available at https: / / arxiv.org / pdf / 2006.06156v1.pdf).

[0064] In some cases, even with precisely aligned image pairs, slight morphological differences may still exist between successive tissue slices and / or re-embedded tissue probes, which can negatively impact the quality of digitally stained images. Cycle-GAN can be used in these situations. Since training Cycle-GAN often introduces instability, training based on registered images and pixel-by-pixel comparisons can be used. This allows for training on general chromaticity. During continuous training, the effects of pixel-by-pixel correlations can be reduced, and a stable Cycle-GAN can be trained.

[0065] On the other hand, the present invention also relates to a system for generating digitally stained images of biological tissue probes and / or for training artificial intelligence systems. This system includes:

[0066] - An optical microscopy system used to obtain physical images of biological tissue probes via simultaneous multimodal microscopy.

[0067] - Data storage for storing a large number of image pairs, each pair including:

[0068] - Physical images of biological tissue probes obtained through optical microscopy, and

[0069] - A stained image of the probe obtained by a physical staining method.

[0070] - Processing unit, used for execution:

[0071] -The imaging method described above, and / or

[0072] - The training method described above.

[0073] The above methods can be performed using this system.

[0074] In this system, an optical microscopy system, such as that disclosed in European patent application EP 20188187.7 and any possible subsequent patent applications prior to that application, can be used. More specifically, the system may include at least one first basic unit comprising at least one electrical and / or optical basic component, at least one scanning unit comprising at least one scanning component, and at least one detection unit comprising at least one detection component. The at least one first basic unit may include at least two electrical and / or optical basic components. The at least one scanning unit may include at least two scanning components. The at least one detection unit may include at least two detection components.

[0075] The scanning unit and / or detection unit are freely movable, specifically possessing six degrees of freedom. The scanning unit and / or detection unit can be connected to the first basic unit via at least one flexible connecting line, specifically at least one optical connecting line and / or at least one electrical connecting line. At least one basic component, at least one scanning component, and at least one detection component can be operatively coupled to each other, such that at least one basic component and / or at least one scanning component and / or at least one detection component can be used in combination, specifically simultaneously for more than one mode. In other words, two components of the three units (basic unit, scanning unit, detection unit) can be used in combination with the same component or the same set of components in the remaining units.

[0076] In another aspect, the present invention also relates to a non-transitory storage medium containing instructions that, when executed on a computer, cause the computer to perform the methods described above. Attached Figure Description

[0077] Detailed embodiments and further advantages of the present invention will now be explained with reference to the following accompanying drawings, wherein...

[0078] Figure 1 A schematic diagram of the method is shown;

[0079] Figure 2 A schematic diagram of a system for performing this method is shown;

[0080] Figure 3 Several images of biological tissues containing cell nuclei are shown. Detailed Implementation

[0081] Figure 1 The schematic diagram illustrates images 70 and 70' in image space I, feature vectors 71 and 71' in feature space F, and transformations 72, 73, 72', and 73' between these images and vectors. According to the present invention, transformations 72, 73, 72', and 73' are obtained by a neural network.

[0082] according to Figure 1In the upper part, the primary image 70 obtained by multiphoton microscopy from the unstained probe is transformed into a first vector 71 in the feature space F through a first image-to-feature transformation 72 (step T2.U.1). The first vector 71 is then transformed into a digitally stained image 70' in the image space I through a first feature-to-image transformation 72' (step T2.U.2). In the generation method, these digitally stained images 70' are displayed on the monitor 60. During training, the digitally stained image 70' in the image space I is transformed into a second vector 71' in the feature space F through a second image-to-feature transformation 73 (step T2.U.3). Subsequently, the second vector 71' is transformed into a secondary unstained image 70 through a second feature-to-image transformation 73' (step T2.U.4). The neural network is to be trained or has already been trained such that the primary unstained image 70 on the left is similar to the secondary image 70 on the right (step T2.U.5).

[0083] according to Figure 1 In the lower half, the primary image 70' obtained from the stained probe by multiphoton microscopy is transformed into a first vector 71' in the feature space F through a second image-to-feature transformation 73 (step T2.S.1). The first vector 71' is then transformed into a digital unstained image 70 in image space I through the second feature-to-image transformation 73' (step T2.S.2). Then, the digital stained image 70' in image space I is transformed into a second vector 71 in the feature space F through a first image-to-feature transformation 72 (step T2.S.3). Subsequently, the second vector 71' is transformed into a secondary stained image through the first feature-to-image transformation (step T2.S.4). The neural network is to be trained or has already been trained such that the primary stained image 70' on the left is similar to the secondary image 70 on the right (step T2.S.5).

[0084] According to Figure 2 In one embodiment, a tissue probe 50 (e.g., a rapid section, frozen section, fixed section, fresh section, or whole tissue, such as from a perforated biopsy) is positioned on a glass slide (not shown). The slide is then scanned using a multimodal microscopy system as disclosed in the aforementioned European patent application EP 20188187.7.

[0085] More specifically, Figure 2 The multimodal microscopy system 1 shown includes a first basic unit 2 (hereinafter referred to as basic unit 2), a scanning unit 4, and a detection unit 5.

[0086] Basic unit 2 includes several light sources for different modes: a laser source 14, a light source 15 for fluorescence imaging, a laser 16 for Raman scattering, a light source 17 for optical coherence tomography (OCT), an amplifier-pumped laser 18, and a white light source 19. Basic unit 2 also includes electronic components 23, software 24, and a power supply 25.

[0087] The scanning unit 4 includes several scanning components: an optical amplifier 20, a transmission / scanning optics 21, and an excitation-emission filter 22. Each light source 14, 15, 16, 17, 18, 19 is connected to the scanning unit 4 via a separate flexible connection line 6 (e.g., an optical fiber cable). Thus, each light source 14, 15, 16, 17, 18, 19 is operatively connected to the same set of scanning components 20, 21, 22 so that this single set of scanning components 20, 21, 22 can be provided for different modes associated with the light sources 14, 15, 16, 17, 18, 19.

[0088] The scanning unit 4 also includes an objective lens 12 for guiding the analytical light to the probe 50. More specifically, the objective lens 12 is arranged in the scanning unit 4 such that signals emitted from the probe are transmitted back through the objective lens 12. A filter 22 is arranged such that signals emitted from the probe 50 are filtered by the filter 22. The scanning unit 4 is also connected to the base unit 2 via a cable 29, which powers and controls the scanning unit 4.

[0089] Detection unit 5 is operatively connected to scanning unit 4 and includes several detection components: a filter detector 7, a single-photon counter 8, a spectrometer 9, an optical power meter 10, and a fluorescence camera 11. Scanning unit 4 and detection unit 5 are freely movable with six degrees of freedom. Each detection component 7, 8, 9, 10, 11 is connected to scanning unit 4 via a separate flexible connection line 28 (e.g., fiber optic cable). Therefore, each detection component 7, 8, 9, 10, 11 is operatively connected to the same set of scanning components 20, 21, 22 so that this single set of scanning components 20, 21, 22 can be provided for different modes associated with detection components 7, 8, 9, 10, 11. Alternatively, light emitted from light source 17 can be directly emitted onto probe 50. Detection unit 5 is also connected to base unit 2 via cable 30, which powers and controls detection unit 5.

[0090] System 1 also includes a switching unit 3, which allows signals emitted from probe 50 to be selectively transmitted to detection unit 5 according to the selected mode.

[0091] exist Figure 3Several images of the biological probe are shown. The probe is a 3 μm thick human skin sample embedded in paraffin. The left image shows the probe, which has been physically stained with H&E. This image was obtained using a 20x NA 0.75 microscope. The cell nuclei 52 are clearly visible. The middle image shows a digitally stained image where not all cell nuclei are identifiable; only their locations are marked with 52'. This can be achieved by using at least one phase-contrast modality in the method of the present invention.

Claims

1. An imaging method for generating a digital stained image (70') of a biological tissue probe (50) from a physical image (70') of an unstained biological tissue probe (50), comprising the following steps: G1) Obtain physical images (70) of unstained biological tissue probes (50) by optical microscopy. G2) A digital staining image (70') is generated from the physical image (70) by using an artificial intelligence system, wherein the system is trained to predict the digital staining image (70') that can be obtained by staining the probe (50) by a physical staining method. Its features are, Step G1) includes obtaining a physical image (70) of the unstained probe (50) by simultaneous multimodal microscopy, without requiring spatial registration of different imaging modalities.

2. The imaging method according to claim 1, wherein, The method further includes the following steps: G3) Displays the digitally stained image (70') on the display device (60).

3. A training method for training an artificial intelligence system, the training method being used in the imaging method according to claim 1 or 2, the training method comprising the following steps: T1) obtains a large number of image pairs (70, 70'), each pair including - Physical image (70) of unstained biological tissue probe (50) obtained by optical microscopy, and - A stained image (70') of the probe (50) obtained by a physical staining method. T2) Based on the image pair (70, 70'), the artificial intelligence system is trained to predict a digitally stained image (70') from the physical image (70) of the unstained probe (50), the digitally stained image (70') being obtained by staining the probe (50) using the staining method. Its features are, Step T1) includes obtaining a physical image (70) of the unstained probe (50) by simultaneous multimodal microscopy, without requiring spatial registration of different imaging modalities.

4. The training method according to claim 3, wherein, The physical staining method is a method used in pathological disciplines selected from the group consisting of: histology; cytology; serology; microbiology; molecular pathology; clonal analysis; PARR (PCR for antigen receptor rearrangement); and molecular genetics.

5. The training method according to claim 4, wherein, The histology referred to is immunohistochemistry.

6. The training method according to claim 5, wherein, The immunohistochemistry is the same as immunohistochemistry.

7. The training method according to any one of claims 3 to 6, wherein, The artificial intelligence system includes at least one neural network that uses a physical image (70) of an unstained biological tissue probe (50) as input to provide a corresponding digitally stained image (70') as output.

8. The training method according to claim 7, wherein, The neural network is a convolutional neural network or a generative adversarial network.

9. The training method according to claim 7, wherein, The neural network transforms the images (70, 70') in the image space (I) and the images (70, 70') obtained by multimodal microscopy into vectors (71) or matrices in the feature space (F).

10. The training method according to claim 9, wherein, The training in step T2 includes: - Steps of the first sequence T2.U.1) transforms the primary unstained training image (70) in the image space (I), i.e. the unstained primary training image (70) obtained by multimodal microscopy from an unstained probe (50), into the first vector (71) or matrix in the feature space (F) through the first image-to-feature transformation (72); T2.U.2) transforms the first vector (71) or matrix from the feature space (F) into a digitally stained image (70') in the image space (I) through the first feature-to-image transformation (72'); T2.U.3) transforms the digitally stained image (70') from the image space (I) into a second vector (71') or matrix in the feature space (F) through a second image-to-feature transformation (73); T2.U.4) transforms the second vector (71') or matrix in the feature space (F) into a secondary uncolored image (70'') through the second feature-to-image transformation (73'); T2.U.5) compares the secondary uncolored image (70'') with the primary uncolored training image (70); - Steps in the second sequence T2.S.1) The primary staining training image (70') in the image space (I), i.e. the primary staining training image (70') obtained from the physical staining probe (50) by multimodal microscopy, is transformed into a first vector (71') or matrix in the feature space (F) by the second image-to-feature transformation (73). T2.S.2) Transforms the first vector (71') or matrix from the feature space (F) into a digital uncolored image (70'') in the image space (I) through the second feature-to-image transformation (73'); T2.U.3) transforms the digital uncolored image (70'') from the image space (I) into a second vector (71) or matrix in the feature space (F) through the first image-to-feature transformation (72); T2.U.4) transforms the second vector (71) or matrix in the feature space (F) into a secondary stained image (70) through the first feature-to-image transformation (72'); T2.S.5) compare the secondary staining image (70) with the primary staining training image (70').

11. The training method according to claim 7, wherein, The neural network includes: - An imaging architecture for predicting digitally stained images (70'), which are obtainable by staining the probe (50) using a physical staining method, and - The training architecture used to train the neural network. The training architecture differs from the imaging architecture and includes at least one network component used only for training the neural network.

12. The training method according to claim 11, wherein, The training architecture includes a generator-discriminator network.

13. The training method according to any one of claims 3 to 6, wherein, The input image (70) of the unstained probe (50) and the digitally stained image (70') have different numbers of modalities.

14. The training method according to any one of claims 3 to 6, wherein, The simultaneous multimodal microscopy includes at least two different modes selected from the group consisting of: two-photon excited fluorescence, two-photon autofluorescence, fluorescence lifetime imaging, autofluorescence lifetime imaging, second harmonic generation, third harmonic generation, incoherent / spontaneous Raman scattering, coherent anti-Stokes Raman scattering (CARS), broadband or multiple CARS, stimulated Raman scattering, coherent Raman scattering, stimulated emission loss (STED), nonlinear absorption, confocal Raman microscopy, optical coherence tomography (OCT), single-photon / linear fluorescence imaging, bright-field imaging, dark-field imaging, three-photon, four-photon, second harmonic generation, third harmonic generation, fourth harmonic generation, phase contrast microscopy, photoacoustic techniques such as single-frequency and multi-frequency spectral photoacoustic imaging, photoacoustic tomography, photoacoustic microscopy, and photoacoustic remote sensing.

15. A system for generating digitally stained images (70) of biological tissue probes (50), the system comprising: - An optical microscopy system (1) for obtaining physical images (70) of biological tissue probes (50) via simultaneous multimodal microscopy without requiring spatial registration of different imaging modalities. - Data storage for storing a large number of image pairs, each pair including: - Physical images (70) of unstained biological tissue probes (50) obtained by simultaneous multimodal microscopy, and - A stained image (70') of the probe (50) obtained by a physical staining method. - Processing unit, used for execution: - The imaging method according to any one of claims 1 and 2.

16. A system for training an artificial intelligence system, the system comprising: - An optical microscopy system (1) for obtaining physical images (70) of biological tissue probes (50) via simultaneous multimodal microscopy without requiring spatial registration of different imaging modalities. - Data storage for storing a large number of image pairs, each pair including: - Physical images (70) of unstained biological tissue probes (50) obtained by simultaneous multimodal microscopy, and - A stained image (70') of the probe (50) obtained by a physical staining method. - Processing unit, used for execution: - The training method according to any one of claims 3 to 14.

17. A non-transitory storage medium containing instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 14.

18. The non-transitory storage medium of claim 17, wherein the method of any one of claims 3 to 14 is performed in the system of claim 16.

Images