Method of image generation in multimodal microscopy
Patent Information
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- POLITECNICO DI TORINO
- Filing Date
- 2024-07-15
- Publication Date
- 2026-05-20
AI Technical Summary
Current clinical microscopy methods for disease diagnosis are costly and time-consuming, requiring multiple expensive techniques and stains, which can compromise tissue samples and lead to inaccurate results due to destructive nature of some techniques.
A method using generative models to generate multimodal microscopy images that combine morphological and functional information from different microscopy techniques and stains, reducing the need for physical application of stains and techniques.
This method significantly reduces time and costs by enabling accurate and timely diagnosis through the generation of congruent and accurate information from various microscopy techniques, without compromising the original tissue sample composition.
Smart Images
Figure IB2024056859_23012025_PF_FP_ABST
Abstract
Description
[0001] "Method for generating images in multimodal microscopy"
[0002] DESCRIPTION
[0003] TECHNICAL FIELD
[0004] The present invention relates to methods for generating images in multimodal microscopy .
[0005] STATE OF THE ART
[0006] In the field of clinical microscopy for the diagnosis of diseases , various microscopy techniques can be used to analyse a tissue sample both from a morphological point of view, for example , to observe its cellular structure , and from a functional point of view, for example , to as sess the composition and cellular activity .
[0007] For this purpose , numerous microscopy techniques are known, including bright- field optical microscopy, fluorescence and immunofluorescence microscopy, phase-contrast microscopy, and photoacoustic microscopy ( PAM) .
[0008] In the field of bright- field optical microscopy, sections of tissue samples , only a few millimetres thick, are chemically stained, meaning they undergo chemical reactions that cause the tissue structures to take on di f ferent colours depending on the morphological and functional characteristics intended to be highlighted .
[0009] Among histochemical stains for comparing the morphological / cellular characteristics of various tissues and diseases , there are more common and af fordable ones like hematoxylin-eosin (H&E) , and more specialized and expensive ones such as PAS (periodic acid-Schiff) staining or trichrome staining (TRIG) . Immunohistochemical (IHC) stains, on the other hand, are used to highlight functional aspects (e.g., cell proliferation) through the reaction to a specific molecular biomarker. Similarly, for fluorescence and immunofluorescence microscopy, different fluorescent substances (fluorochromes) are used to identify cells, sub- cellular components, or functional properties of the tissue. Optimal diagnosis would require the analysis of tissue samples through a variety of microscopy techniques and / or the use of different stains in order to gather as much complementary information as possible. However, this procedure would be very expensive both economically— since some microscopy techniques are more costly than others— and in terms of time, as it would require the involvement of laboratory technicians and pathologists.
[0010] Additionally, another problem arises from the fact that the application of certain stains and microscopy techniques to the tissue sample may compromise its original composition, thus altering subsequent analyses.
[0011] If microscopy techniques, especially destructive ones, are to be used on multiple contiguous sections of the sample, there is a risk of obtaining inaccurate results. In fact, even if the sections are adjacent, they do not show precise correspondence between cellular structures, as they belong to different portions of the tissue.
[0012] SUMMARY OF THE INVENTION The purpose of the present invention is to solve , at least in part , the problems highlighted in the known art by providing a method that allows the analysis of a tissue sample through the application of various microscopy techniques and multiple stains , in such a way that congruent and accurate information is generated, enabling a specialist to formulate a precise diagnosis .
[0013] The invention also relates to a method for generating multimodal microscopy images , which include both morphological and functional information, by applying at least one generative model to the digiti zed image .
[0014] The method comprises the following steps :
[0015] • Receiving a digiti zed image that includes a histological sample obtained via a microscopy technique ;
[0016] • Receiving a first parameter identi fying an input microscopy technique related to the received image and a second parameter identi fying an output microscopy technique related to an image to be generated;
[0017] • Selecting, from a predefined plurality of generative models trained to generate an image according to a first microscopy technique starting from an image obtained via a second microscopy technique , based on the first and second parameters , a generative model trained to obtain an image according to the said output microscopy technique based on the image obtained via the said input microscopy technique ;
[0018] • Dividing the entire image into a plurality of N patches ;
[0019] • Applying the selected generative model to each n-th patch in such a way as to generate , for each n-th patch, a corresponding k-th patch with characteristics of the said output microscopy technique ;
[0020] • For each k-th patch, applying a cropping technique to delimit an area within which the patch remains unchanged and applying a smoothing filter outside of said area, preferably along the edges , to reduce arti facts between the patch and the adj acent ones , thus allowing the aggregation of said k-th patches ;
[0021] • Generating a complete output image by composing said k- th patches .
[0022] The method of the present invention allows for a reduction in both time and costs , compared to manually performed stains and microscopy techniques , to obtain complementary information from di f ferent microscopy techniques , enabling timely and accurate diagnoses .
[0023] Preferably, the method for generating multimodal microscopy images also includes a step for normali zing the intensities of the input images .
[0024] The described method further reduces input variability and optimi zes the final output in generating an image based on one microscopy technique from an input image obtained via a di f ferent microscopy technique through pre- and postprocessing steps around the application of the generative model .
[0025] Preferably, the method for training a generative model also includes a step for normali zing the intensities of the input images .
[0026] In a preferred embodiment , the image obtained using the multimodal microscopy image generation method is saved in an original non-proprietary format or in a pyramidal TI FF format .
[0027] BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Preferred embodiments of the present invention will be described below, purely by way of example , with reference to the attached drawings , in which :
[0029] • Fig . 1 shows , using functional blocks , a first embodiment of the method for generating microscopy images through the use of trained generative models .
[0030] • Fig . 2 shows , via a flowchart , a second embodiment of the method from Fig . 1 .
[0031] • Fig . 3 shows , using a flowchart , a method for training a plurality of generative models in multimodal microscopy .
[0032] • Fig . 4 shows a first comparative example between an image generated according to the method of the present invention and a real image for a gastric tissue sample .
[0033] • Fig . 5 shows a second comparative example between an image generated according to the method of the present invention and a real image for an esophageal tissue s amp 1 e .
[0034] DETAILED DESCRIPTION OF THE INVENTION
[0035] The following detailed description of preferred embodiments refers to the attached drawings , which form part of this document and illustrate speci fic embodiments of the present invention by way of example . The following description is therefore not to be interpreted in a limiting sense , and the scope of the inventions is defined only by the appended claims .
[0036] Fig . 1 shows , using functional blocks , a first embodiment of the method according to the present invention for generating microscopy images through the use of trained generative models . In particular, the steps to be carried out are in the following order :
[0037] • Receiving a digiti zed image that includes a histological sample obtained through a microscopy technique .
[0038] • Receiving an input microscopy technique related to the technique used on the sample from which the digiti zed image originates , and an output microscopy technique related to the type of image to be generated .
[0039] The information related to the input and output microscopy techniques is used at this stage to select the most appropriate generative model from a plurality of available models , where each model is trained to generate an image according to a first microscopy technique starting from an input image generated us ing a second microscopy technique di f ferent from the first .
[0040] • In the next step, a trained generative model is selected from a predefined plurality of generative models , each trained to generate an image according to a first microscopy technique starting from an image obtained through a second microscopy technique to obtain an image according to the selected output microscopy technique based on the image generated using the input microscopy technique . As will be described later, the selection of this generative model can be made based on an evaluation of quantitative metrics related to the quality of the image that the model can generate .
[0041] • The entire image is divided into N patches , where in the context of image processing, the term "patch" refers to a rectangular or square portion of the image on which further processing is performed . The si ze of the patch ranges from 512x512 to 3000x3000 pixels .
[0042] • Subsequently, the selected generative model is applied to the N patches , which have the characteristics of the first microscopy technique , in such a way that for each n-th patch, a corresponding k-th patch is obtained, possessing the characteristics of the second microscopy technique . By " characteristics , " we mean morphological and / or functional characteristics such as the type of histochemical staining ( e . g . , H&E , PAS , TRIG, GIEMSA) or the fluorescent dye / biomarker used in immunohistochemical or immunofluorescence staining, etc .
[0043] The following step works by applying a meta-heuristic algorithm to the k-th patches to reduce arti facts , such as intensity discontinuities that occur between one patch and the next . Said algorithm is configured to retain, for each k-th patch, only the central part of the generative model ' s prediction, using a technique known as " centre crop" and afterward, a smoothing filter is applied to reduce edge arti facts between each k-th patch and the next . The centre crop technique allows for cropping the k-th patch starting from the centre with a speci fied width and shape . For example , by providing a pair of parameters representing the height and width of the area to be cropped, the resulting portion from the centre crop operation will have a square or rectangular shape , depending on the pair of values provided as input parameters .
[0044] The smoothing filter is known in image processing for blurring and reducing noise caused by arti facts present in the image . Speci fically, these filters are called mean filters because they replace the value of a pixel in the image with the mean value of the surrounding pixels . The mean is calculated based on a statistical technique known as a kernel smoother, which assigns higher weights to the closest pixels , so the value of the replaced pixel is a function of the nearby pixels . At this stage , the si ze and shape of the " center crop" and the type of kernel for the filter are optimi zed based on the cellular structures present in the output microscopy technique .
[0045] • Finally, in the last step, an image is generated that has the characteristics of the selected microscopy technique according to the chosen generative model .
[0046] Preferably, the method also includes a step of normali zing the intensities of the n-th patches before the step of applying the generative model to generate the corresponding k-th patches .
[0047] A preferred embodiment of intensity normali zation for bright- field optical microscopy is described in application no . WO2021191723A1 , which allows adj usting the colouring of an input image by forcing it toward a target colouring of a reference digital image . For fluorescence and phase-contrast microscopy, techniques such as brightness histogram normalization or equalization, like Contrast Limited Adaptive Histogram Equalization (CLAHE) , known to those skilled in the art, can be used.
[0048] In particular, the normalization step applied before the generative model and the artifact reduction step applied afterward allow reducing input variability and optimizing the final output, respectively.
[0049] The normalization step standardizes the intensities, aligning the dynamics of the patches that will be fed into the generative model. If the model receives highly variable or low-information data as input, the final performance will be compromised. Specifically, standardizing the dynamics ensures that RGB images, which exhibit different colouring intensities due to the manual tissue staining process (e.g., in bright-field optical microscopy) , share the same tonal range in the colour profile. Additionally, grayscale images (e.g., in fluorescence or phase-contrast microscopy) will reflect the same pixel intensity distribution in the brightness histogram.
[0050] When the trained generative model works on the n-th patches, into which the entire digitized image is divided, to generate the k-th patches, there can be a loss of information at the edges between one patch and the next. To solve this problem and optimize the final output of the generative model, a patch aggregation technique is applied to minimize the presence of these artifacts.
[0051] Preferably, the image generated by the method of the present invention is saved in an original non-proprietary format or in a pyramid TIFF format.
[0052] According to a further embodiment, shown in Fig. 2, during the generative model selection phase, the chosen generative model is the one with the best performance in terms of quantitative metrics that evaluate image quality, to minimize the presence of artifacts that could make the image unrealistic .
[0053] To reduce computation time, the selection of the optimal model is performed by testing the models only on a summary image that is representative of the entire digitized image. The calculated metrics between the real and generated image can be of the full-reference type (i.e., Peak Signal-to- Noise Ratio and Structural Similarity Index) or segmentation-based (i.e., Dice Similarity Coefficient and Hausdorff Distance) , well-known to those skilled in the field .
[0054] The goal is to obtain an image that presents the same characteristics as the real image.
[0055] According to this embodiment, the steps for selecting the generative model, as shown in Fig. 2, include:
[0056] • Extracting a summary and representative portion of the entire digitized image to reduce the computation time needed for identifying the generative model to be used from the plurality of trained generative models available ;
[0057] • Applying the i-th generative model from the plurality of trained generative models to obtain a corresponding image with colouring style and functional information belonging to the domain of the microscopy technique associated with that i-th generative model; • Calculating quantitative metrics to minimize the presence of artifacts that could make the image unrealistic, such as intensity discontinuities. The metrics calculated between the real image and the generated one can be of the full-reference type (i.e., Peak Signal-to-Noise Ratio and Structural Similarity Index) or segmentation-based (i.e., Dice Similarity Coefficient and Hausdorff Distance) ;
[0058] • Once all i-th generative models have been applied and the related metrics calculated, the generative model that provides the best performance according to those image quality metrics is selected.
[0059] The summary and representative portion of the entire digitized image refers to selecting a number of regions, evenly distributed within the tissue, to provide a general representation of the content of the digitized image.
[0060] The step of selecting the generative model based on the evaluation of quantitative metrics, as described above, allows selecting the most suitable generative model depending on the provided digitized image.
[0061] Preferably, the selection of the generative model with the best performance in terms of quantitative metrics that assess image quality, as shown in Fig. 2, also includes a step of intensity normalization of the summary and representative portion of the entire digitized image, similar to the previously described method for generating microscopy images using trained generative models.
[0062] The trained generative models of the previously described method are trained according to the steps shown in Fig.
[0063] 3 : • In the first phase , a plurality of paired images is acquired from a dataset of histological samples that were previously treated with at least one stain or biomarker and subj ected to at least one microscopy technique . These samples are then divided into N patches to reduce the computation time necessary for identi fying the generative model based on quantitative metrics that assess the quality of the image generated by each model ;
[0064] • In the second phase , image co-registration algorithms are applied to both the input and output microscopy techniques to minimi ze any misalignment defects and to achieve an optimal , preferably perfect , match between cellular structures in the two techniques . These algorithms are based on estimating global trans formation matrices ( rigid, af fine , or homographic ) or local deformation fields , so that the two images share the same reference system . Co-registration of images refers to the algorithmic operation of aligning one image to another used as a reference . In general , the two images to be aligned correspond to the same anatomical area, acquired at di fferent times or with di f ferent sensors and methods ;
[0065] • The best pairs of patches are then selected for use in a subsequent phase of training the generative models , based on both quantitative metrics that assess registration quality, such as mutual information, correlation coefficient , and Target Registration Error ( TRE ) , and a visual check for overlap of cellular structures , distortion, or loss of detail following registration;
[0066] • Next, the selected patches are fed into one or more generative models based on artificial intelligence. A non-exhaustive list of generative model families includes: Generative Adversarial Networks (GANs) , Variational Autoencoders (VAE) , Flow-based models, Generative Moment Matching Networks (GMMN) , and diffusion models.
[0067] Said plurality of images is provided to a "generator" component, whose purpose is to generate, for each received image, a second image using the microscopy technique applied to the input image. The "generator" can receive, as an additional input, a vector of random numbers, called "noise," and use it to generate the image. For instance, for a type of model that does not include a "discriminator, " such as diffusion models, the generator uses noise to introduce variety among the images while simultaneously improving the quality of the generated image .
[0068] During the training process, the generator will be instructed through the use of appropriate metrics and loss functions to ensure the generation of images that are semantically consistent and share the same characteristics as real images, such as shape, texture, and colour.
[0069] The generative model may include a second component called the "discriminator, " whose role is to distinguish between real images and the images generated by the "generator." The "discriminator" will calculate appropriate similarity metrics between the real and generated images in order to identify any differences. These metrics include both global image parameters , such as colour and intensity, and local parameters , such as texture parameters , well-known to those skilled in the field . This training strategy is repeated for various configurations of generative model families to ensure that there is at least one trained generative model for each pair of microscopy modes .
[0070] Preferably, the training phase of the generative models , as shown in Fig . 3 , also includes a step of intensity normali zation of the entire digiti zed image , similar to the normali zation steps previously described .
[0071] These trained generative models allow for the generation of images using dif ferent microscopy techniques without the need to physically perform staining or conduct such operations on real tissue samples . This brings the advantage of reducing the time and costs required to obtain such images since laboratory operations on the samples are no longer necessary .
[0072] Figures 4 and 5 show examples of images generated by trained generative models based on the hematoxylin-eosin (H&E ) and phosphohistone ( PH3 ) staining pair, which are extremely realistic .
[0073] (H&E ) staining is used for the morphological characteri zation of tissues , while ( PH3 ) is an immunohistochemical stain that highlights cells actively engaged in the duplication process (mitosis ) .
[0074] In Figures 4 and 5 , the results of images generated by a generative model are shown, allowing for the creation of an image with the characteristics of the immunohistochemical technique starting from an input image with the characteristics of the histochemical technique of brightfield optical microscopy .
[0075] Speci fically, Fig . 4 shows images of gastric tissue , where figure ( a ) is a real image showing the characteristics of optical microscopy performed on a tissue sample with histochemical staining, image (b ) is a real image showing the characteristics of immunohistochemistry performed on the tissue sample , while image ( c ) is an image generated by a trained generative model , possessing the chromatic and functional characteristics of immunohistochemistry .
[0076] In Fig . 5 , images of esophageal tissue are shown, where figure ( a ) is a real image showing the characteristics of optical microscopy performed on a tissue sample with histochemical staining, image (b ) is a real image showing the characteristics of immunohistochemistry performed on the tissue sample , while image ( c ) is an image generated by a trained generative model , possessing the morphological and functional characteristics of immunohistochemistry .
Claims
CLAIMS1 . A computer-implemented method for generating multimodal microscopy images comprising morphological and functional information, said method comprising the steps of :• receiving a digiti zed image comprising a histological sample obtained using a microscopy technique ;• receiving a first identi fying parameter of an input microscopy technique related to the received image and a second identi fying parameter of an output microscopy technique related to an image to be generated;• selecting from a predefined plurality of trained generative models , each trained to generate an image according to a first microscopy technique from an image of a second microscopy technique , based on the first and second parameters , a generative model trained to obtain an image according to said output microscopy technique based on the image comprising said input microscopy technique ;• dividing the entire image into a plurality of N patches ;• applying to each n-th patch the selected generative model to generate for each n-th patch a corresponding k-th patch having characteristics of the said output microscopy technique ;• for each k-th patch, applying a cropping technique to delimit a zone within which said patch remains unchanged and applying a smoothing filter to the outside of said zone , preferably at the edges , to reduce artifacts between said patch and the subsequent one , to allow theaggregation of said k-th patches ;• generating a complete output image by composing said k- th patches .2 . The method according to claim 1 , wherein the training of a generative model comprises the steps of :• acquiring a plurality of images of histological samples obtained from a dataset of histological samples previously treated with at least one dye or biomarker and subj ected to at least one microscopy technique ;• dividing the entire images into N patches ;• applying at least one algorithm for co-registration of images in the two microscopy modalities of input and output to ensure that the images share the same reference system to achieve optimal correspondence of cellular structures ;• selecting pairs of patches to be used in the subsequent training phase of generative models based on at least one criterion among : quantitative metrics , preferably mutual information, correlation coef ficient , and / or Target Registration Error ( TRE ) , visual inspection of cellular structure overlap, distortion, or detail loss ;• training at least one arti ficial intelligence-based generative model comprising a "generator" component instructed using metrics and loss functions to produce images consistent with the corresponding real images .3 . The method according to claim 2 , wherein the generative model in the training phase is based on an arti ficial intelligence model selected from a list including :Generative Adversarial Networks (GANs) , VariationalAutoencoders (VAE) , Flow-based models, Generative Moment Matching Networks (GMMN) , diffusion models .
4. The method according to at least one of claims 2 to 3, wherein the generative model in the training phase receives, in addition to the pairs of patches, a vector of random numbers defined as "noise" to generate an image to form texture and colour in relation to the same real image.
5. The method according to at least one of claims 2 to 3, wherein the generative model in the training phase also includes a "discriminator" component tasked with identifying differences between a real image and an image generated by the generative model according to similarity metrics including global parameters such as color and intensity and local parameters such as texture .
6. The method according to claim 1, wherein the phase of applying is preceded by a phase of normalizing the intensity of the n-th patches, where depending on the microscopy technique of the input image, the normalization is one of: forced color variation towards a target coloration of a reference digital image, or histogram equalization of luminosities.
7. The method according to claim 2, wherein the phase of selecting is preceded by a phase of normalizing theintensity of the entire digiti zed image , where depending on the microscopy technique of the input image , the normali zation is one of : forced color variation towards a target coloration of a reference digital image , or histogram equali zation of luminosities .8 . The method according to claim 1 , wherein the step of selecting the generative model is preceded by the steps of :• extracting a summary and representative portion of the entire digiti zed image by selecting a certain number of regions evenly distributed within the tissue , to have a general representation of the content in the digiti zed image , to reduce the computation time needed for the generative model selection phase ;• applying to the representative image the i-th generative model from the plurality of generative models to obtain a corresponding image with colouring style and functional information belonging to the domain of the microscopy technique associated with said i-th generative model ;• calculating quantitative metrics of full-reference and / or segmentation-based type for the image generated by the i-th model to identi fy generated images that are not realistic ;• selecting the generative model that provides the best performance according to metrics related to the quality of the generated image .
9. The method according to claim 8, wherein the metrics in the calculating phase are at least one of: Peak Signal- to-Noise Ratio, Structural Similarity Index, Dice Similarity Coefficient, Hausdorff Distance.
10. The method according to claim 8, wherein the phase of applying is preceded by a step of normalizing the intensity of the summary and representative portion, where depending on the microscopy technique of the input image, the normalization is one of: forced color variation towards a target coloration of a reference digital image, or histogram equalization of luminosities .
11. The method according to claim 1, wherein the complete image is saved in a non-proprietary original format or in a pyramid TIFF format.