Virtual HE dyeing system of undyed and unmarked bright field image based on deep learning
By combining conventional optical microscopy with deep learning technology, and using bright-field microscopy for non-destructive imaging to generate virtual H&E staining images, the complexity and high cost of traditional H&E staining processes are solved, achieving low-cost, non-destructive, and efficient virtual staining, thus improving the quality of pathological analysis.
Patent Information
- Application Number
- CN202511539826.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-27
- Publication Date
- 2026-02-13
AI Technical Summary
Traditional H&E staining is complex, costly, requires professional personnel and facilities, is prone to damaging samples and is difficult to re-stain. Existing virtual H&E staining technology relies on expensive and complex equipment and is difficult to operate.
By combining conventional optical microscopy with deep learning technology, non-destructive imaging is achieved through bright-field microscopy, and virtual H&E staining images are generated using deep learning models, avoiding damage from chemical reagents, reducing costs, and simplifying operations.
It enables efficient and accurate virtual H&E staining of low-cost, non-destructive samples, reduces human error, preserves the integrity of tissue samples, creates conditions for multi-angle analysis, and promotes the development of the biomedical field.
Smart Images
Figure CN121521854A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of information technology and biomedicine, in particular to a system for generating virtual H&E staining images by combining deep learning technology and unstained, unlabeled bright-field images. BACKGROUND
[0002] Currently, Hematoxylin and eosin (H&E) staining, as the gold standard in the field of histological analysis, plays an irreplaceable role in many key aspects of modern medicine. In the intraoperative scenario, doctors can provide a solid and reliable basis for surgical treatment decisions with the help of H&E staining results, which directly affects the direction of surgery and the prognosis of patients. After surgery, by observing the H&E stained tissue samples, the wound healing effect can be effectively monitored, potential problems can be found in time and appropriate measures can be taken. H&E staining provides accurate postoperative prognosis for doctors, helping to develop subsequent treatment plans for patients. However, the traditional H&E staining process in the histopathological workflow faces many difficulties. From the collection, fixation, dehydration, embedding of tissue samples, to slicing, staining and finally mounting observation, it not only involves a large number of chemical reagents and professional equipment, but also requires a large amount of manpower. From sample processing to finally obtaining a stained section for observation, it often takes several hours or even days, which is relatively inefficient. In addition, such a complex operation process requires high professional skills of technicians, and only experienced and skilled personnel who have undergone long-term training can ensure the accuracy of the staining results. More importantly, the process of tissue chemical staining has certain risks. In the staining process, chemical reagents may cause damage to tissue sections, affecting the original structure and morphology of tissue cells, and thus interfering with the accuracy of judgment. Moreover, once the traditional H&E staining is completed, it becomes extremely difficult to perform other types of staining on the same tissue section, which greatly limits the in-depth development of research when multi-dimensional analysis of tissue samples is required.
[0003] While existing virtual H&E staining techniques offer new insights into solving some problems of traditional staining, they also have their limitations. Currently, these virtual H&E staining techniques mainly rely on expensive and nonlinear optical microscopy or structured illumination microscopy. To achieve high-resolution imaging, these microscopes typically require high-precision peripheral equipment, which is extremely difficult to obtain. For example, multiple lasers of specific frequencies are needed, emitting different wavelengths to meet different imaging requirements; supercontinuum light sources can provide a wider spectral range, enhancing image contrast and detail; and highly sensitive optical detectors are indispensable for capturing weak optical signals. However, these devices are not only expensive but also limited in market supply, making them unaffordable for many research and medical institutions. Furthermore, the operation of these microscopes is extremely complex, requiring specialized technicians for debugging and maintenance, and their imaging process is strictly limited by scanning time and excitation intensity. For example, autofluorescence imaging requires a high-sensitivity fluorescence camera capable of capturing extremely weak fluorescence signals to achieve high-quality imaging. Simultaneously, a near-ultraviolet laser and a high-resolution DAPI filter (OSFI3-DAPI-5060C) are required, with an excitation wavelength of 377nm and an emission wavelength of 447nm. In actual operation, excessively high laser excitation intensity may damage the sample, while insufficient intensity will result in an inability to obtain a clear image; conversely, excessively long scanning times not only reduce work efficiency but may also cause changes in the sample, affecting the imaging results. Summary of the Invention
[0004] In view of this, the present invention provides a virtual H&E staining system and method based on deep learning for unstained, label-free bright-field images, which at least solves some of the above-mentioned technical problems. This system combines ordinary optical microscopy with deep learning technology, opening a new path for a simplified virtual H&E staining method. Using an ordinary bright-field microscope, unstained, label-free tissue sections can be imaged non-destructively, avoiding the damage to the sample caused by chemical reagents in traditional staining processes. Then, image processing equipment equipped with an advanced deep learning model (virtual H&E staining network) is used to analyze and process the acquired images, achieving accurate virtual H&E staining. This technology has significant advantages, achieving breakthroughs in many of the challenges faced by traditional chemical staining. It greatly reduces costs, avoids the biotoxicity risks of chemical reagents in traditional staining, and reduces the fluctuations and deviations in results caused by manual operation in sample processing, reagent preparation, and staining time control, which are affected by the operator's skill level and work status. At the same time, this technology fully preserves the integrity and authenticity of the original tissue sample, creating favorable conditions for subsequent in-depth analysis of the tissue sample from multiple angles and levels.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0006] In a first aspect, embodiments of the present invention provide a virtual H&E staining system based on deep learning for unstained, label-free bright-field images. The system includes: a bright-field microscope and an image processing device, wherein:
[0007] The bright-field microscope is used for non-destructive imaging of unstained and unlabeled tissue sections;
[0008] The image processing device includes: a data acquisition module, an image preprocessing module, and a virtual H&E staining network constructed based on a deep learning model, wherein:
[0009] The data acquisition module is used to acquire images of unstained and unlabeled tissue sections captured by a bright-field microscope, and at the same time, collect the corresponding H&E images to provide control samples for subsequent model training.
[0010] The image preprocessing module is used to perform image registration and color normalization operations on the acquired images;
[0011] The virtual H&E staining network generates virtual H&E staining images based on the preprocessed images.
[0012] In an optional implementation, the data acquisition module acquires stable and reliable images of unstained, label-free tissue sections by capturing multiple bright-field images at incremental z-axis height, using image entropy analysis to assess variability trends.
[0013] In an optional implementation, the image registration of the image preprocessing module includes translation and rotation operations.
[0014] In an optional implementation, the image preprocessing module uses the Reinhard color normalization method to process image color differences and achieve image data standardization.
[0015] In an optional implementation, the virtual H&E staining network employs a generative adversarial network, comprising: a generator G and a discriminator D, wherein:
[0016] The generator G adopts a U-Net structure. First, it extracts multi-scale features of the unlabeled image through the encoder, then combines skip connections and upsampling to complete feature fusion, and finally optimizes with adversarial loss and reconstruction / perceptual loss to generate the corresponding virtual H&E staining image.
[0017] Discriminator D uses a PatchGAN structure to extract multi-scale features from real / generated H&E images, distinguishes the image source by predicting the probability of authenticity, and evaluates the accuracy of the generated virtual H&E staining image.
[0018] In an optional implementation, PSNR, FID, and SSIM are used to evaluate the accuracy of the generated virtual H&E staining images.
[0019] Secondly, embodiments of the present invention also provide a virtual H&E staining method for unstained, label-free bright-field images based on deep learning, characterized in that a virtual H&E staining image is generated by applying the aforementioned virtual H&E staining system for unstained, label-free bright-field images based on deep learning, and the method includes:
[0020] The tissue sections to be analyzed are placed under a bright-field microscope to acquire unstained and unlabeled tissue section images. After data acquisition optimization and image preprocessing operations are performed on the acquired images, they are input into a virtual H&E staining network that has been trained, validated, and tested and optimized. The network processes and transforms the input images according to the learned mapping relationship to generate virtual H&E staining images.
[0021] Compared with existing technologies, this invention provides a virtual H&E staining system and method for unstained, label-free bright-field images based on deep learning, overcoming the limitations of existing technologies and having at least the following beneficial technical effects:
[0022] This invention combines ordinary optical microscopy with deep learning technology, paving a new path for a simplified virtual H&E staining method. Using a standard bright-field microscope, unstained and unlabeled tissue sections can be imaged non-destructively, avoiding the damage to samples caused by chemical reagents in traditional staining processes. Then, image processing equipment equipped with an advanced deep learning model (virtual H&E staining network) analyzes and processes the acquired images to achieve accurate virtual H&E staining.
[0023] This technology offers significant advantages, achieving breakthroughs in many areas where traditional chemical staining faces numerous challenges. It greatly reduces costs, avoids the biotoxicity risks associated with chemical reagents in traditional staining, and minimizes fluctuations and deviations in results caused by manual operation in sample processing, reagent preparation, and staining time control, which are affected by the operator's skill level and work status.
[0024] Meanwhile, this technology fully preserves the integrity and authenticity of the original tissue samples, creating favorable conditions for subsequent in-depth analysis from multiple angles and levels. This will greatly stimulate scientists in related fields to explore the relationship between the structure and function of biological samples, and will have immeasurable significance for assisting medical judgment and improving the efficiency of drug development, and is expected to propel the entire biomedical field into a new stage of development.
[0025] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.
[0026] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0028] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.
[0029] Figure 1 This is a schematic diagram of the virtual H&E staining technique provided in an embodiment of the present invention.
[0030] Figure 2 This is a schematic diagram of the virtual H&E staining technique provided in an embodiment of the present invention. Detailed Implementation
[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.
[0032] In the description of this invention, it should be noted that some processes described in this application specification and drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may be performed in any order or in parallel. Furthermore, various numbers are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0033] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0034] This invention aims to combine deep learning and bright-field microscopy to address the problems of traditional hematoxylin and eosin (H&E) staining and existing virtual H&E staining techniques. Specifically, it addresses the issues of complex procedures, high costs, reliance on professional personnel and facilities, sample damage, and difficulty in secondary staining associated with traditional H&E staining. It also overcomes the challenges of existing virtual H&E staining, which relies on expensive and complex equipment and is difficult to operate, thereby achieving efficient and accurate acquisition of virtual H&E staining images with simple equipment, low cost, and non-destructive sample handling, thus improving the quality of pathological analysis.
[0035] The purpose of this invention is to provide a virtual H&E staining function that utilizes a deep learning model to achieve real-time, accurate, safe, and uniform accuracy. It has the advantages of high portability and high speed, and is dedicated to rapid imaging and assisting doctors in making judgments. This invention has low equipment configuration requirements, is simple to operate, and has good stability and economy.
[0036] See Figure 1 and Figure 2 As shown, to overcome the above problems and achieve the above objectives, this invention provides a virtual H&E staining system based on deep learning for unstained, label-free bright-field images. The system includes: a bright-field microscope and an image processing device, wherein:
[0037] Bright-field microscopes are used for non-destructive imaging of unstained and unlabeled tissue sections.
[0038] The image processing device includes: a data acquisition module, an image preprocessing module, and a virtual H&E staining network built based on a deep learning model, wherein:
[0039] The data acquisition module is used to acquire images of unstained and unlabeled tissue sections captured by a bright-field microscope, and at the same time, collect the corresponding H&E images to provide control samples for subsequent model training.
[0040] The image preprocessing module is used to perform image registration and color normalization operations on the acquired images;
[0041] A virtual H&E staining network generates virtual H&E staining images based on preprocessed images.
[0042] The specific implementation methods and working principles of the system of the present invention will be described in detail below:
[0043] Cells and their nuclei are typically in the micrometer range, a scale that falls precisely within the theoretical resolution (200 nanometers) achievable by traditional optical microscopes. Based on this characteristic, this invention combines ordinary optical microscopy with deep learning technology, paving a new path for a simplified virtual H&E staining method. Using a standard bright-field microscope, unstained and unlabeled tissue sections can be imaged non-destructively, avoiding damage to the sample from chemical reagents during traditional staining. Then, an advanced deep learning model is used to analyze and process the acquired images, achieving accurate virtual H&E staining. The specific implementation method is as follows:
[0044] 1. Hardware equipment selection:
[0045] In this embodiment of the invention, a common bright-field microscope is used as the imaging device. It is simple in structure, low in cost, and easy to acquire and operate. The bright-field microscope can perform non-destructive imaging of unstained and unlabeled tissue sections, avoiding the damage to the sample caused by chemical reagents during traditional staining processes. This maximizes the preservation of the integrity and authenticity of the original tissue sample, providing reliable raw data for subsequent image processing.
[0046] 2. Data Collection:
[0047] When acquiring label-free images with constant spectral transmittance at a fixed z-axis height, the images are easily affected by environmental vibrations, temperature changes, and slice thickness, hindering subsequent neural network training and reducing performance. To effectively address this issue, this invention captures multiple bright-field images at incremental z-axis heights and uses image entropy analysis to assess variability trends, ensuring the acquisition of stable and reliable unstained, label-free tissue slice images. Simultaneously, corresponding H&E images are collected to provide control samples for subsequent model training. Specifically, multiple bright-field images are captured incrementally along the z-axis height (different z-axis heights result in varying transmittance distributions in the acquired bright-field images). Image entropy analysis is then used to quantify the disorder of grayscale distribution in these differing images, thereby accurately assessing the trend of image transmittance changes. This image entropy analysis method effectively overcomes the difficulty in judging data reliability due to differences in transmittance distribution at different z-axis heights, providing a quantitative basis for selecting stable and reliable image data, and ultimately laying a high-quality data foundation for subsequent image processing and model training.
[0048] 3. Image preprocessing:
[0049] ① Image registration: This involves translation and rotation operations and is divided into two steps: coarse registration and fine registration. Coarse registration quickly adjusts the approximate position of the image, laying the foundation for fine registration; fine registration further optimizes image details, enabling different images to be precisely aligned in spatial position, facilitating subsequent analysis and processing.
[0050] ② Coloring Normalization: The Reinhard coloring normalization method is used to handle image color differences and standardize image data. This not only enhances the convergence and generalization ability of the virtual H&E coloring network method, but also makes the images display consistent color balance and contrast, improving image quality and providing higher-quality data for model training.
[0051] 4. Deep learning model construction (virtual H&E staining network):
[0052] ① Network Structure: The generator G adopts a U-Net structure. It first extracts multi-scale features from the unlabeled image (bright-field imaging under white light) through an encoder, then combines skip connections and upsampling to complete feature fusion. Finally, it optimizes with adversarial loss and reconstruction / perceptual loss to generate the corresponding virtual H&E image. The discriminator D adopts a PatchGAN structure, extracting multi-scale features from the real / generated H&E images. It distinguishes the image source by predicting the probability of authenticity, and optimizes with adversarial loss. For evaluation, quantitative parameters such as PSNR, FID, and SSIM are used.
[0053] ② Training Process: Training was conducted on a server equipped with a GeForce RTX 3090 graphics processing unit (GPU). The training dataset contained 24,901 (512×512×1) unlabeled images of 14 glioma tissues and their corresponding H&E staining image pairs. Through complex supervised training, the model parameters were continuously optimized, enabling the generator G and discriminator D to cooperate and improve together.
[0054] ③ Validation and Testing: The validation dataset contains microscopic images of one glioma tissue, totaling 1,015 related image pairs, used to validate the model's performance and generalization ability during training. Testing was conducted on a computer equipped with a GeForce RTX 4090 GPU and an Intel i9-13900KF CPU. The test dataset contains microscopic images of six glioma tissues, totaling 18,253 image pairs, used to evaluate the final performance of the optimized virtual H&E staining network. After training, validation, and testing, the optimized virtual H&E staining network can accurately distinguish the differences in optical properties between the cell nucleus and cytoplasm, thus reconstructing pseudo-H&E staining images with high quality.
[0055] Experimental results show that the virtual H&E staining network achieves a resolution of 9.01 mm on the test set. 2 The algorithm achieved a speed of [speed value] / s and obtained a maximum average structural similarity index of 0.7744 and a correlation coefficient of 0.7338. A virtual staining network was used on 512×512×3 images (field of view ~0.286mm). 2The virtual staining network was processed with an average blind inference time of 0.042 s. Therefore, H&E staining of 100 images (5,440 × 3,648 pixels) can be completed in about 3.33 s, which is significantly shorter than the time required for traditional staining of frozen sections (about 5-10 minutes).
[0056] Furthermore, embodiments of the present invention also provide a virtual H&E staining method for unstained, label-free bright-field images based on deep learning. The method utilizes the aforementioned virtual H&E staining system for unstained, label-free bright-field images based on deep learning to generate virtual H&E staining images. The method includes:
[0057] The tissue sections to be analyzed are placed under a bright-field microscope to acquire unstained and unlabeled tissue section images. After data acquisition optimization and image preprocessing operations are performed on the acquired images, they are input into a virtual H&E staining network that has been trained, validated, and tested and optimized. The network processes and transforms the input images according to the learned mapping relationship to generate virtual H&E staining images.
[0058] The generated virtual staining images can be directly observed and analyzed on a computer. Doctors or researchers can clearly see the contrast between the cell nucleus and cytoplasm through the images, just like observing real H&E stained sections.
[0059] The method provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned system embodiment. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the aforementioned device embodiment, and will not be repeated here.
[0060] As will be apparent to those skilled in the art from the above description of the embodiments, the present invention proposes a virtual H&E staining system and method for unstained, label-free bright-field images based on deep learning, which has the following advantages and beneficial effects:
[0061] This invention combines ordinary optical microscopy with deep learning technology, paving a new path for a simplified virtual H&E staining method. Using a standard bright-field microscope, unstained and unlabeled tissue sections can be imaged non-destructively, avoiding the damage to samples caused by chemical reagents in traditional staining processes. Then, image processing equipment equipped with an advanced deep learning model (virtual H&E staining network) analyzes and processes the acquired images to achieve accurate virtual H&E staining. This technology offers significant advantages, achieving breakthroughs in many of the challenges faced by traditional chemical staining. It greatly reduces costs, avoids the biotoxicity risks associated with chemical reagents in traditional staining, and minimizes fluctuations and deviations in results caused by manual operation in sample processing, reagent preparation, and staining time control, which are affected by the operator's skill level and work status. Simultaneously, this technology fully preserves the integrity and authenticity of the original tissue sample, creating favorable conditions for subsequent in-depth analysis of the tissue sample from multiple angles and levels. This will greatly stimulate scientists in related fields to explore the relationship between the structure and function of biological samples, and will have immeasurable significance for assisting medical judgment and improving the efficiency of drug development. It is expected to promote the entire biomedical field to a new stage of development.
[0062] The embodiments of the present invention have been described in detail above, and the principles and implementation methods of the present invention have been explained. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention.
[0063] This specification uses a progressive approach in its description; similar or identical parts between different embodiments can be referred to interchangeably. Parts of the embodiments of this invention not described herein can be obtained from the corresponding product manuals or existing technologies in the field, and are considered well-known in the art, and will not be elaborated upon further.
[0064] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The words "a" or "an" preceding a component do not exclude the presence of a plurality of such components. This invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer.
[0065] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A virtual H&E staining system for unstained, label-free bright-field images based on deep learning, characterized in that, The system includes: a bright-field microscope and an image processing device, wherein: The bright-field microscope is used for non-destructive imaging of unstained and unlabeled tissue sections; The image processing device includes: a data acquisition module, an image preprocessing module, and a virtual H&E staining network constructed based on a deep learning model, wherein: The data acquisition module is used to acquire images of unstained and unlabeled tissue sections captured by a bright-field microscope, and at the same time, collect the corresponding H&E images to provide control samples for subsequent model training. The image preprocessing module is used to perform image registration and color normalization operations on the acquired images; The virtual H&E staining network generates virtual H&E staining images based on the preprocessed images.
2. The virtual H&E staining system for unstained, label-free bright-field images based on deep learning according to claim 1, characterized in that, The data acquisition module captures multiple bright field images at the incremental z-axis height, uses image entropy analysis to assess variability trends, and acquires stable and reliable images of unstained, label-free tissue sections.
3. The virtual H&E staining system for unstained, label-free bright-field images based on deep learning according to claim 1, characterized in that, The image registration in the image preprocessing module includes translation and rotation operations.
4. The virtual H&E staining system for unstained, label-free bright-field images based on deep learning according to claim 1, characterized in that, The image preprocessing module uses the Reinhard color normalization method to process image color differences and achieve image data standardization.
5. The virtual H&E staining system for unstained, label-free bright-field images based on deep learning according to claim 1, characterized in that, The virtual H&E staining network employs a generative adversarial network, comprising: a generator G and a discriminator D, wherein: The generator G adopts a U-Net structure. First, it extracts multi-scale features of the unlabeled image through the encoder, then combines skip connections and upsampling to complete feature fusion, and finally optimizes with adversarial loss and reconstruction / perceptual loss to generate the corresponding virtual H&E staining image. Discriminator D uses a PatchGAN structure to extract multi-scale features from real / generated H&E images, distinguishes the image source by predicting the probability of authenticity, and evaluates the accuracy of the generated virtual H&E staining image.
6. A virtual H&E staining system for unstained, label-free bright-field images based on deep learning according to claim 5, characterized in that, The accuracy of the generated virtual H&E staining images was evaluated using PSNR, FID, and SSIM.
7. A virtual H&E staining method for unstained, label-free bright-field images based on deep learning, characterized in that, Using a virtual H&E staining system based on deep learning for unstained, label-free bright-field images as described in any one of claims 1-6, a virtual H&E staining image is generated, the method comprising: The tissue sections to be analyzed are placed under a bright-field microscope to acquire unstained and unlabeled tissue section images. After data acquisition optimization and image preprocessing operations are performed on the acquired images, they are input into a virtual H&E staining network that has been trained, validated, and tested and optimized. The network processes and transforms the input images according to the learned mapping relationship to generate virtual H&E staining images.