A pathological section image virtual restaining method, system, device and medium

Abstract

The application provides a virtual restaining method of pathological section images, comprising the following steps: obtaining a histopathological image of a pathological section; constructing a virtual restaining network of the histopathological image based on a diffusion model; training the virtual restaining network, and taking the trained virtual restaining network as a virtual restaining model of a pathological image; segmenting the histopathological image into a plurality of segmented images; virtually restaining the segmented images through the virtual restaining model of the pathological image to obtain restained segmented images; and splicing the restained segmented images to obtain a restained histopathological image. The application can directly generate an IHC image from a scanned HE staining image, avoids time, money, manpower and material resources spent in the IHC section manufacturing process, makes clinical pathological diagnosis more convenient, and promotes the development of the medical and health industry.

Description

TECHNICAL FIELD

[0001] The present application relates to the medical technology field, and in particular to a pathological section image virtual restaining method, system, device and medium. BACKGROUND

[0002] Cancer is one of the most important diseases that endanger human health today. According to WHO data, millions of people die from various cancers every year. The key to cancer treatment lies in early detection, early diagnosis and early treatment. Among them, the early and accurate diagnosis of cancer is the key. Cancer diagnosis relies on imaging, clinical laboratory and pathological examination and other means, and pathological examination is the gold standard for cancer diagnosis.

[0003] The general process of pathological examination is to first cut a certain thickness of thin section (usually 5um) of the tissue on a glass slide, and then stain it with different dyes to expose the special structure between tissues and cells, so as to achieve the purpose of diagnosis. In clinical diagnosis, hematoxylin-eosin (HE) is generally used for staining first. Although through this step, part of the tumor can make a preliminary benign and malignant diagnosis, but sometimes the result of HE staining alone is not enough to make an accurate diagnosis. Accurate typing diagnosis of tumors is very important, which is related to the choice of treatment and prognosis of the disease. For example, although both are renal cell carcinoma, different types of renal cell carcinoma have different treatment methods and prognosis; renal clear cell carcinoma and renal chromophobe carcinoma have lower malignant degree, so under the condition of allowing, kidney-sparing surgery should be performed, while papillary renal carcinoma has higher malignant degree, and in the case of being unable to ensure that the tumor is completely removed, the tumor and the affected kidney should be removed together, and active adjuvant therapy should be performed after surgery. Therefore, in order to achieve accurate diagnosis in clinical practice, immunohistochemical staining (IHC) based on antigen and antibody specific binding is needed to assist diagnosis on the basis of HE staining.

[0004] The preparation process of IHC stained pathological section includes steps such as specimen fixation, embedding section preparation, baking, deparaffinization, antigen repair, biotin blocking, primary antibody incubation, secondary antibody incubation, DAB color development, hematoxylin restaining dehydration and transparent mounting. In order to prepare a high-quality IHC section, 5-7 days of time and hundreds or thousands of yuan of money are needed, and several experienced technicians are needed to complete the process. On the one hand, this leads to high investment in manpower, material resources, time and money; on the other hand, due to the complexity of the process and the long preparation time, this technology cannot be applied to intraoperative pathological biopsy, which affects the judgment of the surgeon on the scope of resection, leading to problems such as too small resection range for some patients with high malignant degree and too large resection range for some patients with low malignant degree, affecting the treatment and prognosis of patients. SUMMARY

[0005] The application aims to provide a pathological section image virtual restaining method, system, device and medium, which realizes direct generation of IHC images from scanned HE staining images.

[0006] In order to achieve the above-mentioned purpose, the application provides the following technical scheme: a pathological section image virtual restaining method, comprising the following steps: obtaining a histopathological image of a pathological section; constructing a virtual restaining network of the histopathological image based on a diffusion model; training the virtual restaining network, and taking the trained virtual restaining network as a pathological image virtual restaining model; segmenting the histopathological image into a plurality of block images; virtually restaining the block images through the pathological image virtual restaining model to obtain restained block images; and splicing the restained block images to obtain a restained histopathological image.

[0007] Further, the histopathological image comprises an HE image.

[0008] Further, before the virtual restaining of the block images through the pathological image virtual restaining model, the block images are numbered.

[0009] Further, the splicing of the restained block images comprises: determining the splicing position of each restained block image according to the number before the restaining of the restained block images; and splicing the restained block images according to the splicing position.

[0010] Further, the pathological image virtual restaining model comprises: an up-sampling module, a down-sampling module, a denoising module and a linear low-pass filter module, wherein the denoising module comprises a DDIM module or a DDPM module; the up-sampling module can extract the structural features of the block images, the DDIM module or the DDPM module can remove the noise in the structural features of the block images and set the staining style of the block images, the linear low-pass filter module can integrate the structural features of the block images and the staining style of the block images, and the down-sampling module can output the integrated structural features of the block images and the staining style of the block images as restained block images.

[0011] Further, the training of the virtual restaining network comprises: obtaining existing IHC images; adding noise to the existing IHC images; and removing the noise of the existing IHC images through the DDIM module or the DDPM module.

[0012] Further, the staining styles that can be set by the DDIM module or the DDPM module comprise: ILVR style, SDEdit style and EGSDE style.

[0013] In another aspect, a virtual restaining system of a pathological section image is provided, comprising: a pathological image acquisition module, acquiring a histopathological image of a pathological section; a staining network construction module, constructing a virtual restaining network of the histopathological image based on a diffusion model; a model training module, training the virtual restaining network, and taking the trained virtual restaining network as a pathological image virtual restaining model; an image segmentation module, segmenting the histopathological image into a plurality of segmented images; a staining module, virtually restaining the segmented images by the pathological image virtual restaining model to obtain restained segmented images; and a splicing module, splicing the restained segmented images to obtain a restained histopathological image.

[0014] In another aspect, an electronic device is provided, comprising: a processor and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-mentioned virtual restaining method.

[0015] In another aspect, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the computer-readable storage medium, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the above-mentioned virtual restaining method.

[0016] It can be analyzed that the present application discloses a virtual restaining method of a pathological section image, and the present application can directly generate an IHC image from a scanned HE staining image, avoids time, money, manpower and material resources spent in the IHC section manufacturing process, makes clinical pathological diagnosis more convenient, and promotes the development of medical and health undertakings. BRIEF DESCRIPTION OF DRAWINGS

[0017] The drawings accompanying the specification of this application are used to provide further understanding of the present application, the illustrative embodiments of the present application and the description thereof serve to explain the present application, and do not constitute an improper limitation on the present application. Among them:

[0018] Figure 1 Flowchart of an embodiment of the present application.

[0019] Figure 2 Virtual restaining process schematic diagram of an embodiment of the present application.

[0020] Figure 3 Virtual restaining model workflow schematic diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0021] The present application will be described in detail below with reference to the attached drawings and in conjunction with embodiments. The various examples are provided by way of explanation of the present application and are not meant as limiting the present application. It will be apparent to those skilled in the art that modifications and variations can be made in the present application without departing from the scope or spirit of the application. For example, features shown or described as part of one embodiment can be used in another embodiment to yield still a further embodiment. It is, therefore, desired that what is claimed be what the application is intended to cover.

[0022] One or more examples of the present application are illustrated in the accompanying drawings. The detailed description uses numerical and letter designations to refer to features in the drawings. Like or similar designations in the drawings and description have been used to refer to like or similar parts of the application. As used herein, the terms "first", "second", "third", and "fourth" and the like can be used interchangeably to distinguish one component from another and are not intended to signify location or importance of the individual components.

[0023] As shown in Figure 1 and Figure 2 According to an embodiment of the present application, a virtual restaining method of a pathological section image is provided, comprising the following steps:

[0024] Step S101, obtaining a histopathological image of a pathological section; the histopathological image comprises an HE image.

[0025] The histopathological image of the pathological section obtained above is usually obtained by scanning the pathological section, and the histopathological image comprises an HE (hematoxylin-eosin staining) image. The present application generates an IHC image based on the HE image.

[0026] The present application does not need to label the pathological section after obtaining the pathological section, but directly scans the original pathological section.

[0027] Step S102, constructing a virtual restaining network of the histopathological image based on a diffusion model.

[0028] The virtual restaining network is established for the need of restaining the pathological section. The virtual restaining network is a virtual restaining model of an untrained pathological image. The virtual restaining network is established based on a diffusion model and usually comprises a training data part and an image generation part. The training data part can perform image denoising training based on an existing IHC block image, and the image generation part can perform staining on the image.

[0029] Step S103, training the virtual restaining network, and taking the trained virtual restaining network as a virtual restaining model of a pathological image.

[0030] wherein training the virtual restaining network comprises:

[0031] At step S1031, an existing IHC image is obtained.

[0032] Based on the target domain dataset (IHC patch image set) composed of real IHC patch images, the existing real IHC image is used as a training sample, and the training of the pathological image virtual restaining model can be more accurate through the real IHC image.

[0033] At step S1032, noise is added to the existing IHC image.

[0034] The IHC image is added with noise to form an IHC image with noise, thereby providing a denoising training sample for the DDIM module or the DDPM module (denoising diffusion probability model and denoising diffusion implicit model). In use, the DDPM module and the DDIM module are usually trained at the same time, and one of the models (DDIM module or DDPM module) can be selected each time to generate in combination with different staining methods. In actual application, the number of fuzzy iterations t of the DDIM module or the DDPM module can be appropriately reduced.

[0035] At step S1033, the noise of the existing IHC image is removed through the DDIM module or the DDPM module.

[0036] As shown in Figure 3 , by repeatedly adding noise to the IHC image, the training process includes a forward noise adding process and a backward noise removing process. In the process of continuously adding noise to the target domain IHC patch image to become a real noise image and continuously removing noise from the real noise image to restore the target domain IHC patch image, the noise removing process is learned. Thus, the noise is removed from the real noise image, and the DDIM module or the DDPM module can effectively remove the noise in the IHC image.

[0037] The trained pathological image virtual restaining model comprises an upsampling module, a downsampling module, a DDIM module or a DDPM module, and a linear low-pass filter module.

[0038] The upsampling module can extract the structural features of the patch image, the DDIM module or the DDPM module can remove the noise in the structural features of the patch image and set the staining style of the patch image, the linear low-pass filter module can integrate the structural features of the patch image and the staining style of the patch image, and the downsampling module can output the integrated structural features of the patch image and the staining style of the patch image as a restained patch image.

[0039] The dyeing styles that can be set by the DDIM module or the DDPM module described above include an ILVR (iterative latent variable refinement) style, an SDEdit (stochastic difference edit) style, and an EGSDE (energy-guided stochastic differential equation) style.

[0040] In step S104, the histopathological image is segmented into a plurality of patch images.

[0041] The segmented image is an HE image, and the HE image is segmented into a plurality of patches.

[0042] Since the pixels of a complete HE image are large, a large amount of calculation is required if the complete HE image is directly processed, and the image quality generated by directly processing the complete image is also low. Currently, there is no method for processing a complete image. The segmentation and dyeing are performed in sequence to reduce the calculation amount of a single process and improve the detail effect.

[0043] Before the patch images are virtually re-dyed by the pathological image virtual re-dyeing model, the patch images are numbered, so that each number corresponds to the position of the patch image in the original histopathological image.

[0044] When the pathological image is segmented, the segmentation is usually performed according to a ratio of 256*256 (the pixel size of each patch image is 256*256), and other ratios can also be selected according to actual requirements.

[0045] In step S105, the patch images are virtually re-dyed by the pathological image virtual re-dyeing model to obtain re-dyed patch images.

[0046] The re-dyeing of the patch images is implemented based on the pathological image virtual re-dyeing model, and the dyeing styles of the re-dyed patch images remain consistent under the processing of the pathological image virtual re-dyeing model.

[0047] In step S106, the re-dyed patch images are spliced to obtain a re-dyed histopathological image.

[0048] The splicing of the re-dyed patch images described above includes:

[0049] The splicing position of each re-dyed patch image is determined according to the number of the re-dyed patch image before dyeing, and the re-dyed patch images are spliced according to the splicing positions.

[0050] Since the images are numbered after segmentation, the re-dyed histopathological image, that is, the IHC image, can be spliced from the re-dyed patch images according to the numbers of the re-dyed patch images, and the pathologist can further perform typing diagnosis by using the re-dyed histopathological image.

[0051] Since the pathological image virtual restaining model of the application is established based on a diffusion model, the edges of each small block generated by the virtual restaining of the pathological image virtual restaining model are subjected to homogenization transition processing, so that the staining style of the restained block image remains consistent, and there is no obvious trace at the splicing position, avoiding the block effect between the blocks after splicing due to too large difference.

[0052] The application further discloses a virtual restaining system of a pathological section image, which comprises a pathological image acquisition module, a staining network construction module, a model training module, an image segmentation module, a staining module and a splicing module.

[0053] The application further discloses an electronic device, which comprises a processor and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the virtual restaining method.

[0054] The application further discloses a computer readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to realize the virtual restaining method.

[0055] From the above description, it can be seen that the above-mentioned embodiments of the application achieve the following technical effects: the application can directly generate an IHC image from a scanned HE staining image, avoids the time, money and manpower and material resources spent in the IHC section manufacturing process, makes the clinical pathological diagnosis more convenient, and promotes the development of the medical and health industry.

[0056] The above description is only preferred embodiments of the application and is not used to limit the application. For those skilled in the art, the application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A method of virtual restaining of a pathology slide image, characterized in that, The method comprises the following steps: obtaining a histopathological image of a pathological section; constructing a virtual restaining network of the histopathological image based on a diffusion model; training the virtual restaining network, and taking the trained virtual restaining network as a pathological image virtual restaining model; segmenting the histopathological image into a plurality of block images; virtually restaining the block images through the pathological image virtual restaining model to obtain restained block images; stitching the restained block images to obtain a restained histopathological image; numbering the block images before virtually restaining the block images through the pathological image virtual restaining model; the stitching of the restained block images comprises: determining the stitching position of each restained block image according to the number before restaining of the restained block image; stitching the restained block images according to the stitching position; the pathological image virtual restaining model comprises an up-sampling module, a down-sampling module, a denoising module and a linear low-pass filter module, wherein the denoising module comprises a DDIM module or a DDPM module; the up-sampling module can extract the structural features of the block image, the DDIM module or the DDPM module can remove the noise in the structural features of the block image and set the staining style of the block image, the linear low-pass filter module can integrate the structural features of the block image and the staining style of the block image, and the down-sampling module can output the integrated structural features of the block image and the staining style of the block image as a restained block image; the training of the virtual restaining network comprises: obtaining an existing IHC image; adding noise to the existing IHC image; removing the noise of the existing IHC image through the DDIM module or the DDPM module; the staining styles that can be set by the DDIM module or the DDPM module include ILVR style, SDEdit style and EGSDE style.

2. The method of claim 1, wherein the method further comprises: The histopathological image comprises an HE image.

3. A virtual restaining system of a pathology slide image, characterized in that, comprise: a pathological image acquisition module that acquires a histopathological image of a pathological section; a staining network construction module that constructs a virtual restaining network of the histopathological image based on a diffusion model; an image segmentation module that segments the histopathological image into a plurality of block images; The model training module trains the virtual restaining network, and the trained virtual restaining network is used as a pathological image virtual restaining model. The pathological image virtual restaining model comprises an upsampling module, a downsampling module, a denoising module and a linear low-pass filter module. The denoising module comprises a DDIM module or a DDPM module. The upsampling module can extract structural features of the divided images. The DDIM module or the DDPM module can remove noise in the structural features of the divided images and set a staining style of the divided images. The linear low-pass filter module can integrate the structural features of the divided images and the staining style of the divided images. The downsampling module can output the integrated structural features of the divided images and the staining style of the divided images as restained divided images. The training of the virtual restaining network comprises the following steps: obtaining existing IHC images; adding noise to the existing IHC images; removing noise from the existing IHC images by the DDIM module or the DDPM module. The staining styles that can be set by the DDIM module or the DDPM module include an ILVR style, an SDEdit style and an EGSDE style. The staining module virtually restains the divided images by the pathological image virtual restaining model to obtain restained divided images. Before the virtual restaining of the divided images by the pathological image virtual restaining model, the divided images are numbered. The splicing module splices the restained divided images to obtain a restained histopathological image. The splicing of the restained divided images comprises the following steps: determining a splicing position of each restained divided image according to the number of the restained divided image before staining; and splicing the restained divided images according to the splicing positions.

4. An electronic device, comprising: The electronic device comprises a processor and a memory for storing executable instructions of the processor. The processor is configured to perform the virtual restaining method in any one of claims 1-2.

5. A computer readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the virtual restaining method in any one of claims 1-2.

Images