Method and system for generating HE pathological images from CT images based on generative adversarial networks
By generating an adversarial network, the CT images are converted into pseudopathological images and constructing a diagnostic model, the limitations of CT scans in SCLC and NSCLC diagnosis are solved, high-precision non-invasive differential diagnosis is achieved, and the guidance of personalized treatment is improved.
Patent Information
- Application Number
- CN202411288769.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2044-09-14
AI Technical Summary
The diagnostic results of prior art in distinguishing early SCLC from NSCLC in lung cancer depend on observer experience, and CT scans have limitations, lack of high-precision non-invasive differential diagnosis methods, which affect personalized treatment decisions.
Using a method based on generative adversarial network, CT images are converted into pseudopathological images, and pseudopathological diagnostic models are constructed through transfer learning, which improves the non-invasive diagnostic accuracy of SCLC and NSCLC. Swin Transformer is used to build discriminators, and 256×256 and 64 are generated in stages. ×64 pseudopathological small images, combined with real pathological images for training and optimization.
It improves the preoperative diagnostic accuracy of lung cancer histological subtype, realizes histological subtype evaluation without invasiveness, guides the selection of personalized treatment methods, and reduces the dependence of diagnostic results on clinical experience.
Smart Images
Figure CN118800414B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of image processing, and in particular to a method and system for generating HE pathological images from CT images based on a generative adversarial network. Background Art
[0002] Lung cancer is mainly divided into two types: small cell lung cancer (SCLC) and non-small cell lung cancer (NSCLC). There are significant differences between the two in clinical manifestations, malignancy, growth characteristics and treatment methods. SCLC is highly invasive, with rapid tumor cell proliferation and high malignancy. It is very easy to metastasize to distant sites in the early stage and is usually in the late stage when diagnosed. Therefore, surgical treatment is not common. Chemotherapy and radiotherapy are the main treatments for SCLC. Although the initial efficacy is good, the recurrence rate is high and the prognosis is relatively poor. NSCLC grows relatively slowly and has a lower malignancy. For patients with early NSCLC, surgical resection is the preferred treatment method, and surgery can achieve the possibility of radical cure. According to the tumor stage and other pathological characteristics, adjuvant chemotherapy or radiotherapy after surgery can reduce the risk of recurrence and improve the survival rate. Therefore, early differential diagnosis of SCLC and NSCLC is crucial for the formulation of personalized treatment plans.
[0003] Pathological examination is the gold standard for differential diagnosis of SCLC and NSCLC, for example, pathological analysis is performed using (hematoxylin-eosintaining, HE) pathological images, but it requires obtaining samples through invasive biopsy or surgery. CT scanning is the most commonly used lung cancer examination method before surgery, but its diagnostic results are highly dependent on the clinical experience of the observer and still have limitations in distinguishing SCLC from NSCLC. Therefore, a high-precision non-invasive differential diagnosis method for SCLC and NSCLC before surgery is urgently needed in clinical practice. Summary of the invention
[0004] In response to the above-mentioned defects, the present invention proposes a method and system for generating HE pathological images from CT images based on a generative adversarial network, generates pseudo pathological images reflecting the microstructural characteristics of the tumor based on the global characteristics of the tumor on CT, and constructs a pseudo pathological differential diagnosis model to improve the non-invasive diagnostic accuracy of SCLC and NSCLC, and provide personalized treatment decision recommendations for lung cancer patients before surgery.
[0005] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:
[0006] The first aspect of the present disclosure provides a method for generating a HE pathology image from a CT image based on a generative adversarial network, comprising the following steps:
[0007] Obtain the patient's CT image, HE pathological image, and pathological diagnosis results, perform feature extraction on the CT image, and obtain CT image features;
[0008] Generate a first pseudo pathological image using the CT image feature, determine the authenticity of the first pseudo pathological image, and obtain a first loss;
[0009] The first loss is used as a generation condition, the first pseudo pathological image is used as input, a second pseudo pathological image is generated, and the authenticity of the second pseudo pathological image is judged to obtain a second loss;
[0010] The pathology diagnosis model in the discriminator is transferred to the pseudo-pathology diagnosis task, and the model parameters are optimized according to the pathology diagnosis results to obtain a trained pseudo-pathology diagnosis model.
[0011] As a further implementation method, the acquired data is also preprocessed.
[0012] As a further implementation method, the CT image is preprocessed, specifically:
[0013] The CT images were resampled to unify the voxel sizes of all patients to the same size, a CT tumor bounding box was generated based on the tumor area annotated by the experts, and the CT image data of the lung window within the bounding box was cropped.
[0014] As a further implementation method, the HE pathological image is preprocessed, specifically:
[0015] The tumor area marked by the expert was evenly cropped into several 256×256 pathology small images, and then each 256×256 pathology small image was cropped into 16 64×64 pathology small images. The Vahadane method was used to perform color correction on the pathology small images, and the colors of all pathology small images were unified into the same color space.
[0016] As a further implementation method, the authenticity of the first pseudo-pathological image is determined by using a real 64×64 pathological thumbnail;
[0017] The authenticity of the second pseudo-pathological image is determined using a real 256×256 pathological thumbnail.
[0018] As a further implementation method, a pathological diagnosis model in the discriminator is constructed based on Swin Transformer. The pathological diagnosis model includes a pathological diagnosis model trained based on a 256×256 pathological small image and a pathological diagnosis model trained based on a 64×64 pathological small image, which are respectively used to discriminate the authenticity of the second pseudo pathological image and the first pseudo pathological image.
[0019] As a further implementation method, the pathology diagnosis model in the discriminator trained based on 256×256 pathology images is migrated to the pseudo-pathology diagnosis task.
[0020] A second aspect of the present disclosure provides a system for generating HE pathological images from CT images based on a generative adversarial network, comprising:
[0021] The data acquisition module is configured to: acquire the patient's CT image, HE pathological image and pathological diagnosis result, perform feature extraction on the CT image, and obtain CT image features;
[0022] The first pseudo pathological image generating module is configured to: generate a first pseudo pathological image using CT image features, determine the authenticity of the first pseudo pathological image, and obtain a first loss;
[0023] The second pseudo pathological image generation module is configured to: use the first loss as a generation condition and the first pseudo pathological image as input to generate a second pseudo pathological image, determine the authenticity of the second pseudo pathological image, and obtain a second loss;
[0024] The model training module is configured to: migrate the pathology diagnosis model in the discriminator to the pseudo-pathology diagnosis task, and optimize the model parameters according to the pathology diagnosis results to obtain a trained pseudo-pathology diagnosis model.
[0025] A third aspect of the present disclosure provides a medium having a program stored thereon, which, when executed by a processor, implements the steps in the method for generating HE pathology images from CT images based on a generative adversarial network as described in the first aspect of the present disclosure.
[0026] A fourth aspect of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the method for generating HE pathological images from CT images based on a generative adversarial network described in the first aspect of the present disclosure are implemented.
[0027] Compared with the prior art, the present invention has the following beneficial effects:
[0028] The method and system for generating HE pathological images from CT images based on a generative adversarial network proposed in the present invention aims to solve the problem that the accurate histopathological differential diagnosis of SCLC and NSCLC by CT images is highly dependent on the clinical experience of the observer, resulting in inaccurate diagnostic results. The method and system innovatively propose a pathological generation strategy, explore the mapping relationship between the macroscopic structure of lung cancer CT and the microscopic structure of tissue pathology, convert pre-biopsy and preoperative CT images into pseudo-pathological images of tumors based on a generative adversarial network, construct a pseudo-pathological diagnosis model by combining real pathological images and transfer learning, improve the accuracy of preoperative diagnosis of histological subtypes of lung cancer, and realize non-invasive histological subtype assessment, thereby guiding the selection of personalized treatment methods for patients.
[0029] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.
[0031] Figure 1 is a data processing flow chart of a method for generating HE pathology images from CT images based on a generative adversarial network disclosed in the present invention;
[0032] Figure 2 It is a framework flow chart of the method for generating HE pathology images from CT images based on a generative adversarial network disclosed in the present invention. DETAILED DESCRIPTION
[0033] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.
[0034] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.
[0035] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.
[0036] Embodiment 1
[0037] like Figure 1-2 As shown, the first embodiment of the present disclosure provides a method for generating HE pathological images from CT images based on a generative adversarial network, which mainly includes the following steps:
[0038] S1 obtains the patient's CT image, HE pathological image and pathological diagnosis result, extracts features from the CT image, and obtains CT image features;
[0039] S2 generates a first pseudo pathological image using the CT image feature, determines the authenticity of the first pseudo pathological image, and obtains a first loss;
[0040] S3 uses the first loss as a generation condition and the first pseudo pathological image as input to generate a second pseudo pathological image, and determines the authenticity of the second pseudo pathological image to obtain a second loss;
[0041] S4 transfers the pathological diagnosis model in the discriminator to the pseudo-pathological diagnosis task, and optimizes the model parameters according to the pathological diagnosis results to obtain a trained pseudo-pathological diagnosis model.
[0042] This paper collects CT and pathological data of lung cancer patients, and cuts the tumor area outlined by pathologists on the full pathological slice into several 256×256 pathological small pictures as the generation target. Then, a three-dimensional convolutional neural network (3D CNN) is used to extract the deep features within the tumor bounding box on CT. Then, the CT deep features are used as the generation conditions of the conditional generative adversarial network (cGAN), which gradually generates 256×256 pseudo pathological small pictures through a two-stage generation method, and identifies the generated images by image similarity and pathological diagnosis accuracy. Then the pathological diagnosis performance of the real tissue pathology image is transferred to the pathological diagnosis model constructed based on the pseudo tissue pathology image generated by CT. Finally, the effectiveness of the pathology generation strategy is verified by comparing the diagnostic performance with the CT model and the tissue pathology model. The specific process is as follows:
[0043] Baseline CT images, hematoxylin-eosin staining (HE) stained tissue pathology images, and pathological diagnosis results of 1000 adult lung cancer patients in two hospitals were collected. The pathological diagnosis results are the gold standard for distinguishing lung cancer subtypes. The subtypes of lung cancer patients to be included are adenocarcinoma, squamous cell carcinoma, and small cell carcinoma (SCLC). The tumor area on CT was annotated by imaging experts, and the tumor area on the pathological full-slice image was annotated by pathological experts. The CT tumor bounding box was generated based on the annotated tumor area on CT, and the tumor area on the pathological full-slice was evenly cropped into several 256×256 pathological small images. Among all patients, the data of the first hospital were divided into training set, test set, and internal validation set in a ratio of 4:1:1, and the data of the other hospital were used as the external validation set.
[0044] like Figure 1 As shown, after acquiring the data, before generating the pseudo pathological image, the acquired data needs to be preprocessed.
[0045] The CT images are preprocessed as follows:
[0046] The CT images were resampled to unify the voxel sizes of all patients to the same size, a CT tumor bounding box was generated based on the tumor area annotated by the experts, and the CT image data of the lung window within the bounding box was cropped.
[0047] The HE pathological images are preprocessed as follows:
[0048] The tumor area annotated by the expert is evenly cropped into several 256×256 pathology images, and then each 256×256 pathology image is cropped into 16 64×64 pathology images to adapt to the subsequent two-stage cGAN generation process. Next, the Vahadane method is used to perform color correction on the pathology images, unifying the colors of all pathology images into the same color space to eliminate color differences between pathology images caused by factors such as staining time and staining equipment.
[0049] Construction and verification of pathological diagnosis model based on Swin Transformer.
[0050] Based on real HE pathology image data, a pathology diagnosis model is constructed to improve the efficiency of pathology diagnosis and supervise the generation of subsequent pseudo pathology images. First, the patient-level pathology diagnosis results are assigned to each pathology thumbnail cropped from it. Then, the Swin Transformer network is used to construct two pathology diagnosis models: one is based on 64×64 pathology thumbnails, and the other is based on 256×256 pathology thumbnails.
[0051] The pathology diagnosis model is trained and used as part of the discriminator in the generative adversarial task to determine the authenticity of pseudo-pathology images. After the generative adversarial model is trained, it is used to generate 256×256 pseudo-pathology images. The pathology diagnosis model based on the 256×256 pathology images is then migrated to the pseudo-pathology image diagnosis task.
[0052] Swin Transformer uses a hierarchical structure to provide feature information at all scales. It uses windows to limit the calculation scope of attention, improves the efficiency of the network, and enables information interaction between adjacent windows through sliding window operations. Swin Transformer can run on images of multiple resolutions and is suitable for the construction of models of different resolutions in this disclosure.
[0053] Construction and verification of CT pathology thumbnail generation model based on two-stage cGAN.
[0054] like Figure 2 As shown, the present invention gradually transforms CT features into pathological images by constructing a two-stage cGAN model. The model structure is as follows Figure 1 cGAN can generate conditional images, but the image accuracy is low. In order to improve the resolution of the generated images, this paper divides the generation task into two stages. First, a 64×64 low-precision pathology image is generated, and then a 256×256 high-precision image is further generated based on the generated results.
[0055] In the first stage, the CT depth features are used as generation conditions and input into the generator together with random noise to perform an upsampling operation, that is, a gradually larger convolutional network, to generate a 64×64 pseudo pathology thumbnail (i.e., the first pseudo pathology image). The 64×64 pseudo pathology thumbnail and the real 64×64 pathology thumbnail are placed in the discriminator for downsampling, that is, a gradually smaller convolutional network, to determine whether the pseudo pathology image is real. At the same time, the 64×64 pathology diagnosis model outputs the diagnosis result of the pseudo pathology thumbnail. The total loss L of conditional generation 1 for:
[0056]
[0057] in, D is the discriminator, G is the generator, x is a 64×64 pseudo pathology image, z is a noise image, y is the pathology diagnosis result, c is the CT feature, is the distribution of 64×64 pseudo pathology images, is the distribution of the noise image, is the expected value of the distribution function.
[0058] In the second stage, the generation loss L of the first stage is 1 As a condition, the 64×64 pseudo pathology image generated in the first stage is used as input and put into the generator together. The upsampling operation is also performed, and the pseudo pathology image of 256×256 is gradually enlarged to generate the second pseudo pathology image. It is put into the discriminator together with the real 256×256 pathology image for downsampling to determine whether the 256×256 pseudo pathology image is real. At the same time, the 256×256 pathology diagnosis model outputs the diagnosis result of the 256×256 pseudo pathology image. The total loss L of conditional generation 2 for:
[0059]
[0060] Among them, x is a 256×256 pseudo-pathology image, z is a 64×64 pseudo-pathology image, y is the pathological diagnosis result, and L 1 is the first stage generation loss, is the distribution of 256×256 pseudo-pathology images. is the distribution of 64×64 pseudo pathology images, is the expected value of the distribution function.
[0061] Construction and verification of pseudo-pathology diagnosis model based on pathology generation strategy.
[0062] Based on the two-stage cGAN model, several 256×256 pseudo pathological images are generated for each CT scan, and the pathological diagnosis model trained based on the 256×256 real pathological images is transferred to the pseudo pathological image diagnosis task using the transfer learning method.
[0063] The pathological diagnosis results of each patient are assigned to each 256×256 pseudo pathological image generated based on CT, and a pseudo pathological diagnosis model is constructed based on the Swin Transformer network. The parameters of the 256×256 real pathological image model are transferred to the pseudo pathological diagnosis model, and the model parameters are further optimized through training with 256×256 pseudo pathological images and pathological diagnosis results to improve the identification effect of SCLC and NSCLC. On the one hand, this transfer learning strategy can make the model suitable for pseudo pathological images, and on the other hand, the pathological images generated by cGAN can effectively expand the pathological dataset, further optimize the model parameters, and improve the diagnostic performance of the model.
[0064] Finally, the effectiveness of the pathology generation strategy was verified by comparing its performance with the CT diagnosis model and the real pathology diagnosis model.
[0065] Embodiment 2
[0066] Embodiment 2 of the present disclosure provides a system for generating HE pathological images from CT images based on a generative adversarial network, including:
[0067] The data acquisition module is configured to: acquire the patient's CT image, HE pathological image and pathological diagnosis result, perform feature extraction on the CT image, and obtain CT image features;
[0068] The first pseudo pathological image generating module is configured to: generate a first pseudo pathological image using CT image features, determine the authenticity of the first pseudo pathological image, and obtain a first loss;
[0069] The second pseudo pathological image generation module is configured to: use the first loss as a generation condition and the first pseudo pathological image as input to generate a second pseudo pathological image, determine the authenticity of the second pseudo pathological image, and obtain a second loss;
[0070] The model training module is configured to: migrate the pathology diagnosis model in the discriminator to the pseudo-pathology diagnosis task, and optimize the model parameters according to the pathology diagnosis results to obtain a trained pseudo-pathology diagnosis model.
[0071] The more detailed steps are the same as those in the first embodiment and will not be repeated here.
[0072] Embodiment 3
[0073] Embodiment 3 of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps in the method for generating HE pathological images from CT images based on a generative adversarial network as described in Embodiment 1 of the present disclosure are implemented.
[0074] The more detailed steps are the same as those in the first embodiment and will not be repeated here.
[0075] Embodiment 4
[0076] Embodiment 4 of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for generating HE pathological images from CT images based on a generative adversarial network as described in Embodiment 1 of the present disclosure are implemented.
[0077] The more detailed steps are the same as those in the first embodiment and will not be repeated here.
[0078] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for generating HE-stained pathological images from CT images based on generative adversarial networks. It is characterized in that The following steps are involved: Obtain the patient's CT image, HE-stained pathological image, and pathological diagnosis results, perform feature extraction on the CT image, and obtain CT image features; Generate a first pseudo pathological image using the CT image feature, determine the authenticity of the first pseudo pathological image, and obtain a first loss; The first loss is used as a generation condition, the first pseudo pathological image is used as input, a second pseudo pathological image is generated, and the authenticity of the second pseudo pathological image is judged to obtain a second loss; The pathology diagnosis model in the discriminator is transferred to the pseudo-pathology diagnosis task, and the model parameters are optimized according to the pathology diagnosis results to obtain a trained pseudo-pathology diagnosis model. Specifically, the pathological diagnosis results of each patient are assigned to each 256×256 pseudo pathological image generated based on CT, a pseudo pathological diagnosis model is constructed based on the Swin Transformer network, the parameters of the 256×256 real pathological image model are transferred to the pseudo pathological diagnosis model, and the model parameters are further optimized through training with 256×256 pseudo pathological images and pathological diagnosis results; The pathological diagnosis model is constructed based on real HE-stained pathological image data. First, the patient-level pathological diagnosis results are assigned to each pathological thumbnail cropped out, and then the pathological diagnosis model in the discriminator is constructed based on Swin Transformer. The pathological diagnosis model includes a pathological diagnosis model trained based on a 256×256 pathological thumbnail and a pathological diagnosis model trained based on a 64×64 pathological thumbnail, which are respectively used to discriminate the authenticity of the second pseudo-pathological image and the first pseudo-pathological image, and output the diagnosis results of the second pseudo-pathological image and the first pseudo-pathological image, respectively.
2. The method for generating HE-stained pathological images from CT images based on a generative adversarial network as claimed in claim 1, It is characterized in that It also includes preprocessing of the acquired data.
3. The method for generating HE-stained pathological images from CT images based on a generative adversarial network as claimed in claim 2, It is characterized in that The CT images are preprocessed as follows: The CT images were resampled to unify the voxel sizes of all patients to the same size, a CT tumor bounding box was generated based on the tumor area annotated by the experts, and the CT image data of the lung window within the bounding box was cropped.
4. The method for generating HE-stained pathological images from CT images based on a generative adversarial network as claimed in claim 2, It is characterized in that The HE stained pathological images were preprocessed as follows: The tumor area marked by the expert was evenly cropped into several 256×256 pathology small images, and then each 256×256 pathology small image was cropped into 16 64×64 pathology small images. The Vahadane method was used to perform color correction on the pathology small images, and the colors of all pathology small images were unified into the same color space.
5. The method for generating HE-stained pathological images from CT images based on a generative adversarial network as claimed in claim 4, It is characterized in that The authenticity of the first pseudo-pathological image is determined by using a real 64×64 pathological thumbnail; The authenticity of the second pseudo-pathological image is determined using a real 256×256 pathological thumbnail.
6. A system for generating HE-stained pathological images from CT images based on generative adversarial networks. It is characterized in that include: The data acquisition module is configured to: acquire the patient's CT image, HE-stained pathological image, and pathological diagnosis result, perform feature extraction on the CT image, and obtain CT image features; The first pseudo pathological image generating module is configured to: generate a first pseudo pathological image using CT image features, determine the authenticity of the first pseudo pathological image, and obtain a first loss; The second pseudo pathological image generation module is configured to: use the first loss and the first pseudo pathological image as generation conditions, generate a second pseudo pathological image, determine the authenticity of the second pseudo pathological image, and obtain a second loss; The model training module is configured to: transfer the pathology diagnosis model in the discriminator to the pseudo-pathology diagnosis task, and optimize the model parameters according to the pathology diagnosis results to obtain a trained pseudo-pathology diagnosis model; The pathology diagnosis model in the discriminator is transferred to the pseudo-pathology diagnosis task, and the model parameters are optimized according to the pathology diagnosis results to obtain a trained pseudo-pathology diagnosis model. Specifically, the pathological diagnosis results of each patient are assigned to each 256×256 pseudo pathological image generated based on CT, a pseudo pathological diagnosis model is constructed based on the Swin Transformer network, the parameters of the 256×256 real pathological image model are transferred to the pseudo pathological diagnosis model, and the model parameters are further optimized through training with 256×256 pseudo pathological images and pathological diagnosis results; The pathological diagnosis model is constructed based on real HE-stained pathological image data. First, the patient-level pathological diagnosis results are assigned to each pathological thumbnail cropped out, and then the pathological diagnosis model in the discriminator is constructed based on Swin Transformer. The pathological diagnosis model includes a pathological diagnosis model trained based on a 256×256 pathological thumbnail and a pathological diagnosis model trained based on a 64×64 pathological thumbnail, which are respectively used to discriminate the authenticity of the second pseudo-pathological image and the first pseudo-pathological image, and output the diagnosis results of the second pseudo-pathological image and the first pseudo-pathological image, respectively.
7. A medium having a program stored thereon, It is characterized in that When the program is executed by a processor, the steps in the method for generating HE-stained pathological images from CT images based on a generative adversarial network as described in any one of claims 1 to 5 are implemented.
8. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the steps in the method for generating HE-stained pathological images from CT images based on a generative adversarial network as described in any one of claims 1 to 5 are implemented.