Virtual IHC dyeing model based on mask-guided adversarial transfer learning
By adopting an adversarial transfer learning model based on mask guidance in image analysis technology, the problem of poor generation of IHC stained sections in the prior art is solved, especially in the details of the positive reaction areas, and higher generation details and diagnostic accuracy are achieved.
Patent Information
- Application Number
- CN202411768912.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-04
- Publication Date
- 2025-04-11
AI Technical Summary
The prior art is difficult to effectively generate realistic IHC stained sections, especially in the details of the positive reaction areas, which affects the accuracy of disease diagnosis.
Using an adversarial transfer learning model based on mask guidance, the H&E and IHC stained sections are featured and generated through the H&E generator and the IHC generator respectively. Combined with the discriminator and the optimizer, the virtual slice mask is extracted and the pixel loss is calculated, and the model parameters are updated to improve the generation details of the positive reaction area.
The detailed effect of generating positive reaction areas in IHC stained sections is significantly improved, and the model's realization ability of important diagnostic areas is enhanced, thereby improving the accuracy of disease diagnosis.
Smart Images
Figure CN119810225A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image analysis, and in particular to a virtual IHC staining model based on mask-guided adversarial transfer learning. Background Art
[0002] Immunohistochemistry (IHC) staining helps to qualitatively and quantitatively analyze the performance of specific markers in staining, providing key insights into prognosis and guiding immunotherapy strategies for patients. However, compared with the commonly used H&E staining, IHC staining is more expensive, time-consuming, and requires additional tissue samples and chemical reagents. To address these challenges, synthesizing virtual IHC staining from H&E staining has become a promising approach. However, traditional generation methods often fail to produce realistic IHC staining sections due to the lack of annotation of immunohistochemically positive reaction areas.
[0003] At present, virtual IHC staining of pathological images based on deep learning usually trains an adversarial transfer learning model or a diffusion generation model to output new IHC staining slices, and iteratively updates the model parameters by calculating the similarity with slices with the same staining method. However, most of the methods improve the generation effect by modifying the model structure and adjusting the data input, but ignore the improvement of the generation effect of the positive reaction area, which is crucial for the diagnosis of related diseases. Poor generation effect will affect the subsequent judgment results. Summary of the invention
[0004] In view of the shortcomings of the prior art, the present invention proposes a virtual IHC staining model based on mask-guided adversarial transfer learning, which can improve the generation of details in the positive reaction area. The specific technical solution is as follows:
[0005] A virtual IHC staining model based on mask-guided adversarial transfer learning is provided, which in a first achievable manner includes:
[0006] An H&E generator is configured to perform feature extraction on the H&E stained slices and output corresponding stained slices, and perform feature extraction on the stained slices output by the IHC generator to obtain corresponding virtual stained slices;
[0007] An IHC generator is configured to perform feature extraction on the IHC stained slices and output corresponding stained slices, and perform feature extraction on the stained slices output by the H&E generator to obtain corresponding virtual stained slices;
[0008] A discriminator group is configured to compare the virtual stained sections obtained by the H&E generator and the IHC generator with the real H&E stained sections and the IHC stained sections, respectively, and output corresponding discrimination results;
[0009] The optimizer is configured to extract a virtual slice mask of the virtual stained slice, calculate the pixel loss between the virtual slice mask and the actual slice mask in combination with the discrimination result, and update the model parameters according to the pixel loss.
[0010] In combination with the first possible implementation, in a second possible implementation, the H&E generator and the IHC generator have the same structure and both include:
[0011] The residual module is configured to perform feature extraction on the stained slices and obtain a corresponding preliminary feature map;
[0012] A multi-head attention layer configured to fuse context information of the preliminary feature map to obtain a feature map with contextual connection;
[0013] A feed-forward neural network layer is configured to map the feature map to a corresponding stained slice.
[0014] In combination with the first implementable manner, in a third implementable manner, the discriminator group includes two discriminators of the same structure, and the two discriminators are used to calculate the differences between the virtual stained sections obtained by the H&E generator and the IHC generator and the real H&E stained sections and IHC stained sections, respectively, to obtain corresponding discrimination results.
[0015] In combination with the first implementable manner, in a fourth implementable manner, the optimizer includes a mask loss layer configured to extract a virtual slice mask of the virtual stained slice and calculate a mask loss between the virtual slice mask and an actual slice mask.
[0016] In combination with the fourth implementable manner, in a fifth implementable manner, the mask loss layer uses an unsupervised mask extraction method based on a multi-threshold Otsu method to extract the virtual slice mask.
[0017] In combination with the fifth implementable manner, in a sixth implementable manner, the unsupervised mask extraction method based on the multi-threshold Otsu method includes:
[0018] Cutting the virtual stained slice to obtain a virtual slice image of a suitable size;
[0019] The virtual slice image is divided into three color spaces: red, green and blue, and a virtual slice image that effectively displays cell nuclei and blood vessels is selected therefrom;
[0020] All virtual slice images were analyzed using the multi-threshold Otsu method, and multiple appropriate thresholds were selected and fused to determine the mask threshold that retained only the cell nucleus and the mask threshold that retained the cell nucleus and blood vessels;
[0021] extracting a mask containing only cell nuclei and a mask containing cell nuclei and blood vessels from the virtual slice image based on corresponding mask thresholds;
[0022] Obtaining a mask containing only blood vessels by using an intersection method according to the mask containing only cell nuclei and the mask containing cell nuclei and blood vessels;
[0023] A mask containing only blood vessels and a mask containing only cell nuclei are used as the virtual slice masks.
[0024] In combination with the sixth implementable method, in the seventh implementable method, an intersection method is used to obtain a mask containing only blood vessels, including:
[0025] The obtained mask is denoised and hole filled.
[0026] In combination with the first implementable manner, in the eighth implementable manner, an unsupervised mask extraction method based on the multi-threshold Otsu method is used to extract the mask of the real stained slice to obtain the corresponding actual slice mask.
[0027] Beneficial effects: By using the virtual IHC staining model based on mask-guided adversarial transfer learning of the present invention, the H&E generator and the IHC generator can extract features of H&E stained sections and IHC stained sections respectively to generate corresponding stained sections. Then, the corresponding stained sections are input into the IHC generator and the H&E generator respectively to obtain corresponding virtual stained sections.
[0028] The discriminator group can calculate the difference in feature maps between the generated stained slices and the real stained slices. The optimizer can extract the corresponding cell nucleus and blood vessel masks from the virtual stained slices, and calculate the pixel loss between the generated virtual slice mask and the actual slice mask based on the discriminator group's discrimination results. The model parameters are optimized and adjusted according to the pixel loss. The resulting staining model can improve the generation of details in the positive reaction area. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the specific implementation of the present invention, the following will briefly introduce the drawings required for use in the specific implementation. In all the drawings, each element or part is not necessarily drawn according to the actual scale.
[0030] Figure 1 A structural diagram of a virtual IHC staining model based on mask-guided adversarial transfer learning provided by one embodiment of the present invention;
[0031] Figure 2 A flowchart of an unsupervised mask extraction method based on the multi-threshold Otsu method provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0032] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.
[0033] It should be understood that H&E staining is the most widely used staining technique in tissue cases. By combining the two dyes of hematoxylin and eosin, a wide range of cytoplasm, nucleus and extracellular matrix characteristics can be obtained. IHC staining uses the principle of specific binding between antigens and antibodies. Through chemical reactions, the color developer such as fluorescein, enzyme, metal ion, isotope, etc. labeled with antibodies can be colored to determine the antigens in tissue cells, such as peptides and proteins.
[0034] like Figure 1 The structure diagram of the virtual IHC staining model based on mask-guided adversarial transfer learning is shown, and the staining model includes:
[0035] An H&E generator is configured to perform feature extraction on the H&E stained slices and output corresponding stained slices, and perform feature extraction on the stained slices output by the IHC generator to obtain corresponding virtual stained slices;
[0036] An IHC generator is configured to perform feature extraction on the IHC stained slices and output corresponding stained slices, and perform feature extraction on the stained slices output by the H&E generator to obtain corresponding virtual stained slices;
[0037] A discriminator group is configured to compare the virtual stained sections obtained by the H&E generator and the IHC generator with the real H&E stained sections and the IHC stained sections, respectively, and output corresponding discrimination results;
[0038] The optimizer is configured to extract a virtual slice mask of the virtual stained slice, calculate the pixel loss between the virtual slice mask and the actual slice mask in combination with the discrimination result, and update the model parameters according to the pixel loss.
[0039] Specifically, the staining model includes a generator group, a discriminator group and an optimizer. The generator group includes an H&E generator and an IHC generator. Among them, the H&E generator can extract features from the received H&E stained slices and generate virtual H&E stained slices based on the extracted feature map. The IHC generator can extract features from the virtual H&E stained slices and generate corresponding IHC virtual stained slices based on the extracted feature map.
[0040] Similarly, the IHC generator can extract features from the received IHC stained slices and generate virtual IHC stained slices based on the extracted feature maps. The H&E generator can extract features from the virtual IHC stained slices and generate corresponding H&E virtual stained slices based on the extracted feature maps.
[0041] The discriminator group includes two discriminators, one for calculating the difference between the IHC virtual stained sections output by the IHC generator and the real IHC stained sections, and the other for calculating the difference between the H&E virtual stained sections generated by the model and the real H&E stained sections.
[0042] The optimizer can perform mask extraction on the H&E virtual stained slices and IHC virtual stained slices generated by the model to obtain the corresponding cell nucleus and vascular masks. The mask loss between the cell nucleus and vascular masks corresponding to the virtual stained slices and the cell nucleus and vascular masks corresponding to the real stained slices is calculated. Finally, the pixel loss between the virtual slice mask and the actual slice mask is calculated by combining the difference between the generated stained slices and the real stained slices calculated by the discriminator, and the model parameters are updated according to the pixel loss.
[0043] It should be understood that conventional adversarial transfer learning models and diffusion generation models cannot effectively control the effect of generated images. By introducing mask loss and discriminator loss, the model can be well guided to produce effective generation, and the details of the generated virtual images are more realistic, so as to improve the generation details of the positive reaction area.
[0044] In this embodiment, optionally, the H&E generator and the IHC generator have the same structure and both include:
[0045] The residual module is configured to perform feature extraction on the stained slices and obtain a corresponding preliminary feature map;
[0046] A multi-head attention layer configured to fuse context information of the preliminary feature map to obtain a feature map with contextual connection;
[0047] A feed-forward neural network layer is configured to map the feature map to a corresponding stained slice.
[0048] Specifically, the H&E generator and the IHC generator have the same structure, including a residual module, a multi-head attention layer, and a feedforward neural network layer. Among them, the residual module preliminarily extracts the features of the input slices and obtains a preliminary feature map of the slices to enhance the model depth and improve the model fitting ability. The multi-head attention layer can fuse the context information of the features extracted by the residual module to obtain a feature map with context information, thereby building its long-range dependencies to extract more powerful features. The feedforward neural network layer is used to map the feature map with context information to the corresponding stained slices.
[0049] Taking the H&E generator as an example, the residual module in the H&E generator extracts features from H&E stained slices to obtain preliminary feature maps of H&E stained slices. The multi-head attention layer performs linear transformation, segmentation, and scaling point attention calculation on the preliminary feature maps output by the residual module, and finally outputs feature maps with context information. The feedforward neural network layer maps the feature maps output by the multi-head attention layer to virtual H&E stained slices through linear transformation and nonlinear activation functions such as ReLU and sigmoid functions.
[0050] Similarly, the residual module in the H&E generator can perform feature extraction on the virtual IHC stained slices generated by the IHC generator to obtain the preliminary feature map corresponding to the virtual IHC stained slices, and then obtain the H&E virtual stained slices through the multi-head attention layer and the feedforward neural network layer.
[0051] In this embodiment, optionally, the discriminator group includes two discriminators with the same structure, and the two discriminators are used to calculate the difference between the virtual stained sections obtained by the H&E generator and the IHC generator and the real H&E stained sections and IHC stained sections, respectively, to obtain corresponding discrimination results.
[0052] Specifically, the discriminator group includes two discriminators with the same structure, and the two discriminators are used to calculate the difference between the stained sections generated by the H&E generator and the IHC generator and the real stained sections. The discriminator includes a residual module and a feedforward neural network layer. Among them, the residual module is used to extract the features of the stained sections generated by the generator and the real stained sections. The feedforward neural network layer is used to calculate the features of the generated stained sections and the difference between the features of the real stained sections to obtain the discriminant results for the generated stained sections.
[0053] Take calculating the difference between the stained slices generated by the H&E generator and the real H&E stained slices as an example. The residual module in the discriminator can extract the features of the stained slices generated by the H&E generator and the features of the real H&E stained slices obtained after the H&E stained slices are stained. The feedforward neural network can calculate the difference between the features of the generated stained slices extracted by the residual module and the features of the real H&E stained slices, thereby obtaining the model's discrimination result corresponding to the H&E stained slices.
[0054] In this embodiment, optionally, the optimizer includes a mask loss layer, which is configured to extract a virtual slice mask of the virtual stained slice and calculate a mask loss between the virtual slice mask and the actual slice mask.
[0055] Specifically, the optimizer includes a mask loss layer, which is used to extract the corresponding virtual slice mask from the H&E virtual stained slice and the IHC virtual stained slice generated by the generator group, and calculate the mask loss between the extracted virtual slice mask and the actual slice mask corresponding to the real stained slice. The optimizer adds the mask loss to the discrimination result calculated by the discriminator to obtain the pixel loss between the virtual slice mask generated by the model and the actual slice mask, thereby updating the model model and improving the generation detail effect of the positive reaction area.
[0056] In this embodiment, optionally, the mask loss layer may use an unsupervised mask extraction method based on a multi-threshold Otsu method to extract the virtual slice mask.
[0057] In this embodiment, optionally, an unsupervised mask extraction method based on the multi-threshold Otsu method is used to extract the mask of the real stained slice to obtain the corresponding actual slice mask.
[0058] Specifically, it should be understood that most existing staining models use a supervised approach to train the models, and data annotation consumes a lot of time and labor costs. In addition, IHC staining is costly and time-consuming, requires additional tissue samples and chemical reagents, and it is difficult to collect more trainable data.
[0059] To this end, when constructing the training data for the model, an unsupervised mask extraction method based on the multi-threshold Otsu method can be used to directly extract the corresponding actual slice mask from the real stained slice. Then, the training data is constructed based on the extracted slices, the real stained slices, and the actual slice masks, without the need for data annotation, thereby reducing training time and labor costs, and rich training data can be constructed.
[0060] Specifically, first, multiple copies of real IHC stained sections and H&E stained sections obtained after IHC staining and H&E staining can be obtained. Then, an unsupervised mask extraction method based on the multi-threshold Otsu method is used to extract masks from real IHC stained sections and H&E stained sections to obtain IHC actual section masks and H&E actual section masks corresponding to each section. Finally, training samples are constructed based on real IHC stained sections and H&E stained sections, as well as IHC actual section masks and H&E actual section masks, and a training set is constructed based on the training samples corresponding to all sections.
[0061] In this embodiment, optionally, Figure 2 As shown, the unsupervised mask extraction method based on the multi-threshold Otsu method includes:
[0062] Step 1: cropping the virtual stained slice to obtain a virtual slice image of a suitable size;
[0063] Step 2, dividing the virtual slice image into three color spaces of red, green and blue, and selecting a virtual slice image that effectively displays the cell nucleus and blood vessels;
[0064] Step 3, using the multi-threshold Otsu method to analyze all virtual slice images, select multiple appropriate thresholds to fuse and determine the mask threshold that retains only the cell nucleus, and the mask threshold that retains the cell nucleus and blood vessels;
[0065] Step 4: extracting a mask containing only the cell nucleus and a mask containing the cell nucleus and blood vessels from the virtual slice image based on a corresponding mask threshold;
[0066] Step 5: Obtain a mask containing only blood vessels by using an intersection method according to the mask containing only cell nuclei and the mask containing cell nuclei and blood vessels;
[0067] Step 6: Using the mask containing only blood vessels and the mask containing only cell nuclei as the virtual slice mask.
[0068] Specifically, first, the virtual stained slices generated by the generator can be cropped using a cropping tool to obtain a virtual slice image of a suitable size containing pathological information. Then, the virtual slice image can be divided into a red, green and blue color space, and a virtual slice image that can effectively display the cell nucleus and blood vessels can be selected. After that, the existing multi-threshold law is used to analyze all selected virtual slice images, and multiple appropriate thresholds are selected to fuse to determine the mask threshold corresponding to the mask containing only the cell nucleus, and the mask threshold corresponding to the mask containing the cell nucleus and blood vessels. Then, based on these two mask thresholds, a mask containing only the cell nucleus and a mask containing the cell nucleus and blood vessels are extracted from the virtual slice image. After that, the mask containing only the cell nucleus is merged with the mask containing the cell nucleus and blood vessels by the intersection method to obtain a mask containing only blood vessels. Finally, the mask containing only the cell nucleus and the mask containing only the blood vessels are used as virtual slice masks for subsequent mask loss calculations.
[0069] In this embodiment, optionally, a mask containing only blood vessels is obtained by using an intersection method, including:
[0070] The obtained mask is denoised and hole filled.
[0071] Specifically, before using the intersection method to determine the mask containing only blood vessels, the extracted mask containing only cell nuclei and the mask containing cell nuclei and blood vessels can be denoised and hole-filled. Denoising can reduce noise interference with smaller pixel areas, and hole filling can reduce the holes inside the mask, thereby improving the accuracy of subsequent mask loss calculations and further improving the generation of details in the positive reaction area.
[0072] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.
Claims
1. A virtual IHC staining model based on mask-guided adversarial transfer learning, characterized in that: include: An H&E generator is configured to perform feature extraction on the H&E stained slices and output corresponding stained slices, and perform feature extraction on the stained slices output by the IHC generator to obtain corresponding virtual stained slices; An IHC generator is configured to perform feature extraction on the IHC stained slices and output corresponding stained slices, and perform feature extraction on the stained slices output by the H&E generator to obtain corresponding virtual stained slices; A discriminator group is configured to compare the virtual stained sections obtained by the H&E generator and the IHC generator with the real H&E stained sections and the IHC stained sections, respectively, and output corresponding discrimination results; The optimizer is configured to extract a virtual slice mask of the virtual stained slice, calculate the pixel loss between the virtual slice mask and the actual slice mask in combination with the discrimination result, and update the model parameters according to the pixel loss.
2. The virtual IHC staining model based on mask-guided adversarial transfer learning according to claim 1, characterized in that: The H&E generator and IHC generator have the same structure and both include: The residual module is configured to perform feature extraction on the stained slices and obtain a corresponding preliminary feature map; A multi-head attention layer configured to fuse context information of the preliminary feature map to obtain a feature map with contextual connection; A feed-forward neural network layer is configured to map the feature map to a corresponding stained slice.
3. The virtual IHC staining model based on mask-guided adversarial transfer learning according to claim 1, characterized in that: The discriminator group includes two discriminators with the same structure, and the two discriminators are used to calculate the difference between the virtual stained sections obtained by the H&E generator and the IHC generator and the real H&E stained sections and the IHC stained sections, respectively, to obtain corresponding discrimination results.
4. The virtual IHC staining model based on mask-guided adversarial transfer learning according to claim 1, characterized in that: The optimizer includes a mask loss layer configured to extract a virtual slice mask of the virtual stained slice and calculate a mask loss between the virtual slice mask and an actual slice mask.
5. The virtual IHC staining model based on mask-guided adversarial transfer learning according to claim 4, characterized in that: The mask loss layer uses an unsupervised mask extraction method based on a multi-threshold Otsu method to extract the virtual slice mask.
6. The virtual IHC staining model based on mask-guided adversarial transfer learning according to claim 5, characterized in that: The unsupervised mask extraction method based on the multi-threshold Otsu method comprises: Cutting the virtual stained slice to obtain a virtual slice image of a suitable size; The virtual slice image is divided into three color spaces: red, green and blue, and a virtual slice image that effectively displays cell nuclei and blood vessels is selected therefrom; All virtual slice images were analyzed using the multi-threshold Otsu method, and multiple appropriate thresholds were selected and fused to determine the mask threshold that retained only the cell nucleus and the mask threshold that retained the cell nucleus and blood vessels; extracting a mask containing only cell nuclei and a mask containing cell nuclei and blood vessels from the virtual slice image based on corresponding mask thresholds; Obtaining a mask containing only blood vessels by using an intersection method according to the mask containing only cell nuclei and the mask containing cell nuclei and blood vessels; A mask containing only blood vessels and a mask containing only cell nuclei are used as the virtual slice masks.
7. The virtual IHC staining model based on mask-guided adversarial transfer learning according to claim 6, characterized in that: The mask containing only blood vessels is obtained by intersection, including: The obtained mask is denoised and hole filled.
8. The virtual IHC staining model based on mask-guided adversarial transfer learning according to claim 1, characterized in that: An unsupervised mask extraction method based on multi-threshold Otsu method is used to extract the mask of the real stained slice and obtain the corresponding actual slice mask.