Medical image cross-dyeing domain conversion method based on multi-modal feature fusion
By using a diffusion model that integrates multimodal features, the problems of tissue morphology distortion and mode collapse in cross-staining domain conversion of medical images are solved, achieving high-quality and accurate mapping between H&E and IHC stained images, and meeting the applicability requirements of clinical whole slide images.
Patent Information
- Application Number
- CN202510945589.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-09
- Publication Date
- 2025-10-31
AI Technical Summary
Existing technologies for cross-staining domain conversion of medical images suffer from problems such as distortion of tissue morphology features, difficulty in modeling nonlinear domain differences in multimodal staining images, and mode collapse of generative models, resulting in poor quality of pathological images and difficulty in meeting the applicability of clinical whole slide image scenarios.
We employ a diffusion model based on multimodal feature fusion. By constructing a learnable multi-scale feature decoupling module and a pathological prior guidance mechanism, combined with a progressive training framework, we achieve cross-modal semantic alignment of H&E and IHC staining features. Furthermore, through a task-adaptive prior distribution learning strategy and a conditional diffusion model, we alleviate the information decay problem and generate high-quality cross-chromodomain medical images.
It exhibits significant advantages in structural fidelity and color consistency, with an average SSIM index of 0.7300, PSNR of 27.2638dB, and staining vector similarity (SVS) of 0.8050. It achieves accurate mapping of H&E and IHC staining spectral characteristics and solves the problems of cell boundary blurring and texture loss.
Smart Images
Figure CN120876641A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of staining standardization technology in medical images, and specifically relates to a method for cross-staining domain conversion of medical images based on multimodal feature fusion. Background Technology
[0002] In the field of digital pathology, staining standardization is a key technical challenge to ensure consistency in multicenter pathological diagnosis. Current mainstream medical image cross-staining domain conversion methods mainly face three core dilemmas: (1) the progressive distortion of tissue morphology features during staining conversion, leading to blurred cell nuclear boundaries and degraded cytoplasmic texture; (2) the difficulty in modeling nonlinear domain differences between multimodal staining images (such as H&E staining and IHC staining), and the tendency of traditional methods to produce spectral artifacts during visible light band mapping; (3) the mode collapse problem that is common in existing generative models during cross-modal conversion, which seriously restricts the applicability of the methods in clinical whole slide image (WSI) scenarios.
[0003] While cross-chromatographic domain transformation methods based on Generative Adversarial Networks (GANs) can achieve basic modality mapping, their inherent limitations are significantly amplified in medical image scenarios: the dynamic game between the generator and discriminator easily leads to irreversible loss of tissue microstructures (such as nucleocytoplasmic ratio and mitotic features); the vanishing gradient problem in adversarial training causes mode collapse, resulting in homogenization bias in key pathological features in the generated images. Recent studies have attempted to improve these methods by introducing CycleGAN or Attention-GAN mechanisms, but these methods still fall short in dealing with the complex feature entanglement in multimodal pathological images, especially in handling the spatial correspondence between immunohistochemical staining specific expressions and tissue structures, where it is difficult to maintain the topological consistency of subcellular features.
[0004] The unique advantages of diffusion models in generative quality control offer a new approach to cross-chromatographic transformation. Their progressive denoising mechanism theoretically better preserves the continuous evolution of histological features, while explicit probability density modeling helps mitigate mode collapse. However, directly applying the denoising diffusion probability model (DDPM) to medical image staining transformation faces two major obstacles: first, the strict spatial correspondence requirements of pathological images fundamentally conflict with the inherent random generation characteristics of diffusion models; second, the lack of pixel-level alignment ground truth constraints among multimodal staining data makes it difficult for traditional diffusion paradigms to establish reliable cross-modal mapping relationships. Summary of the Invention
[0005] To address the problems existing in the prior art, this invention proposes a medical image cross-chromatographic domain conversion method based on multimodal feature fusion. The method includes: acquiring a medical image of the target chromatographic domain and a source chromatographic domain image; preprocessing the medical image of the target chromatographic domain and the source chromatographic domain image; extracting rich texture detail features from the preprocessed source chromatographic domain image and rich color features from the preprocessed target chromatographic domain image; fusing the rich texture detail features and rich color features and using them as a prior distribution of fusion knowledge; performing a target search on the prior distribution of fusion knowledge to find a dataset prior distribution that meets the requirements of the fusion task, obtaining the optimal distribution solution; initializing the optimal distribution solution and performing forward diffusion to obtain a noisy fused image; and performing back diffusion on the noisy fused image to obtain the fused image.
[0006] The beneficial effects of this invention are:
[0007] Compared to traditional staining normalization methods (such as Macenko and Reinhard) and cross-domain transfer methods based on generative adversarial networks (such as Vahadane), this invention demonstrates significant advantages in structural fidelity and color consistency. Regarding structural fidelity, the average SSIM index reaches 0.7300, and the PSNR is 27.2638 dB, effectively solving the problems of blurred cell boundaries and lost nucleolar texture. In terms of color consistency, the staining vector similarity (SVS) reaches 0.8050, and the histogram distance (HD) is reduced to 0.0816, achieving accurate mapping between H&E and IHC staining spectral characteristics. Attached Figure Description
[0008] Figure 1 This is an overall flowchart of the present invention;
[0009] Figure 2 This is a comparison chart of the indicators of the present invention. Detailed Implementation
[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0011] This invention proposes an innovative dual-stream conditional diffusion fusion architecture, overcoming the technical bottleneck of traditional cross-chromodomain conversion. In the forward diffusion stage, this architecture constructs a learnable multi-scale feature decoupling module, achieving cross-modal semantic alignment of H&E and IHC staining features through dynamic weight allocation, effectively establishing biomarker correspondences between pathological images. In the inverse denoising stage, it innovatively introduces a pathological prior guidance mechanism, encoding cell morphological constraints such as nuclear membrane integrity and chromatin distribution patterns into implicit diffusion conditions, and achieving dynamic aggregation of multimodal features through gated attention units. Addressing the core challenge of ground truth loss in medical image fusion, this invention employs a task-adaptive prior distribution learning strategy, encoding multimodal complementary information into differentiable diffusion conditions through a hierarchical feature search network, and combining this with a progressive training framework to achieve collaborative optimization of "feature decoupling-condition generation".
[0012] In this embodiment, the present invention extracts "visual features" (including global visual attributes such as color space distribution) from the target staining domain (e.g., immunohistochemical staining, IHC) and "structural features" (e.g., anatomical information such as cell nuclear morphology and texture distribution) from the source staining domain (e.g., hematoxylin and eosin staining, H&E). Through a preliminary fusion strategy, a "universal fusion distribution" containing source domain structural information and target domain visual characteristics is constructed as a "knowledge prior" for subsequent generation tasks. This stage aims to achieve a balance between the fidelity of the anatomical structure of the source image and the transferability of the visual features of the target staining domain. The fusion knowledge prior distribution is used for target search to find a dataset prior distribution that meets the requirements of the fusion task, obtaining the optimal distribution solution. Guided by the fusion knowledge prior, an image degradation path is simulated through a "diffusion forward process": Gaussian noise is gradually added to the fused image, establishing a continuous degradation chain from a clear image to pure noise. The "marginal distribution" generated in this process characterizes the probability distribution characteristics of the image in the noise space. It is worth noting that during the diffusion process, the source domain structural information (such as cell boundaries) and target domain features (such as staining intensity) exhibit a gradual decoupling phenomenon, providing a dynamic basis for subsequent feature recombination. Based on the edge distribution, a "backsampling process" is performed. By introducing the "optimal distribution solution" as a constraint, the noisy image is gradually reconstructed into target staining domain features. Simultaneously, the original source staining domain image and the local region annotations of the target staining domain are accessed to construct a Conditional Diffusion Model (CDM). The Denoising Diffusion Module (DDM) receives the noisy fused image. Noise removal is achieved through a progressive multi-layered R-Block (Residual Block) approach. Cross-Scale Skip Connections between R-Blocks facilitate the synergistic optimization of low-level anatomical structures (such as cell nuclear contours) and high-level staining features (such as stainer distribution), effectively mitigating information decay. The MDRM module semantically enhances the DDM output by dynamically adjusting the fusion weights of source and target domain features through an attention mechanism, ensuring the generated image maintains medical semantic consistency with the target staining domain. A Multimodal Feature Transfer Strategy is employed to inject local features of the target staining domain (such as specific antigen expression patterns) into the global structure. An adaptive weight allocation mechanism dynamically balances the feature contributions of the source and target domains based on image content. The final output is a fused image. While preserving the integrity of the anatomical structure of H&E staining, it accurately reproduces the visual representation characteristics of IHC staining, meeting the clinical needs of cross-staining domain medical image generation.
[0013] Another implementation method is a medical image cross-chromosome transformation method based on multimodal feature fusion, such as... Figure 1 As shown, the method includes: acquiring a medical image of the target chromatid domain and a source chromatid domain image; preprocessing the medical image of the target chromatid domain and the source chromatid domain image; learning the rich texture details of the preprocessed source chromatid domain image and the rich colors of the preprocessed target chromatid domain image to form a general view representing the fusion strategy, which serves as a prior distribution of fusion knowledge; performing a target search on the prior distribution of fusion knowledge to find a prior distribution of the dataset that meets the requirements of the fusion task, obtaining the optimal distribution solution; initializing the optimal distribution solution and performing forward diffusion to obtain a noisy fused image; and performing back diffusion on the noisy fused image to obtain the fused image. The DDM module is a diffusion denoising module, the MDRM module is a multimodal diffusion optimization module, and the DIB module is a combination of these two modules. The bidirectional diffusion process is implemented through t DIB modules.
[0014] In this embodiment, the preprocessing of medical images includes: For the preprocessing process of H&E and IHC paired pathological images, this study adopts a systematic quality control scheme: First, a sliding window strategy is used to standardize and cut 6000 1024×1024 full-view digital pathological images (WSI) into 256×256 pixels. Then, pathological images with more blank areas are removed by manual screening. The paired images are divided into training set, test set and validation set.
[0015] In this image fusion task, given I H&E and IIHC Image, fused with prior knowledge including the target color of the image and retain rich detail textures The set of fused prior knowledge distributions can be concisely represented as:
[0016]
[0017] Fusion of prior knowledge distribution The establishment of such a system represents a common viewpoint in fusion strategies, generally yielding better visual information and quality. However, in existing fusion knowledge priors, the concentration of fused information varies depending on the task's objective. To alleviate conflicts between different objectives and improve orientation performance, target search can effectively address this issue within the existing fusion knowledge priors.
[0018] In this embodiment, the target search aims to find a dataset prior distribution that meets the requirements of the fusion task within the high-dimensional flow field of image fusion knowledge priors. In this search process, we define the target search function f... ts (i) Defined by an evaluation function, primarily considering similarity, visual fidelity, and the preservation of gradient texture information. From this perspective, the simplest general sample search is adopted, where SSIM(i) is used for similarity, VIF(i) for visual fidelity, and Q... AB / F (i) Used for texture information preservation:
[0019]
[0020] In the formula: ω ssim ω vif and ω QAB / F This is a hyperparameter. Specifically, the hyperparameter is set to ω. ssim =2, ω vif =1 and This is just one example of implementing the equation; in practical applications, it can be flexibly adjusted according to the needs of the fusion task.
[0021] Based on prior knowledge, the sample images in the fusion distribution are fused, and the search function is rewritten in the following format:
[0022]
[0023] In the formula: ω n Weights for different search evaluation metrics. T n (I j Let represent the evaluation function for a given distribution of the j-th sample. j These are sample images from a fused distribution.
[0024] In simple terms, the target search process F tsIt can be represented as:
[0025]
[0026] in, It is the optimal distribution solution obtained through target search in the fusion of prior knowledge; F ts For the target search process, For the preprocessed dataset, I j For sample images in the fusion distribution, f ts Let be the target search function. From a distributional perspective, it is a complex flow distribution. Therefore, this task will use a diffusion model to learn a fusion distribution for this target search.
[0027] In this embodiment, bidirectional diffusion of the image includes forward diffusion and backward diffusion; the Denoising Diffusion Probabilistic Model (DDPM) generates samples through a predefined forward Markov chain process and a progressively backward backward process. It is worth noting that the fusion knowledge prior only supports the forward diffusion model during training and does not participate in the backward fusion process.
[0028] During the forward pass, the given data distribution x0~q(x0) is incrementally and repeatedly supplemented with Gaussian noise, resulting in the following distribution q(x0~q(x0)). t |x t-1 ):
[0029]
[0030] α t =1-β t
[0031] Where, x t To give the previous step x t-1 The data distribution after adding Gaussian noise again Let α be the normal distribution function. t The proportion retained from the original data, where I is the identity matrix, and β t It is a variable that controls the noise variance added to the data in step t.
[0032] Based on the above distribution, the marginal distribution is obtained:
[0033]
[0034] in, The product of the proportions retained from the original data, where t is the point at each step of the forward process, and x0 is the distribution of the original data.
[0035] When the marginal distribution q(x) t |x0) approaches a standard normal distribution Right now At that time, the forward diffusion process is completed.
[0036] The forward diffusion process specifically includes:
[0037] First, input fixed parameters: denoising network {∈ θ}、Refine the network{n θ}、Source Image {I H&E}、Target staining domain image {I IHC}, Integrating prior knowledge Target search {F ts} and noise scheduling α t And it will be initialized by incorporating prior knowledge:
[0038]
[0039] Then, loop through time step t from 1 to T:
[0040] 1. From the perspective of integrating prior knowledge Initial state of mid-sampling
[0041]
[0042] 2. From the standard normal distribution Sampling noise ∈:
[0043]
[0044] 3. According to the noise plan Calculate noisy states
[0045]
[0046] 4. Use a denoising network ∈ θ Predicted noise
[0047]
[0048] 5. Calculate the denoised state (This task cannot be obtained directly during the reverse process) The following formula is an estimate:
[0049]
[0050] 6. Using refined networks n θ Further optimize the state after noise reduction
[0051]
[0052] 7. Update the parameters θ using gradient descent on the gradient of the loss function L:
[0053]
[0054] in, Let L be the gradient descent function. diff The loss function is defined by noise constraints.
[0055] Finally, when the parameter θ converges, the loop ends, and the entire forward diffusion process also ends.
[0056] Backdiffusion is the process of generating the target data distribution from an isotropic Gaussian distribution. Specifically, this is the posterior p... θ (x t-1 |x t A distribution of ). This can be derived using Bayesian theory and mathematical principles:
[0057]
[0058] in, and Let these represent the mean and variance of the distribution, respectively. Specifically, according to the definition of diffusion, they can be expressed as:
[0059]
[0060] Where, ∈ θ For noise reduction networks, Let be the variance of the distribution.
[0061] This invention achieves the reverse diffusion generation process by predicting the mean through a network. The introduction of conditions can be broadly categorized into two methods: a score-based unconditional diffusion paradigm and a conditional diffusion paradigm. A conditional diffusion multimodal image fusion paradigm is used to achieve more accurate and effective fusion.
[0062] Conditional diffusion: The conditional diffusion model is guided by a conditional neural network and directly estimates the mean.
[0063]
[0064] in Conditional diffusion can be obtained through a trained diffusion model. Compared to unconditional diffusion, conditional diffusion allows for more targeted guidance and control of generation based on task characteristics, making the diffusion process more compact and suitable for image fusion tasks. Our goal is to propose a multimodal image fusion paradigm based on conditional diffusion to achieve more accurate and effective fusion.
[0065] The reverse diffusion process specifically includes:
[0066] First, input fixed parameters: denoising network {∈θ}、Refine the network{n θ}、Source Image {I H&E}、Target staining domain image {I IHC} and noise scheduling α t And initialize the sample:
[0067]
[0068] Then, the time step t is looped from T-1 to 1:
[0069] t = T-1,…,1
[0070] 1. Using a denoising network ∈ θ Predict the current noisy state noise
[0071]
[0072] 2. Calculate the denoised state
[0073]
[0074] 3. Using a refined network n θ Further optimize the state after noise reduction (That is, the task of the Multi-modal Diffusion Refinement Module (MDRM)):
[0075]
[0076] 4. From the standard normal distribution New noise ∈ in the sampling:
[0077]
[0078] 5. According to the noise plan Calculate the noisy state at the next time step t-1
[0079]
[0080] Finally, output the state at time step t=0. This is the merged image.
[0081] The entire process is based on conditional input (such as I). IHC I H&E The network is guided by [the network name]. The posterior conditional distribution can be obtained as follows:
[0082]
[0083] The mean of the posterior conditional distribution is:
[0084]
[0085] The predicted mean is transferred to the predicted added noise. That is, the added Gaussian noise is directly predicted through the diffusion denoising module (DDM).
[0086] In this embodiment, the denoising network ∈ θ and refined network n θ The network is implemented using time-step t modulation. Specifically, it consists of a time-step coding layer and an image encoder-decoder. The SR3 backbone network is used for the denoising network. This invention modifies the Restromer backbone, representing its blocks as R blocks, achieving dual-input implementation and efficient coupling in the refinement network. Furthermore, given a source image I... H&E I IHC The approximate value of x0, i.e. It can be represented as:
[0087]
[0088] In the formula: and This represents the encoder of the network. Then, the extracted features are fused using a cascade operator and injected into the decoder via skip connections. It can be represented as:
[0089]
[0090] in It is a network decoder, v x and v r This represents the features extracted by the encoder.
[0091] The training function of the model structure in this application includes a loss function for noise constraints, a fusion of prior knowledge references, and a loss function for strength, gradient, and structural similarity.
[0092] The loss function used for noise constraint is:
[0093]
[0094] The loss function that integrates prior knowledge references, strength, gradient, and structural similarity is:
[0095]
[0096] Among them, L int Used to adjust the intensity of the fused image, L gradL is used to provide additional gradient information for guidance as needed. SSIM This is used to verify the similarity between the fused image and the source image. Based on the characteristics of the cross-domain staining task, a hyperparameter γ was set to meet the task objectives. diff =8, γ int =4,γ grad =5,γ SSIM =1. The effect of this invention compared with other methods is as follows: Figure 2 As shown.
[0097] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for cross-chromosome transformation of medical images based on multimodal feature fusion, characterized in that, include: Acquire medical images of the target staining region and the source staining region; Preprocessing is performed on the medical image of the target chromatid region and the image of the source chromatid region; Rich texture detail features are extracted from the preprocessed source chromaticity image, and rich color features are extracted from the preprocessed target chromaticity image. The rich texture detail features and rich color features are fused and used as the fusion knowledge prior distribution. The fusion knowledge prior distribution is used for target search to find the dataset prior distribution that meets the requirements of the fusion task and obtain the optimal distribution solution. The optimal distribution solution is initialized and forward diffused to obtain a noisy fused image. The noisy fused image is backdiffused to obtain the fused image.
2. The method for cross-chromosome transformation of medical images based on multimodal feature fusion according to claim 1, characterized in that, Image preprocessing includes: segmenting effective images using a sliding window strategy, filtering the segmented images to obtain high-quality pathological images, and dividing the high-quality pathological images into training, testing, and validation sets.
3. The method for cross-chromosome transformation of medical images based on multimodal feature fusion according to claim 1, characterized in that, The fusion of prior knowledge distribution is as follows: in, To integrate datasets with prior knowledge distributions, φ represents detailed texture features, and I... H&E For source staining domain medical images, I IHC For medical images of the target staining region, For the target color of the image, This describes the detailed texture features of the source stained image.
4. The method for cross-chromosome transformation of medical images based on multimodal feature fusion according to claim 1, characterized in that, Performing a target search includes: defining the target search function f ts (i) The target search function is updated based on the sample images in the fusion distribution according to prior knowledge. The updated target search function is used to search for the target and obtain the optimal distribution solution.
5. The method for cross-chromosome conversion of medical images based on multimodal feature fusion according to claim 4, characterized in that, The target search function is: Where SSIM(i) is the similarity function, ω ssim ω vif , All are hyperparameters, VIF(i) is the visual fidelity function, and Q... AB / F (i) is the texture information function.
6. The method for cross-chromosome transformation of medical images based on multimodal feature fusion according to claim 4, characterized in that, The target search function searches for the target as follows: in, It is the optimal distribution solution obtained through target search in the fusion of prior knowledge, I j The sample images are fused from the distribution, where j is the index of the j-th sample. In order to integrate prior knowledge, For the preprocessed dataset, F ts For the target search process, f ts The target search function.
7. The method for cross-chromosome transformation of medical images based on multimodal feature fusion according to claim 1, characterized in that, Forward diffusion includes: S101, Obtain the denoising network {∈ θ }、Refine the network{n θ }、Source Image {I H&E }、Target staining domain image {I IHC }, Integrating prior knowledge Target search {F ts } and noise scheduling α t ; S102. Collect initial state from fused prior knowledge. S103, From the standard normal distribution Sampling noise ∈; S104, Based on noise Calculate noisy states S105, Noise-containing state Input to denoising network ∈ θ Mid-prediction noise S106. Calculate the denoised state. S107, Using a refined network n θ Further optimize the state after noise reduction S108. Apply gradient descent to update parameter θ based on the gradient of loss function L; repeat steps S103 to S108 until parameter θ converges, at which point the loop ends.
8. The method for cross-chromosome transformation of medical images based on multimodal feature fusion according to claim 1, characterized in that, Backdiffusion includes: S201, Obtain the denoising network {∈ θ }、Refine the network{n θ }、Source Image {I H&E }、Target staining domain image {I IHC } and noise scheduling α t And initialize the samples; set the time step t; S202, Using a denoising network ∈ θ Predict the current noisy state noise S203, Calculate the denoised state S204, Using a refined network n θ Further optimize the state after noise reduction S205, From the standard normal distribution New noise is sampled in the middle; S206, According to the noise plan Calculate the noisy state at the next time step t-1 S207. Finally, output the state at time step t=0. This is the merged image.